Forklift early warning method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a forklift early warning method and device, electronic equipment and computer readable storage medium, which are applied to a forklift, an image acquisition module is arranged on the forklift, the method comprises the following steps: acquiring a scene image of a working scene currently where the forklift is located through the image acquisition module; when a target object of a preset type is identified in the scene image through a preset neural network, it is judged whether the target object is located within a preset early warning range, wherein the preset early warning range is determined according to the position of the image acquisition module and a preset characteristic curve, the preset characteristic curve is a safety range boundary of the target object of the preset type relative to the forklift, which is determined according to the framing characteristics of the image acquisition module; if the judgment result is yes, an early warning operation is performed. The method improves the accuracy of forklift early warning and reduces the complexity of the algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of industrial operation safety, and in particular to a forklift early warning method and device, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] A forklift, also known as an industrial truck, is a wheeled vehicle used to load, unload, stack, and transport palletized goods over short distances. It is an essential piece of equipment in the transportation of goods at ports, stations, airports, freight yards, factory workshops, warehouses, distribution centers, and other facilities.

[0003] Due to the particularity and complexity of the forklift working environment, there is frequent interaction between people and vehicles, and the driver's visibility is limited, which can lead to serious accidents due to fatigue, blind spots, reversing, turning, and other factors. In the prior art, the conventional obstacle avoidance early warning method is to install multiple sensors around the forklift, set a warning threshold, and uniformly detect all obstacles around the forklift to obtain real-time distance values of the obstacles. In combination with an obstacle avoidance algorithm, it is determined whether there are obstacles in the warning area around the forklift, and then whether an alarm is triggered.

[0004] However, in actual applications, it is usually not necessary to detect all obstacles, and the existing obstacle avoidance early warning method cannot filter out irrelevant obstacles, resulting in low alarm accuracy and high requirements for algorithm robustness. SUMMARY

[0005] The present application provides a forklift early warning method, device, electronic device, and computer readable storage medium to solve the problem of low alarm accuracy and high requirements for algorithm robustness in existing forklifts.

[0006] In one aspect, the present application provides a forklift early warning method applied to a forklift, wherein an image acquisition module is provided on the forklift. The method comprises:

[0007] acquiring a scene image of a working scene currently occupied by the forklift through the image acquisition module;

[0008] when a target object of a preset type is identified in the scene image through a preset neural network, determining whether the target object is located within a preset warning range, wherein the preset warning range is determined according to the position of the image acquisition module and a preset characteristic curve, and the preset characteristic curve is a safety range boundary of the target object of the preset type relative to the forklift determined according to the framing characteristics of the image acquisition module;

[0009] if the determination result is yes, a warning operation is performed.

[0010] Optionally, the preset characteristic curve is a parabola, and a shape feature of the parabola is determined according to at least three feature points on the safety range boundary.

[0011] Optionally, if the preset type is used to represent a pedestrian, the preset warning range includes a head preset warning range and a foot preset warning range, and the warning operation is performed when it is determined that the target object is located in the head preset warning range and / or the foot preset warning range.

[0012] The head preset warning range is a range formed by the image acquisition module and a first parabola, wherein the first parabola is determined according to a first feature point, a second feature point and a third feature point of a head of a preset calibration object on the safety range boundary.

[0013] The foot preset warning range is a range formed by the image acquisition module and a second parabola, wherein the second parabola is determined according to a fourth feature point, a fifth feature point and a sixth feature point of a foot of the preset calibration object on the safety range boundary.

[0014] Optionally, the determining whether the target object is located in the preset warning range includes:

[0015] Obtaining coordinate information of the target object in a current coordinate system, the coordinate information including an abscissa and an ordinate;

[0016] Calculating a warning ordinate according to the abscissa and a feature function corresponding to the preset characteristic curve;

[0017] Determining whether the ordinate is greater than the warning ordinate.

[0018] Optionally, before the warning operation is performed, the method further includes: determining that the forklift is currently in a moving state.

[0019] Optionally, the image acquisition module includes a 4-way camera and an embedded platform, and the 4-way camera is arranged at four corner positions of the forklift.

[0020] The 4-way camera is respectively configured with a separate Internet Protocol (IP) address, and accesses a router through the respective corresponding IP address, so as to ensure that the 4-way camera and the embedded platform are in the same local area network.

[0021] In a second aspect, the application provides a forklift warning device, including:

[0022] An image acquisition module, configured to obtain a scene image of a working scene in which a forklift is currently located.

[0023] The object detection module is configured to determine whether a target object of a preset type is located within a preset warning range when the target object is identified in the scene image by using the preset neural network, wherein the preset warning range is determined according to a position of the image acquisition module and a preset characteristic curve, and the preset characteristic curve is a safety range boundary of the target object of the preset type relative to the forklift determined according to a framing characteristic of the image acquisition module.

[0024] The object warning module is configured to perform a warning operation.

[0025] Optionally, the preset characteristic curve is a parabola, and a shape feature of the parabola is determined according to at least three feature points on the safety range boundary.

[0026] Optionally, the object warning module is further configured to:

[0027] If the preset type is used to represent a pedestrian, the preset warning range includes a head preset warning range and a foot preset warning range, and the warning operation is performed when the target object is determined to be located within the head preset warning range and / or the foot preset warning range.

[0028] The head preset warning range is a range formed by the image acquisition module and a first parabola, wherein the first parabola is determined according to a first feature point, a second feature point, and a third feature point of a head of a preset calibration object on the safety range boundary.

[0029] The foot preset warning range is a range formed by the image acquisition module and a second parabola, wherein the second parabola is determined according to a fourth feature point, a fifth feature point, and a sixth feature point of a foot of the preset calibration object on the safety range boundary.

[0030] Optionally, the object detection module is specifically further configured to:

[0031] Obtain coordinate information of the target object in a current coordinate system, wherein the coordinate information includes an abscissa and an ordinate; calculate a warning ordinate according to the abscissa and a characteristic function corresponding to the preset characteristic curve; and determine whether the ordinate is greater than the warning ordinate.

[0032] Optionally, the forklift warning device further includes:

[0033] The brake detection module is configured to determine that the forklift is currently in a moving state before the warning operation is performed.

[0034] Optionally, the image acquisition module includes a 4-way camera and an embedded platform, and the 4-way camera is arranged at four corner positions of the forklift.

[0035] The four cameras are respectively configured with separate Internet Protocol (IP) addresses, and access the router through the respective IP addresses, so as to ensure that the four cameras and the embedded platform are in the same local area network.

[0036] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory connected with the processor in communication;

[0037] The memory stores computer-executable instructions.

[0038] The processor executes the computer-executable instructions stored in the memory to implement the forklift early warning method.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the forklift early warning method.

[0040] The present application provides a forklift early warning method, device, electronic device and computer-readable storage medium. By setting an image acquisition module on the forklift, and pre-determining a characteristic curve according to the position of the image acquisition module and the safety range distance of the target object relative to the forklift; acquiring the scene image of the current work of the forklift through the image acquisition module, and identifying whether there is a target object of a preset type in the scene image through a preset neural network, and then determining whether the target object is within the preset early warning range through the characteristic curve, if the result is yes, an early warning operation is performed. By acquiring the scene image through the image acquisition module and identifying the target object through the neural network, other obstacles that do not need to be warned can be filtered out, early warning for specific obstacles is realized, the problem of inaccurate and inflexible alarm is avoided, and the accuracy of forklift early warning is improved. Moreover, the characteristic curve is used to determine whether to early warning, the algorithm is simple, the early warning speed is fast, the requirement for algorithm robustness is not high, and the complexity of the algorithm is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] Figure 1 A flowchart of a forklift early warning method embodiment one provided by the present application embodiment is shown;

[0043] Figure 2 A flowchart of a forklift early warning method embodiment two provided by the present application embodiment is shown;

[0044] Figure 3The implementation scenario diagram of the forklift early warning method embodiment two provided by the embodiment of the application is shown in the figure.

[0045] Figure 4 The camera viewfinder feature and early warning range diagram provided by the embodiment of the application is shown in the figure.

[0046] Figure 5 The structure diagram of the forklift early warning device provided by the embodiment of the application is shown in the figure.

[0047] Figure 6 The structure diagram of the electronic device provided by the embodiment of the application is shown in the figure.

[0048] Through the above-mentioned drawings, the specific embodiments of the application have been shown, and more detailed descriptions will be given hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of the application by any means, but to illustrate the concept of the application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0049] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. In the following description, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments are not representative of all embodiments consistent with the present application. Rather, they are merely examples of devices and methods consistent with some aspects of the present application, as detailed in the appended claims.

[0050] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented, for example, in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0051] With the continuous progress of science and technology, artificial intelligence is entering various industries, and forklift early warning is the most important in forklift operation safety. Application of artificial intelligence in forklift early warning will certainly bring different effects.

[0052] The traditional obstacle avoidance warning method sets a warning threshold, and then uses a radar sensor or a laser radar to perceive the position, shape and posture of all obstacles in the surrounding in real time, and needs to calculate the real-time distance value of the obstacle area. When the obstacle enters the warning position, an alarm is given. The algorithm requires high robustness. Since it does not have visual information, it cannot identify the type of perceived obstacle, so it cannot complete the warning reminder for a specific type of obstacle, and the warning accuracy is not high. In the actual working environment, it is usually not necessary to perceive and detect all obstacles around, nor to obtain the real-time distance value of the target obstacle in real time. When only fixed obstacles need to be warned, the traditional obstacle avoidance warning method will no longer be applicable.

[0053] Therefore, in view of the above problems, in combination with the special working environment and special needs of the forklift, the application provides a forklift warning method. A feature curve is determined in advance according to the position of the image acquisition module and the safe range distance of the target object relative to the forklift. The image acquisition module obtains the scene image of the current working scene of the forklift, and a preset neural network is used to identify whether a target object of a preset type exists in the scene image. Then, the feature curve is used to determine whether the target object is within the preset warning range. If the result is yes, a warning operation is performed. The scene image is collected by the image acquisition module, and the target object is identified by the neural network. Other obstacles that do not need to be warned can be filtered out, the warning for specific obstacles is realized, the problems of inaccurate and inflexible warning are avoided, and the algorithm is simple, the warning speed is fast, and the robustness requirement of the algorithm is not high.

[0054] The technical solutions of the application and how the technical solutions solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the application will be described below with reference to the drawings.

[0055] The forklift warning method provided by the application is applied to a forklift, and an image acquisition module is arranged on the forklift. Figure 1 The flowchart of the forklift warning method provided by the embodiment of the application is shown in the figure. The execution subject of the embodiment can be a server, such as Figure 1 The forklift warning method provided by the embodiment includes:

[0056] S101, obtaining a scene image of a current working scene of a forklift through an image acquisition module.

[0057] Specifically, the image acquisition module acquires the scene image of the current working scene of the forklift in real time. The scene image acquired by the image acquisition module can be different according to different information such as the placement angle, placement position, model and parameters of the image acquisition module. The specific implementation form of the image acquisition module is not limited in the embodiment, as long as it can ensure that all scene images of the environment around the forklift are collected clearly and without dead angles.

[0058] In S102, when the preset type of target object is identified in the scene image by the preset neural network, it is determined whether the target object is located in the preset warning range.

[0059] The neural network is a mathematical model of algorithm that simulates the behavior characteristics of animal neural networks and performs distributed parallel information processing. In actual application, a practical artificial neural network model can be built according to the principle of biological neural networks and the needs of actual application, a corresponding learning algorithm is designed to simulate certain intelligent activities of the human brain, and then it is technically realized to solve actual problems.

[0060] Exemplarily, the neural network of the application can extract high-dimensional features on the image by convolution. Since different types of objects have different features, the fully connected layer of the network can classify the features to further identify the types of objects.

[0061] The preset neural network provided by the application is pre-trained to identify the preset type of target object and determine whether the target object is in the warning range. The preset type of target object is a forklift warning object, for example, it can be an obstacle with the same or similar features that can be identified by the neural network, or it can be a pedestrian. The embodiment is not limited specifically.

[0062] Specifically, the image acquisition module acquires the scene image of the current working scene of the forklift in real time, and continuously transmits the scene image to the neural network. The neural network can identify whether the preset type of target object exists in the scene image. The image acquisition module can transmit the scene image to the neural network through wired transmission, such as a USB interface, or wireless transmission, such as WiFi, Bluetooth, etc. The embodiment is not limited.

[0063] When the neural network identifies that there is no preset type of target object in the scene image, the process for the scene image ends. The neural network continues to identify newly received scene images, and when it identifies that there is a preset type of target object in the scene image, it outputs the coordinates of the target object and determines whether the target object is located in the preset warning range.

[0064] The preset warning range is determined according to the position of the image acquisition module and a preset characteristic curve. The preset characteristic curve is a preset type of target object relative to the safety range boundary of the forklift determined according to the framing feature of the image acquisition module.

[0065] Specifically, the preset characteristic curve is determined according to the safety range boundary of the target object relative to the forklift, combined with the framing feature of the image acquisition module. It can be understood that the image acquisition module collects the scene image in front of the image acquisition module center as the perspective. If multiple coordinates are marked at multiple different positions at the same distance from the image acquisition module, a coordinate system is established in the scene image, and these coordinates can be fitted. The fitting result is a circular arc, which can be compared to a characteristic curve.

[0066] For example, the left boundary of the scene image can be taken as the y-axis, the top as the x-axis, and the intersection of the upper left corner x-axis and y-axis as the origin to establish the coordinate system. Alternatively, the center line of the scene image can be taken as the y-axis, the top as the x-axis, and the intersection of the top x-axis and y-axis as the origin to establish the coordinate system. The specific method of establishing the coordinate system is not limited in the present application.

[0067] In some embodiments, it is assumed that the safety range boundary of the preset type of target object relative to the forklift is 5m, and there are multiple preset types of target objects at different positions 5m away from the forklift. The image acquisition module collects the scene image at this time, takes the center line of the scene image as the y-axis, the top as the x-axis, and the intersection of the top x-axis and y-axis as the origin to establish a rectangular coordinate system, and a characteristic curve containing variables x and y opening upward can be fitted.

[0068] In actual application, the preset characteristic curve determined by fitting the safety range boundary of the preset type of target object relative to the forklift according to the framing feature of the image acquisition module is stored in the warning device in advance. When the neural network identifies that there is a preset type of target object in the scene image, the coordinates (x0, y0) of the target object are output, and the x0 is brought into the preset characteristic curve to calculate the y value. Comparing the size of y value and y0 can determine whether the target object is within the preset warning range.

[0069] If the result of the judgment is that y0 is greater than y, it means that the target object is within the preset warning range; if the result of the judgment is that y0 is less than or equal to y, it means that the target object is not within the preset warning range.

[0070] S103, determining whether to perform a warning operation according to the judgment result.

[0071] Specifically, if the result of the judgment shows that the target object is within the preset warning range, a warning operation is performed; if the result of the judgment shows that the target object is not within the preset warning range, no warning operation is needed.

[0072] The implementation manner of the early warning operation can be installing a warning bell, and the warning bell rings to alarm. The larger the sound of the warning bell is, the higher the alarm level is, and the closer the target object is to the forklift. The early warning operation can also be controlling the driving speed of the forklift while the warning bell alarms, which is not limited in the application. The early warning operation is intended to remind the forklift driver to pay attention to avoiding obstacles and driving safely.

[0073] The forklift early warning method provided in the embodiments of the application obtains the scene image of the current working scene of the forklift through the image acquisition module, and identifies whether the target object of a preset type exists in the scene image through a preset neural network. When it is identified through the preset neural network that the target object of the preset type exists in the scene image, it is determined whether the target object is located within a preset early warning range. If the determination result is that the target object is located within the preset early warning range, an early warning operation is performed. The scene image is obtained by using the image acquisition module, and the coordinates of the target object are determined in combination with the neural network, so that the early warning object and its position can be determined more intuitively, and other obstacles that do not need to be warned are filtered out, which is more flexible and can realize early warning for specific target objects. The early warning range is related to the safety range boundary of the target object relative to the forklift and the viewfinder feature of the image acquisition module, is preset as a feature curve, and the early warning range is locked. Whether early warning is performed can be determined according to the coordinates output by the neural network, which is simple in algorithm, reduces the complexity of the algorithm, and is more beneficial to deployment and application.

[0074] Figure 2 The flowchart of the forklift early warning method provided in Embodiment Two of the application is shown in the figure. The execution subject of the embodiment can be a server, and a pedestrian is taken as an example of the target object of a preset type for detailed description. As shown in the figure, the forklift early warning method provided in the embodiment includes the following steps. Figure 2

[0075] S201, obtaining a scene image of a current working scene of a forklift through an image acquisition module.

[0076] Specifically, the image acquisition module collects the scene image of the current working scene of the forklift in real time to ensure that all scene images of the environment around the forklift are collected clearly and without dead angles.

[0077] Optionally, the image acquisition module includes four cameras and an embedded platform, wherein the four cameras are respectively arranged at four corner positions of the forklift; the four cameras are respectively configured with a separate Internet Protocol (IP) address, and are connected to a router through the respective IP addresses to ensure that the four cameras and the embedded platform are in the same local area network.

[0078] ​Specifically, when the image acquisition module includes four cameras, each positioned at one of the four corners of the forklift's top, and the angle of each camera is adjusted, it ensures that all four cameras can clearly capture images of the entire scene surrounding the forklift without blind spots. The image acquisition module also includes an embedded platform for transmitting the acquired scene images. To avoid data overlap and latency, each of the four cameras is configured with a separate Internet Protocol (IP) address. The four cameras connect to the router via their respective IP addresses, ensuring that the four cameras and the embedded platform are on the same local area network, thus guaranteeing real-time transmission of scene images. The cameras are connected to the embedded platform via wired connections. This application does not limit the specific form of the embedded platform.

[0079] For example, to minimize data latency, the embedded platform can be TX2, and the image transmission between the camera and the embedded platform TX2 can use Real Time Streaming Protocol (RTSP) streaming media.

[0080] S202. Identify whether there are pedestrians in the scene image through a preset neural network.

[0081] Specifically, the embedded platform in the image acquisition module transmits the scene image to the neural network, which then identifies whether pedestrians are present in the scene image. If the neural network determines that no pedestrians are present, it indicates that there are no objects requiring warning in the scene, and the work environment is safe. If the neural network identifies that pedestrians are present, it needs to further determine whether the pedestrians are within the warning range. At this point, the neural network outputs the coordinates of the pedestrians.

[0082] For example, to improve the efficiency of neural networks in recognizing scene images, the neural network can use a YOLOv5 network and be accelerated using the TensorRT library on a GPU.

[0083] S203. Determine whether a pedestrian is within a preset warning range using a preset neural network.

[0084] Optionally, if the preset type is used to represent a pedestrian, the preset warning range includes a head preset warning range and a foot preset warning range. When it is determined that the target object is within the head preset warning range and / or the foot preset warning range, a warning operation is performed.

[0085] Specifically, when the early warning object is a pedestrian, the preset type is used to represent the pedestrian. In the same coordinate system, the same pedestrian can have numerous coordinates at the same distance from the image acquisition module. The head coordinates and foot coordinates can best represent the position of the pedestrian. Therefore, to avoid the situation that the body of the pedestrian is partially blocked by other objects and early warning cannot be performed, the early warning range is set to include a head preset early warning range and a foot preset early warning range. When it is determined that the head or foot of the pedestrian is located in any early warning range, the early warning operation is performed.

[0086] The preset early warning range is determined according to the position of the image acquisition module and a preset characteristic curve. The preset characteristic curve is a safety range boundary of the pedestrian relative to the forklift determined according to the viewing characteristics of the image acquisition module.

[0087] Optionally, the preset characteristic curve is a parabola, and the shape characteristics of the parabola are determined according to at least three feature points on the safety range boundary.

[0088] Specifically, a plurality of coordinates are marked at a plurality of different positions at the same distance from the image acquisition module. The curve fitted in the coordinate system established in the scene image is an arc similar to a parabola. Therefore, the preset characteristic curve can be fitted as a parabola. Understandably, the parabola equation is y=ax 2 +bx+c. According to the coordinates of the three points, the values of the coefficients a, b, and c can be solved, thereby determining the specific function of the parabola.

[0089] The head preset early warning range is a range formed by the image acquisition module and a first parabola. The first parabola is determined according to a first feature point, a second feature point, and a third feature point of the head of the preset calibration object on the safety range boundary.

[0090] The foot preset early warning range is a range formed by the image acquisition module and a second parabola. The second parabola is determined according to a fourth feature point, a fifth feature point, and a sixth feature point of the foot of the preset calibration object on the safety range boundary.

[0091] Specifically, according to the parabola characteristics, three pedestrians can be selected as calibration objects and stand at different positions relative to the safety range boundary of the forklift. The head coordinates of the three pedestrians at this time are obtained as the first feature point, the second feature point, and the third feature point. A set of coefficients a1, b1, and c1 can be calculated, and the first parabola equation y=a1x 2 +b1x+c1 is obtained. At the same time, the foot coordinates of the three pedestrians are obtained as the fourth feature point, the fifth feature point, and the sixth feature point. A set of coefficients a2, b2, and c2 can be calculated, and the second parabola equation y=a2x 2+ b2x + c2.

[0092] Alternatively, only one pedestrian is taken as the calibration object, and the pedestrian stands in different positions on the boundary of the safety range of the forklift in turn, three scene images are obtained, three head coordinates are obtained as the first feature point, the second feature point and the third feature point according to the three scene images, and three foot coordinates of the pedestrian are obtained as the fourth feature point, the fifth feature point and the sixth feature point, and the coefficients are solved to determine the first parabolic equation and the second parabolic equation.

[0093] Optionally, the number of preset feature curves and the feature function are designed to be modifiable.

[0094] Specifically, if only pedestrians are set as the warning objects, the feature curves are two, which are the head parabola and the foot parabola. If the warning objects are pedestrians and other types of obstacles, the feature curves should include the pedestrian feature curve and the obstacle feature curve. The feature function corresponds to the feature curve, and the specific form of the feature function is not limited in the present application, and can be designed according to the actual situation.

[0095] Exemplarily, Figure 3 The implementation scenario diagram of the forklift warning method embodiment two provided by the present application is shown. As shown in Figure 3 Fig. 1 is a forklift, 2A, 2B, 2C and 2D are four cameras arranged at the top four corners of the forklift, 3A, 3B, 3C and 3D are four scene images of the current working scene of the forklift obtained by the four cameras respectively, and 4A, 4B, 4C and 4D are four parabolic views obtained by fitting the scene images obtained by the four cameras respectively.

[0096] Exemplarily, Figure 4 The camera view feature and warning range diagram provided by the present application is shown. As shown in Figure 4 The obtuse angle composed of the arrow 100 and the arrow 200 is the view range of the camera 2A, and the shadow part in the feature curve 300 is the head warning range of the pedestrian. Since the feature curve 300 is similar to a parabola, the feature curve 300 can be fitted as a parabola. As shown in Figure 4 The left side of the diagram shows that the coordinate system is established in the scene image, the parabola 400 is obtained by fitting the feature curve 300, and the shadow part in the diagram is the head warning range of the pedestrian. The size of the obtuse angle is related to the model and configuration of the camera, and represents the view feature of the camera. The present application does not make specific limitations, as long as the four cameras can clearly and without dead angle obtain all the scene images of the environment around the forklift.

[0097] Optionally, the judgment of whether the target object is located in the preset warning range can include:

[0098] S2031. Obtain the coordinate information of the target object in the current coordinate system. The coordinate information includes the horizontal coordinate and the vertical coordinate.

[0099] Specifically, when the neural network identifies a target object in a scene image, it outputs the target object's coordinates, including the horizontal and vertical coordinates. When the target object is a pedestrian, the preset warning range includes a head preset warning range and a foot preset warning range. Therefore, the neural network outputs both the head and foot coordinates simultaneously.

[0100] S2032. Calculate the warning vertical coordinate based on the horizontal coordinate and the characteristic function corresponding to the preset characteristic curve.

[0101] Specifically, the horizontal coordinate output by the neural network is substituted into the feature function corresponding to the preset feature curve for calculation, and the result is the warning vertical coordinate. When the target object is a pedestrian, the horizontal coordinate of the head output by the neural network is substituted into the feature function corresponding to the first parabola to calculate the head warning vertical coordinate, and the horizontal coordinate of the feet is substituted into the feature function corresponding to the second parabola to calculate the foot warning vertical coordinate.

[0102] S2033. Determine whether the vertical axis is greater than the warning vertical axis.

[0103] Specifically, the ordinate output by the neural network is compared with the calculation result. If the ordinate output by the neural network is less than or equal to the calculation result, it indicates that the target object is not within the preset warning range. If the ordinate output by the neural network is greater than the calculation result, it indicates that the target object is within the preset warning range.

[0104] When there are multiple preset characteristic curves in the design, multiple warning ordinates need to be calculated. When judging, if any one of the output ordinates is greater than the calculated result, it indicates that a target object is within the warning range.

[0105] For example, when the target object is a pedestrian, the neural network output coordinates include the pedestrian's head coordinates and foot coordinates. The head warning coordinate and foot warning coordinate need to be calculated separately. As long as at least one of the pedestrian's head or foot is within the warning range, the result is that the pedestrian is within the warning range. Only when neither the head coordinate nor the foot coordinate is within the warning range is the result that the pedestrian is not within the warning range.

[0106] This embodiment calculates the warning coordinates by outputting the horizontal coordinate, and then compares the output coordinates with the warning coordinates to determine whether the target object is within the warning range. The calculation method is simple, and the algorithm logic is straightforward. This method reduces the requirements for algorithm robustness and complexity, making it easier to deploy and apply.

[0107] S204, determine whether the forklift is in a moving state according to the judgment result.

[0108] Specifically, to avoid false alarms to the staff in special working conditions, such as when unloading and loading goods, before the warning operation, it can also include: determining whether the forklift is currently in a moving state.

[0109] If the judgment result is that the pedestrian is within the warning range, determine whether the forklift is in a moving state; if the judgment result is that the pedestrian is not within the warning range, the process for the scene image ends.

[0110] Alternatively, to determine whether the forklift is in a moving state, the warning device can be connected to the control system of the forklift, so that the warning is triggered only when the forklift is moving, and no warning is given when it is stationary. Alternatively, a positioning system can be provided in the warning device to determine whether the forklift is in a moving state. The specific way of determining the current moving state of the forklift is not limited in the present application.

[0111] S205, determine whether to perform a warning operation according to the determination result.

[0112] If it is determined that the forklift is in a moving state, a warning operation is performed; if it is determined that the forklift is in a stationary state, the process ends.

[0113] This embodiment takes pedestrians as an example of a preset type of target object, acquires the scene image of the current working scene of the forklift through the image acquisition module, and identifies whether there is a target object of a preset type in the scene image through a preset neural network. When the target object of a preset type is identified in the scene image through the preset neural network, it is determined whether the target object is within the preset warning range. When the judgment result is that the target object is within the preset warning range, it is further determined whether the forklift is in a moving state. If it is determined that the forklift is in a moving state, a warning operation is performed. Fully considering the characteristics of the target object and the working environment of the forklift, the forklift warning is more accurate. On the one hand, considering that all scene images around the forklift need to be clearly and completely collected, a camera is arranged at each of the four corners of the top of the forklift. At the same time, to avoid image transmission delay, each camera is configured with a separate IP address and is separately connected to a router, so that image data cross does not occur and false alarms are avoided. On the other hand, considering that pedestrians are partially blocked, at least two curves are set when the feature curve is preset, so that the pedestrians cannot be identified due to partial blocking. At the same time, the feature curve is set as a parabola, which has a simple algorithm and does not require high robustness, and is more suitable for deployment and application.

[0114] The following is an embodiment of the device of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0115] Figure 5 A structural schematic diagram of a forklift early warning device provided by an embodiment of the present application is shown in the figure. Figure 5 As shown in the figure, the forklift early warning device 10 of the embodiment includes an image acquisition module 11, an object detection module 12, and an object early warning module 13.

[0116] The image acquisition module 11 is configured to acquire a scene image of a working scene in which the forklift is currently located.

[0117] The object detection module 12 is configured to, when a target object of a preset type is identified in the scene image by a preset neural network, determine whether the target object is located within a preset early warning range, wherein the preset early warning range is determined according to a position of the image acquisition module and a preset characteristic curve, and the preset characteristic curve is a safety range boundary of the target object of the preset type relative to the forklift determined according to a framing characteristic of the image acquisition module.

[0118] The object early warning module 13 is configured to perform an early warning operation.

[0119] Optionally, the preset characteristic curve is a parabola, and a shape feature of the parabola is determined according to at least three feature points on the safety range boundary.

[0120] Optionally, the object early warning module 13 is further configured to:

[0121] If the preset type is used to represent a pedestrian, the preset early warning range includes a head preset early warning range and a foot preset early warning range, and the early warning operation is performed when it is determined that the target object is located within the head preset early warning range and / or the foot preset early warning range.

[0122] The head preset early warning range is a range formed by the image acquisition module and a first parabola, wherein the first parabola is determined according to a first feature point, a second feature point, and a third feature point of a head of a preset calibration object on the safety range boundary.

[0123] The foot preset early warning range is a range formed by the image acquisition module and a second parabola, wherein the second parabola is determined according to a fourth feature point, a fifth feature point, and a sixth feature point of a foot of the preset calibration object on the safety range boundary.

[0124] Optionally, the object detection module is further configured to:

[0125] acquire coordinate information of the target object in a current coordinate system, the coordinate information including an abscissa and an ordinate; calculate a warning ordinate according to the abscissa and a feature function corresponding to the preset characteristic curve; and determine whether the ordinate is greater than the warning ordinate.

[0126] Optionally, the forklift early warning device further includes:

[0127] The brake detection module is configured to determine that the forklift is currently in a moving state before the pre-warning operation is performed.

[0128] Optionally, the image acquisition module comprises a 4-way camera and an embedded platform, and the 4-way camera is arranged at four corner positions of the forklift respectively.

[0129] The forklift pre-warning device provided by the embodiment can be used to execute the forklift pre-warning method of the above embodiment, and has similar implementation principles and technical effects, which will not be described here.

[0130] It should be noted that the division of each module of the above device is only a logical function division, and all or part of the modules can be integrated into one physical entity, or can be physically separated. And these modules can all be implemented in the form of software called by a processing element; all can be implemented in the form of hardware; some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the object detection module can be a separately established processing element, or can be integrated in a chip of the above device. In addition, the functions of the above data processing modules can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or can be independently implemented. The processing element here can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each of the above modules can be completed by the integrated logic circuit of hardware or the instruction of software in the processing element.

[0131] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), etc. For another example, when a certain module above is implemented in the form of scheduling program code by a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together to implement in the form of system-on-a-chip (SOC).

[0132] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. containing one or more available media sets. The available media can be magnetic media (such as floppy disk, hard disk, magnetic tape), optical media (such as DVD), or semiconductor media (such as solid state disk (SSD)) and the like.

[0133] Figure 6 The structure schematic diagram of the electronic device provided by the embodiments of the present application is provided. As shown in the figure, the electronic device 20 includes a processor 21 and a memory 22 in communication with the processor. Figure 6

[0134] The memory 22 stores computer execution instructions; the processor 21 executes the computer execution instructions stored in the memory 22 to realize any one of the forklift warning methods as described above.

[0135] In the specific implementation of the above electronic device, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC) and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor for execution.

[0136] ​The embodiment of the present application further provides a computer readable storage medium, which stores computer execution instructions. The computer execution instructions are executed by a processor to implement the forklift early warning method according to any one of the preceding embodiments.

[0137] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by computer instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program is executed to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various storage media that can store program codes, such as ROM, RAM, magnetic disk or optical disk.

[0138] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The application is intended to cover any variations, uses or adaptations of the application following the general principles thereof and including such departures from the present disclosure as come within known use or custom in the art to which the application pertains. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the application is indicated by the following claims.

[0139] It should be understood that the application is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the application. The scope of the application is limited only by the claims that follow.

Claims

1. A forklift early warning method, characterized in that, Applied to forklifts, wherein the forklifts are equipped with an image acquisition module, the method includes: The image acquisition module acquires a scene image of the forklift's current working environment. When a target object of a preset type is identified in the scene image by a preset neural network, it is determined whether the target object is within a preset warning range. The preset warning range is determined based on the position of the image acquisition module and a preset feature curve. The preset feature curve is the safe range boundary of the target object of the preset type relative to the forklift, determined based on the framing characteristics of the image acquisition module. The preset feature curve is a parabola, and the shape of the parabola is determined by fitting at least three feature points on the safe range boundary. If the judgment result is yes, then an early warning operation will be performed; Wherein, if the preset type is used to characterize a pedestrian, the preset warning range includes a head preset warning range and a foot preset warning range. When it is determined that the target object is within the head preset warning range and / or the foot preset warning range, the warning operation is performed. The head preset warning range is the range formed by the image acquisition module and the first parabola, wherein the first parabola is determined based on the first feature point, the second feature point, and the third feature point of the head of the preset calibration object on the boundary of the safe range. The foot preset warning range is the range formed by the image acquisition module and the second parabola, wherein the second parabola is determined based on the fourth feature point, the fifth feature point, and the sixth feature point of the foot of the preset calibration object on the boundary of the safe range. The determination of whether the target object is within the preset warning range includes: Obtain the coordinate information of the target object in the current coordinate system, the coordinate information including the horizontal coordinate and the vertical coordinate; The warning vertical coordinate is calculated based on the horizontal coordinate and the characteristic function corresponding to the preset characteristic curve; Determine whether the ordinate is greater than the warning ordinate.

2. The forklift early warning method according to claim 1, characterized in that, Before the warning operation is performed, the following is also included: It is determined that the forklift is currently in a moving state.

3. The forklift early warning method according to claim 2, characterized in that, The image acquisition module includes four cameras and an embedded platform, with the four cameras respectively positioned at the four corners of the forklift. Each of the four cameras is configured with a separate Internet Protocol (IP) address and connects to the router through its corresponding IP address to ensure that the four cameras and the embedded platform are on the same local area network.

4. A forklift warning device, characterized in that, include: The image acquisition module is used to acquire scene images of the forklift's current working environment; The object detection module is used to determine whether a target object of a preset type is located within a preset warning range when a preset neural network identifies a target object of a preset type in the scene image. The preset warning range is determined based on the position of the image acquisition module and a preset feature curve. The preset feature curve is the safe range boundary of the target object of the preset type relative to the forklift, determined based on the framing characteristics of the image acquisition module. The preset feature curve is a parabola, and the shape of the parabola is determined by fitting at least three feature points on the safe range boundary. The object early warning module is used to perform early warning operations; The object early warning module is also used for: If the preset type is used to characterize a pedestrian, the preset warning range includes a head preset warning range and a foot preset warning range. When it is determined that the target object is within the head preset warning range and / or the foot preset warning range, the warning operation is performed. The head preset warning range is the range formed by the image acquisition module and the first parabola, wherein the first parabola is determined based on the first feature point, second feature point, and third feature point of the head of the preset calibration object on the boundary of the safe range. The foot preset warning range is the range formed by the image acquisition module and the second parabola, wherein the second parabola is determined based on the fourth feature point, fifth feature point, and sixth feature point of the foot of the preset calibration object on the boundary of the safe range. The object detection module is also used for: Obtain the coordinate information of the target object in the current coordinate system, the coordinate information including the horizontal coordinate and the vertical coordinate; The warning vertical coordinate is calculated based on the horizontal coordinate and the characteristic function corresponding to the preset characteristic curve; Determine whether the ordinate is greater than the warning ordinate.

5. An electronic device, comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-3.

Images