Method and device for water sprinkler to avoid pedestrians based on visual identification area proportion detection

Abstract

The invention discloses a pedestrian avoidance method and device for a watering cart based on visual identification area proportion detection, and the method comprises the steps: constructing a target detection model based on a Yolov5 target detection algorithm, and deploying the target detection model in the watering cart; after the watering cart is started, a camera is started through a GStreamer pipeline to obtain a video stream, an image in the video stream is preprocessed, and an optimized image is obtained; inputting the optimized image into a target detection model for target object detection, and outputting a target object rectangular frame and coordinate information of the rectangular frame; the working mode of the watering cart is selected through lane line recognition; the real-time proportion of the rectangular frame of the target object entering the EWR area is obtained through real-time calculation; and comparing the real-time proportion with a first set threshold value and a second set threshold value, and performing corresponding processing according to a comparison result. According to the method, the reliability and accuracy of visual identification can be effectively optimized, the problems of partial misjudgment, missed judgment and the like are solved, and the performance of a system in a complex scene is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual recognition of sanitation vehicles, and in particular to a method and device for a sprinkler truck to avoid pedestrians based on visual recognition area proportion detection. Background Art

[0002] In urban road maintenance, landscaping and cleaning operations frequently rely on the operation of sprinkler trucks. However, some sprinkler trucks ignore pedestrian safety when performing their tasks. They do not stop the operation of high-pressure water guns even if there are pedestrians around. The strong water flow sprayed by such water guns onto the ground may spray pedestrians because they cannot avoid it in time, causing complaints from pedestrians.

[0003] At present, there are no cases in the market that associate visual recognition systems with new energy sprinkler trucks. Through visual recognition detection, the probability of pedestrians being splashed by sprinkler trucks is effectively reduced, and the probability of pedestrian complaints is also reduced. However, visual recognition detection without EWR is prone to misjudgment (pedestrians are in areas that are not sprayed but the water valve is still closed) or premature closure of the water valve. The EWR ratio detection reduces the misidentification of pedestrians, making sprinkler trucks more accurate and efficient in avoiding pedestrians.

[0004] Therefore, how to invent a method for a sprinkler truck to avoid pedestrians through visual recognition of EWR ratio detection, which can effectively optimize the reliability and accuracy of visual recognition, solve some problems such as misjudgment and missed judgment, and improve the performance of the system in complex scenarios has become an urgent problem to be solved. Summary of the invention

[0005] To this end, the present invention provides a method and device for sprinkler truck to avoid pedestrians based on visual recognition area ratio detection. The judgment condition of the intersection ratio of pedestrian detection area and EWR is added to the sprinkler truck pedestrian avoidance system, which can effectively optimize the reliability and accuracy of visual recognition, solve some problems such as misjudgment and missed judgment, and improve the performance of the system in complex scenes.

[0006] In order to achieve the above-mentioned purpose, the present invention provides the following technical solution: a method for a sprinkler truck to avoid pedestrians based on visual recognition area proportion detection, comprising: Based on the Yolov5 target detection algorithm, a target detection model is constructed; the target detection model is trained by setting a data set to obtain the trained target detection model; the trained target detection model is deployed in a sprinkler truck; After the sprinkler truck is started, the camera is started through the GStreamer pipeline; a video stream is obtained through the camera; and a format conversion tool is set to convert the format of the video stream into a BGR format; Preprocessing the image in the video stream to obtain an optimized image; Inputting the optimized image into the target detection model, performing target object detection through the target detection model, and outputting the position and confidence information of the target object on the optimized image; selecting the target object according to the position and the confidence information, and outputting the target object rectangular frame and the coordinate information of the rectangular frame; Performing lane line recognition through the target detection model, and selecting a sprinkler truck working mode according to the recognition result; According to the coordinate information of the rectangular frame, a real-time ratio of the area of ​​the target object rectangular frame entering the EWR area to the area of ​​the target object rectangular frame is obtained through real-time calculation; Compare the real-time proportion with a first set threshold, and if the real-time proportion is greater than the first set threshold, proceed to the next step; if the real-time proportion is not greater than the first set threshold, continue to detect the target object through the target detection model; When the real-time proportion is greater than the first set threshold, a message is sent through a set communication protocol to notify the upper controller to close the hedging in the direction of the target object; The real-time proportion is compared with a second set threshold value. If the real-time proportion is less than the second set threshold value, the next step of processing is performed; if the real-time proportion is not less than the second set threshold value, the hedging in the direction of the target object is kept closed; When the real-time proportion is less than the second set threshold, a message is sent through the set communication protocol to notify the upper body controller to open the hedging in the direction where the target object is located.

[0007] As a preferred solution for the method of sprinkler truck avoiding pedestrians based on visual recognition area proportion detection, in the process of training the target detection model through the set data set, the images in the set data set include several target objects; the images in the set data set cover several types of lighting, weather, angles and environmental backgrounds.

[0008] As a preferred solution for the method of avoiding pedestrians by a sprinkler truck based on visual recognition area ratio detection, in the process of deploying the trained target detection model in the sprinkler truck, the trained target detection model is used to generate a pt model file; and the pt model file is converted into a rknn model file.

[0009] As a preferred solution of the method for sprinkler trucks to avoid pedestrians based on visual recognition area ratio detection, in the process of preprocessing the images in the video stream, the preprocessing includes image distortion correction processing, letterbox scaling adjustment processing, and sequential conversion processing of color channels.

[0010] As a preferred solution of the method for sprinkler trucks to avoid pedestrians based on visual recognition area ratio detection, in the process of obtaining the real-time ratio of the area of ​​the target object rectangular frame entering the EWR area and the area of ​​the target object rectangular frame through real-time calculation, five judgment calculation strategies are set according to the position of the target object rectangular frame entering the EWR area; First position situation: when three or more of the four coordinate points of the target object rectangular frame enter the EWR area, the real-time proportion calculation is not performed, and it is determined that the target object is in the EWR area; Second position situation: when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular frame are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not, and satisfy <960, <960 or >960, When >960, the real-time proportion calculation formula is: ; ; ; In the formula, The left horizontal coordinate of the target object rectangle; is the upper ordinate of the target object’s rectangular box; is the right horizontal coordinate of the target object rectangle; is the lower ordinate of the target object’s rectangular box; is the area of ​​the target object rectangle within the EWR region; is a linear function of the two straight lines of the fan; is the total area of ​​the target object rectangle; S is the real-time proportion of the target object rectangle entering the EWR area; Third position situation: when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular frame are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not within the EWR area, and <960, When >960, the real-time proportion calculation is not performed, and the target object is determined to be in the EWR area; Fourth position situation: when the lower left corner coordinates and the lower right corner coordinates of the target object rectangular frame are within the EWR area, but the upper left corner coordinates and the upper right corner coordinates are not within the EWR area, the real-time proportion calculation formula is: ; ; ; In the formula, is the area of ​​the target object rectangle within the EWR region; is a fan-shaped function; Fifth position situation: when two or more coordinate points of the target object rectangular frame are not within the EWR area, but the coordinates of the center point are within the EWR area, the real-time proportion calculation is not performed, and the target object is determined to be within the EWR area; The calculation formula for the center point coordinates of the target object rectangular frame is: ; ; In the formula, , is the center point coordinate.

[0011] The present invention also provides a device for avoiding pedestrians by a sprinkler truck based on the detection of the proportion of visual recognition area, and the method for avoiding pedestrians by a sprinkler truck based on the detection of the proportion of visual recognition area comprises:

[0012] The target detection model construction, training and deployment module is used to construct a target detection model based on the Yolov5 target detection algorithm; train the target detection model by setting a data set to obtain the trained target detection model; and deploy the trained target detection model in the sprinkler truck; The video stream acquisition and processing module is used to start the camera through the GStreamer pipeline after the sprinkler truck is started; obtain the video stream through the camera; and convert the format of the video stream into BGR format by setting a format conversion tool; An image preprocessing module, used for preprocessing the image in the video stream to obtain an optimized image; A target object detection and information marking module, used to input the optimized image into the target detection model, perform target object detection through the target detection model, and output the position and confidence information of the target object on the optimized image; select the target object according to the position and the confidence information, and output the target object rectangular frame and the coordinate information of the rectangular frame; A working mode selection module, used to identify lane lines through the target detection model and select a working mode of the sprinkler truck according to the identification result; A real-time ratio calculation module, used for obtaining the real-time ratio of the area of ​​the target object rectangular frame entering the EWR area to the area of ​​the target object rectangular frame through real-time calculation according to the coordinate information of the rectangular frame; A first real-time proportion comparison module is used to compare the real-time proportion with a first set threshold value, and if the real-time proportion is greater than the first set threshold value, the next step of processing is performed; if the real-time proportion is not greater than the first set threshold value, the target object detection is continued through the target detection model; A first control processing module, configured to, when the real-time proportion is greater than the first set threshold, send a message through a set communication protocol to notify the upper controller to close the hedging in the direction where the target object is located; A second real-time proportion comparison module is used to compare the real-time proportion with a second set threshold value, and if the real-time proportion is less than the second set threshold value, the next step of processing is performed; if the real-time proportion is not less than the second set threshold value, the hedging in the direction of the target object is kept closed; The second control processing module is used to send a message through the set communication protocol to notify the upper controller to open the hedging in the direction of the target object when the real-time proportion is less than the second set threshold.

[0013] As a preferred solution for the sprinkler truck pedestrian avoidance device based on visual recognition area ratio detection, in the target detection model construction, training and deployment module, in the process of training the target detection model through the set data set, the images in the set data set include several target objects; the images in the set data set cover several types of lighting, weather, angles and environmental backgrounds.

[0014] As a preferred solution for a sprinkler truck pedestrian avoidance device based on visual recognition area ratio detection, in the target detection model construction, training and deployment module, in the process of deploying the trained target detection model in the sprinkler truck, the trained target detection model is used to generate a pt model file; and the pt model file is converted into a rknn model file.

[0015] As a preferred solution for the sprinkler truck pedestrian avoidance device based on visual recognition area ratio detection, in the image preprocessing module, in the process of preprocessing the image in the video stream, the preprocessing includes image distortion correction processing, letterbox scaling adjustment processing, and sequential conversion processing of color channels.

[0016] As a preferred solution of the sprinkler truck pedestrian avoidance device based on visual recognition area ratio detection, in the real-time ratio calculation module, in the process of obtaining the real-time ratio of the area of ​​the target object rectangular frame entering the EWR area and the area of ​​the target object rectangular frame through real-time calculation, five judgment calculation strategies are set according to the position of the target object rectangular frame entering the EWR area; First position situation: when three or more of the four coordinate points of the target object rectangular frame enter the EWR area, the real-time proportion calculation is not performed, and it is determined that the target object is in the EWR area; Second position situation: when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular frame are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not, and satisfy <960, <960 or >960, When >960, the real-time proportion calculation formula is: ; ; ; In the formula, The left horizontal coordinate of the target object rectangle; is the upper ordinate of the target object’s rectangular box; is the right horizontal coordinate of the target object rectangle; is the lower ordinate of the target object’s rectangular box; is the area of ​​the target object rectangle within the EWR region; is a linear function of the two straight lines of the fan; is the total area of ​​the target object rectangle; S is the real-time proportion of the target object rectangle entering the EWR area; Third position situation: when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular frame are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not within the EWR area, and <960, When >960, the real-time proportion calculation is not performed, and the target object is determined to be in the EWR area; Fourth position situation: when the lower left corner coordinates and the lower right corner coordinates of the target object rectangular frame are within the EWR area, but the upper left corner coordinates and the upper right corner coordinates are not within the EWR area, the real-time proportion calculation formula is: ; ; ; In the formula, is the area of ​​the target object rectangle within the EWR region; is a fan-shaped function; Fifth position situation: when two or more coordinate points of the target object rectangular frame are not within the EWR area, but the coordinates of the center point are within the EWR area, the real-time proportion calculation is not performed, and the target object is determined to be within the EWR area; The calculation formula for the center point coordinates of the target object rectangular frame is: ; ; In the formula, , is the center point coordinate.

[0017] The present invention has the following advantages: the present invention constructs a target detection model based on the Yolov5 target detection algorithm; the target detection model is trained by setting a data set to obtain the trained target detection model; the trained target detection model is deployed in a sprinkler truck; after the sprinkler truck is started, the camera is started through the GStreamer pipeline; a video stream is obtained through the camera; the format of the video stream is converted into the BGR format by setting a format conversion tool; the image in the video stream is preprocessed to obtain an optimized image; the optimized image is input into the target detection model, the target object is detected by the target detection model, and the position and confidence information of the target object on the optimized image are output; the target object is framed according to the position and the confidence information, and a rectangular frame of the target object and the coordinate information of the rectangular frame are output; lane line recognition is performed through the target detection model, and a sprinkler truck working mode is selected according to the recognition result; the rectangular frame is selected according to the coordinate information of the rectangular frame coordinate information, and obtain the real-time ratio of the area of ​​the target object rectangular frame entering the EWR area to the area of ​​the target object rectangular frame through real-time calculation; compare the real-time ratio with the first set threshold, if the real-time ratio is greater than the first set threshold, proceed to the next step; if the real-time ratio is not greater than the first set threshold, continue to detect the target object through the target detection model; when the real-time ratio is greater than the first set threshold, send a message through the set communication protocol to notify the upper body controller to close the hedging in the direction where the target object is located; compare the real-time ratio with the second set threshold, if the real-time ratio is less than the second set threshold, proceed to the next step; if the real-time ratio is not less than the second set threshold, keep closing the hedging in the direction where the target object is located; when the real-time ratio is less than the second set threshold, send a message through the set communication protocol to notify the upper body controller to open the hedging in the direction where the target object is located. The present invention can more accurately identify the position of pedestrians through EWR proportion detection, reduce misjudgments, and improve the reliability and accuracy of visual recognition in complex environments. It can effectively optimize the reliability and accuracy of visual recognition, solve some problems such as misjudgments and missed judgments, and improve the performance of the system in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0019] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0020] Figure 1 A schematic flow chart of a method for a sprinkler truck to avoid pedestrians based on visual recognition area proportion detection provided in Example 1 of the present invention; Figure 2 A schematic diagram of a specific implementation process of the method for a sprinkler truck to avoid pedestrians based on visual recognition area proportion detection provided in Example 1 of the present invention; Figure 3 A schematic diagram of the vehicle position in the greening watering mode in the method for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection provided in Example 1 of the present invention; Figure 4 It is a schematic diagram of EWR in the greening watering mode in the method for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection provided in Example 1 of the present invention; Figure 5 A schematic diagram of vehicle positions in a road dust removal mode in a method for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection provided in Example 1 of the present invention; Figure 6 It is a schematic diagram of EWR in the road dust removal mode in the method for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection provided in Example 1 of the present invention; Figure 7 This is a schematic diagram of an image in a rectangular coordinate system in the method for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection provided in Example 1 of the present invention; Figure 8 A schematic diagram of the first position situation in the method for a sprinkler truck to avoid pedestrians based on visual recognition area proportion detection provided in Example 1 of the present invention; Fig. 9 A schematic diagram of the second position situation in the method for a sprinkler truck to avoid pedestrians based on visual recognition area proportion detection provided in Example 1 of the present invention; Fig.10 A schematic diagram of the third position situation in the method for a sprinkler truck to avoid pedestrians based on visual recognition area proportion detection provided in Example 1 of the present invention; Fig.11 A schematic diagram of the fourth position of the method for a sprinkler truck to avoid pedestrians based on visual recognition area proportion detection provided in Example 1 of the present invention; Fig.12A schematic diagram of the fifth position of the method for a sprinkler truck to avoid pedestrians based on visual recognition area proportion detection provided in Example 1 of the present invention; Fig.13 A schematic diagram of the installation position of the camera in the method for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection provided in Example 1 of the present invention; Fig.14 Schematic diagram of the architecture of the sprinkler truck pedestrian avoidance device based on visual recognition area proportion detection provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0021] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Example 1

[0023] See also Figure 1 and Figure 2 Embodiment 1 of the present invention provides a method for a sprinkler truck to avoid pedestrians based on visual recognition area proportion detection, comprising the following steps: S1. Based on the Yolov5 target detection algorithm, a target detection model is constructed; the target detection model is trained by setting a data set to obtain the trained target detection model; the trained target detection model is deployed in a sprinkler truck; S2, after the sprinkler truck is started, the camera is started through the GStreamer pipeline; a video stream is obtained through the camera; and a format conversion tool is set to convert the format of the video stream into a BGR format; S3, preprocessing the image in the video stream to obtain an optimized image; S4, inputting the optimized image into the target detection model, performing target object detection through the target detection model, and outputting the position and confidence information of the target object on the optimized image; selecting the target object according to the position and the confidence information, and outputting the target object rectangular frame and the coordinate information of the rectangular frame; S5, performing lane line recognition through the target detection model, and selecting a sprinkler truck working mode according to the recognition result; S6. According to the coordinate information of the rectangular frame, obtain the real-time ratio of the area of ​​the target object rectangular frame entering the EWR area to the area of ​​the target object rectangular frame through real-time calculation; S7, comparing the real-time proportion with a first set threshold, if the real-time proportion is greater than the first set threshold, proceeding to the next step; if the real-time proportion is not greater than the first set threshold, continuing to detect the target object through the target detection model; S8. When the real-time proportion is greater than the first set threshold, a message is sent through a set communication protocol to notify the upper controller to close the hedging in the direction of the target object; S9, comparing the real-time proportion with a second set threshold, if the real-time proportion is less than the second set threshold, proceeding to the next step; if the real-time proportion is not less than the second set threshold, keeping the hedging in the direction of the target object closed; S10: When the real-time proportion is less than the second set threshold, a message is sent through the set communication protocol to notify the upper controller to turn on the hedging in the direction of the target object.

[0024] In this embodiment, in step S1, a target detection model is constructed based on the Yolov5 target detection algorithm; the target detection model is trained by setting a data set to obtain the trained target detection model; the trained target detection model is deployed in the sprinkler truck; Specifically, based on the Yolov5 target detection algorithm, a target detection model is built; first, a high-quality dataset containing target objects needs to be built. The images in the dataset should cover different lighting, weather, angles, and environmental backgrounds to ensure the generalization ability of the target detection model. After the image acquisition is completed, the target objects (pedestrians and lane lines) are labeled using a special annotation tool LabelImg. Then the k-means clustering algorithm is used to calculate the anchors suitable for the model to ensure that the overlap between the anchor and the real frame is higher, thereby improving the detection effect. In the training phase, 4000 rounds of training were performed on the dataset. After the training is completed, a pt model file of the target detection model is generated based on the best performance in the training results. In order to deploy the target detection model in actual applications, the pt model file is converted into a rknn model file compatible with the deployment end.

[0025] In this embodiment, in step S2, after the sprinkler truck is started, the camera is started through the GStreamer pipeline; a video stream is obtained through the camera; and the format of the video stream is converted into the BGR format by setting a format conversion tool; Specifically, after the sprinkler truck is started, the camera is started using the GStreamer pipeline to directly capture the video stream in NV12 format, and then converted to BGR format using opencv.

[0026] In this embodiment, the installation position of the camera and some related parameters are as follows: like Fig.13 As shown in the figure, the camera is installed on both sides of the sprinkler truck, specifically located above the tank body of the sprinkler pipe, slightly outside, about 0.8m away from the center of the tank body of the sprinkler truck (the tank body dimensions (long axis × short axis × length) are: 2200 × 1350 × 4500 (mm)), and the angle is adjusted to cover the boundary line of the spraying area to ensure that all pedestrians and non-motor vehicles within the spraying range can be monitored within the camera's field of view. The specific angle is adjusted according to the height of the vehicle body and the width of the spraying area. After actual on-site testing, the angle α between the center line of the camera and the horizontal line should be set between 13-20° to reduce the blind spot of the field of view.

[0027] In this embodiment, in step S3, the image in the video stream is preprocessed to obtain an optimized image; Specifically, a series of preprocessing operations are performed on each frame of the captured video stream, such as image distortion correction, letterbox scaling adjustment, and sequential conversion of color channels.

[0028] In this embodiment, in step S4, the optimized image is input into the target detection model, the target object is detected by the target detection model, and the position and confidence information of the target object on the optimized image are output; the target object is framed according to the position and the confidence information, and the target object rectangular frame and the coordinate information of the rectangular frame are output; Specifically, the optimized image is input into the target detection model, and the target object is detected by the target detection model, the target object in the image is identified, and key information such as its position and confidence on the image is output. After obtaining this information, further processing is performed to select the identified target object on the image, specifically a rectangular box, and output the four coordinate points of the rectangular box.

[0029] In this embodiment, in step S5, lane line recognition is performed using the target detection model, and the sprinkler truck operating mode is selected according to the recognition result; Specifically, lane line recognition is performed through the target detection model, and the specific mode of the sprinkler truck is determined according to the number of lane lines. At the same time, the mode adaptation of EWR is also realized.

[0030] In this embodiment, under normal circumstances, the sprinkler truck needs to turn on the pedestrian avoidance mode when its working area is in the two outermost lanes, because when the sprinkler truck is working in the middle lane, its flushing area will not spray pedestrians. According to the two outermost lanes where the sprinkler truck works, the pedestrian avoidance mode of the sprinkler truck is divided into two modes, namely the greening irrigation mode (when working in the outermost lane line) and the road dust removal mode (when working in the second to last lane line).

[0031] Specifically, when the greening watering mode is turned on, the vehicle position is as follows: Figure 3 As shown, Figure 3 The angle 128° is the horizontal field of view of the camera, which is a fixed camera parameter. The EWR area at this time is as follows Figure 4 As shown, Figure 4 The angle 125.46° in the figure is the sector angle of the EWR area, which is a manually defined sector angle of the EWR area. The EWR area in this mode is an inscribed sector with the bottom midpoint of the image as the vertex and a radius of 1080.

[0032] When the road dust removal mode is turned on, the vehicle position is as follows Figure 5 As shown in Figure 2, the EWR area at this time is as follows: Figure 6 As shown, the EWR area in this mode is an arc obtained by subtracting the rectangle with a length of 495 in the lower half of the image from the inscribed sector of the image with a radius of 1080, with the lower midpoint of the image as the vertex.

[0033] In this embodiment, in step S6, according to the coordinate information of the rectangular frame, the real-time ratio of the area of ​​the target object rectangular frame entering the EWR area to the area of ​​the target object rectangular frame is obtained through real-time calculation; Specifically, the greening watering mode is similar to the road dust removal mode. Taking the greening watering mode as an example, the 1920*1080 image of the camera is regarded as a rectangle with a length of 1920 and a width of 1080 in the "fourth quadrant" of the rectangular coordinate system (the difference from the fourth quadrant is that as the x and y directions extend, the coordinates increase positively). Taking the EWR in the greening watering mode as an example, Figure 7 shown.

[0034] according to Figure 7 As shown, the equation for the fan-shaped EWR can be written as: ; ; In the formula, x and y are the horizontal and vertical coordinates respectively.

[0035] In this embodiment, five determination calculation strategies are set according to the position of the target object rectangular frame entering the EWR area; First position: Figure 8 As shown, when three or more of the four coordinate points of the target object rectangular frame enter the EWR area, the real-time proportion calculation is not performed, and it is determined that the target object is in the EWR area; Second position: Fig. 9 As shown, when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular box are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not, and satisfy <960, <960 or >960, When >960, the real-time proportion calculation formula is: ; ; ; In the formula, The left horizontal coordinate of the target object rectangle; is the upper ordinate of the target object’s rectangular box; is the right horizontal coordinate of the target object rectangle; is the lower ordinate of the target object’s rectangular box; is the area of ​​the target object rectangle within the EWR region; is a linear function of the two straight lines of the fan; is the total area of ​​the target object rectangle; S is the real-time proportion of the target object rectangle entering the EWR area; Third position: Fig.10 As shown, when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular frame are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not, and <960, When >960, the real-time proportion calculation is not performed, and the target object is determined to be in the EWR area; Fourth position: Fig.11 As shown in FIG. 1 , when the lower left corner coordinates and the lower right corner coordinates of the target object rectangular frame are within the EWR area, but the upper left corner coordinates and the upper right corner coordinates are not within the EWR area, the real-time proportion calculation formula is: ; ; ; In the formula, is the area of ​​the target object rectangle within the EWR region; is a fan-shaped function; Fifth position: Fig.12As shown, when two or more coordinate points of the target object rectangular frame are not in the EWR area, but the center point coordinate is in, the real-time proportion calculation is not performed, and it is determined that the target object is in the EWR area; The calculation formula for the center point coordinates of the target object rectangular frame is: ; ; In the formula, , is the center point coordinate.

[0036] In this embodiment, in step S7, the real-time proportion is compared with a first set threshold value. If the real-time proportion is greater than the first set threshold value, the next step of processing is performed; if the real-time proportion is not greater than the first set threshold value, the target object detection is continued through the target detection model; Specifically, the first set threshold is set to 10%. If the real-time proportion is greater than 10%, it is considered that the target object is already in the EWR area, and the next step of processing is performed; if the real-time proportion is not greater than 10%, it is considered that the target object is not in the EWR area, and the target object detection continues through the target detection model.

[0037] In this embodiment, in step S8, when the real-time proportion is greater than the first set threshold, a message is sent through a set communication protocol to notify the upper controller to close the hedging in the direction of the target object; Specifically, when the real-time proportion is greater than 10%, it is considered that the target object is already in the EWR area, and a message is sent through the CAN communication protocol to notify the upper controller to close the hedging in the direction of the target object.

[0038] In this embodiment, in step S9, the real-time proportion is compared with the second set threshold. If the real-time proportion is less than the second set threshold, the next step is performed; if the real-time proportion is not less than the second set threshold, the hedging in the direction of the target object is kept closed; Specifically, the second set threshold is set to 5%. If the real-time proportion is less than 5%, it is considered that the target object is not in the EWR area or has left, and the next step of processing is performed; if the real-time proportion is not less than 5%, it is considered that the target object is still in the EWR area, and the hedging in the direction of the target object is kept closed.

[0039] In this embodiment, in step S10, when the real-time proportion is less than the second set threshold, a message is sent through the set communication protocol to notify the upper body controller to turn on the hedging in the direction of the target object.

[0040] Specifically, when the real-time proportion is less than 5%, it is considered that the target object is not in the EWR area or has left, and a message is sent through the CAN communication protocol to notify the upper controller to open the hedging in the direction of the target object.

[0041] In summary, the present invention constructs a target detection model based on the Yolov5 target detection algorithm; trains the target detection model by setting a data set to obtain the trained target detection model; deploys the trained target detection model in a sprinkler truck; after the sprinkler truck is started, starts the camera through the GStreamer pipeline; obtains a video stream through the camera; converts the format of the video stream into the BGR format by setting a format conversion tool; preprocesses the image in the video stream to obtain an optimized image; inputs the optimized image into the target detection model, performs target object detection through the target detection model, and outputs the position and confidence information of the target object on the optimized image; selects the target object rectangular frame and the coordinate information of the rectangular frame according to the position and the confidence information; performs lane line recognition through the target detection model, and selects the sprinkler truck working mode according to the recognition result; selects the working mode of the sprinkler truck according to the coordinate information of the rectangular frame The target object detection model is used to detect the target object; when the real-time proportion is greater than the first set threshold, a message is sent through the set communication protocol to notify the upper-mounted controller to close the hedging in the direction where the target object is located; the real-time proportion is compared with the second set threshold, if the real-time proportion is less than the second set threshold, the next step is performed; if the real-time proportion is not less than the second set threshold, the hedging in the direction where the target object is located is kept closed; when the real-time proportion is less than the second set threshold, a message is sent through the set communication protocol to notify the upper-mounted controller to open the hedging in the direction where the target object is located. The present invention can more accurately identify the position of pedestrians through EWR proportion detection, reduce misjudgments, and improve the reliability and accuracy of visual recognition in complex environments. It can effectively optimize the reliability and accuracy of visual recognition, solve some problems such as misjudgments and missed judgments, and improve the performance of the system in complex scenes.

[0042] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0043] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0044] Example 2

[0045] See also Fig.14 Embodiment 2 of the present invention further provides a device for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection, comprising: The target detection model construction, training and deployment module 001 is used to construct a target detection model based on the Yolov5 target detection algorithm; train the target detection model by setting a data set to obtain the trained target detection model; and deploy the trained target detection model in the sprinkler truck; The video stream acquisition and processing module 002 is used to start the camera through the GStreamer pipeline after the sprinkler truck is started; obtain the video stream through the camera; and convert the format of the video stream into BGR format by setting a format conversion tool; An image preprocessing module 003 is used to preprocess the image in the video stream to obtain an optimized image; The target object detection and information marking module 004 is used to input the optimized image into the target detection model, perform target object detection through the target detection model, and output the position and confidence information of the target object on the optimized image; select the target object according to the position and the confidence information, and output the target object rectangular frame and the coordinate information of the rectangular frame; The working mode selection module 005 is used to identify the lane line through the target detection model and select the working mode of the sprinkler truck according to the identification result; A real-time ratio calculation module 006 is used to obtain the real-time ratio of the area of ​​the target object rectangular frame entering the EWR area to the area of ​​the target object rectangular frame through real-time calculation according to the coordinate information of the rectangular frame; A first real-time proportion comparison module 007 is used to compare the real-time proportion with a first set threshold value, and if the real-time proportion is greater than the first set threshold value, the next step of processing is performed; if the real-time proportion is not greater than the first set threshold value, the target object detection is continued through the target detection model; The first control processing module 008 is used to send a message through a set communication protocol to notify the upper controller to close the hedging in the direction of the target object when the real-time proportion is greater than the first set threshold; The second real-time proportion comparison module 009 is used to compare the real-time proportion with the second set threshold value. If the real-time proportion is less than the second set threshold value, the next step of processing is performed; if the real-time proportion is not less than the second set threshold value, the hedging in the direction of the target object is kept closed; The second control processing module 010 is used to send a message through the set communication protocol to notify the upper controller to turn on the hedging in the direction of the target object when the real-time proportion is less than the second set threshold.

[0046] In this embodiment, in the target detection model construction, training and deployment module 001, in the process of training the target detection model through the set data set, the images in the set data set include several target objects; the images in the set data set cover several types of lighting, weather, angles and environmental backgrounds.

[0047] In this embodiment, in the target detection model construction, training and deployment module 001, in the process of deploying the trained target detection model in the sprinkler truck, the trained target detection model is used to generate a pt model file; and the pt model file is converted into a rknn model file.

[0048] In this embodiment, in the image preprocessing module 003, during the process of preprocessing the image in the video stream, the preprocessing includes image distortion correction processing, letterbox scaling adjustment processing, and color channel sequential conversion processing.

[0049] In this embodiment, in the real-time proportion calculation module 006, in the process of obtaining the real-time proportion of the area of ​​the target object rectangular frame entering the EWR area and the area of ​​the target object rectangular frame through real-time calculation, five determination calculation strategies are set according to the position of the target object rectangular frame entering the EWR area; First position situation: when three or more of the four coordinate points of the target object rectangular frame enter the EWR area, the real-time proportion calculation is not performed, and it is determined that the target object is in the EWR area; Second position situation: when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular frame are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not, and satisfy <960, <960 or >960, When >960, the real-time proportion calculation formula is: ; ; ; In the formula, The left horizontal coordinate of the target object rectangle; is the upper ordinate of the target object’s rectangular box; is the right horizontal coordinate of the target object rectangle; is the lower ordinate of the target object’s rectangular box; is the area of ​​the target object rectangle within the EWR region; is a linear function of the two straight lines of the fan; is the total area of ​​the target object rectangle; S is the real-time proportion of the target object rectangle entering the EWR area; Third position situation: when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular frame are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not within the EWR area, and <960, When >960, the real-time proportion calculation is not performed, and the target object is determined to be in the EWR area; Fourth position situation: when the lower left corner coordinates and the lower right corner coordinates of the target object rectangular frame are within the EWR area, but the upper left corner coordinates and the upper right corner coordinates are not within the EWR area, the real-time proportion calculation formula is: ; ; ; In the formula, is the area of ​​the target object rectangle within the EWR region; is a fan-shaped function; Fifth position situation: when two or more coordinate points of the target object rectangular frame are not within the EWR area, but the coordinates of the center point are within the EWR area, the real-time proportion calculation is not performed, and the target object is determined to be within the EWR area; The calculation formula for the center point coordinates of the target object rectangular frame is: ; ; In the formula, , is the center point coordinate.

[0050] It should be noted that the information interaction, execution process and other contents between the modules of the above-mentioned system are based on the same concept as the method embodiment in Example 1 of the present application, and the technical effects they bring are the same as those of the method embodiment of the present application. For specific contents, please refer to the description in the method embodiment shown above in the present application, and will not be repeated here.

[0051] Example 3

[0052] Embodiment 3 of the present invention provides a non-transitory computer-readable storage medium, in which a program code for a method for a sprinkler truck to avoid pedestrians based on visual recognition area ratio detection is stored. The program code includes instructions for executing embodiment 1 or any possible implementation thereof, a method for a sprinkler truck to avoid pedestrians based on visual recognition area ratio detection.

[0053] Computer-readable storage media can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0054] Example 4

[0055] Embodiment 4 of the present invention provides an electronic device, including: a memory and a processor; The processor and the memory communicate with each other via a bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute Example 1 or any possible implementation thereof, a method for a sprinkler truck to avoid pedestrians based on visual recognition area proportion detection.

[0056] Specifically, the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software, the processor can be a general-purpose processor implemented by reading software codes stored in a memory. The memory can be integrated in the processor or can be located outside the processor and exist independently.

[0057] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable systems. The computer instructions can be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0058] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing system, they can be concentrated on a single computing system, or distributed on a network composed of multiple computing systems, and optionally, they can be implemented by a program code executable by a computing system, so that they can be stored in a storage system and executed by the computing system, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0059] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made on the basis of the present invention. Therefore, these modifications or improvements made on the basis of not departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for a water truck to avoid pedestrians based on visual recognition area proportion detection, characterized in that: include: Build a target detection model based on the Yolov5 target detection algorithm; The target detection model is trained by setting a data set to obtain the trained target detection model; and the trained target detection model is deployed in a sprinkler truck; After the sprinkler truck is started, the camera is started through the GStreamer pipeline; the video stream is obtained through the camera; Convert the video format to BGR format by setting a format conversion tool; Preprocessing the image in the video stream to obtain an optimized image; Inputting the optimized image into the target detection model, performing target object detection through the target detection model, and outputting the position and confidence information of the target object on the optimized image; selecting the target object according to the position and the confidence information, and outputting the target object rectangular frame and the coordinate information of the rectangular frame; Performing lane line recognition through the target detection model, and selecting a sprinkler truck working mode according to the recognition result; According to the coordinate information of the rectangular frame, a real-time ratio of the area of ​​the target object rectangular frame entering the EWR area to the area of ​​the target object rectangular frame is obtained through real-time calculation; Compare the real-time proportion with a first set threshold, and if the real-time proportion is greater than the first set threshold, proceed to the next step; If the real-time proportion is not greater than the first set threshold, continue to detect the target object using the target detection model; When the real-time proportion is greater than the first set threshold, a message is sent through a set communication protocol to notify the upper controller to close the hedging in the direction of the target object; Compare the real-time proportion with a second set threshold, and if the real-time proportion is less than the second set threshold, proceed to the next step; If the real-time proportion is not less than the second set threshold, the hedging in the direction of the target object is kept closed; When the real-time proportion is less than the second set threshold, a message is sent through the set communication protocol to notify the upper body controller to open the hedging in the direction where the target object is located.

2. The method for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection according to claim 1 is characterized in that: In the process of training the target detection model through the set data set, the images in the set data set include several target objects; the images in the set data set cover several types of lighting, weather, angles and environmental backgrounds.

3. The method for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection according to claim 2 is characterized in that: In the process of deploying the trained target detection model in the sprinkler truck, the trained target detection model is used to generate a pt model file; and the pt model file is converted into a rknn model file.

4. The method for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection according to claim 3 is characterized in that: In the process of preprocessing the image in the video stream, the preprocessing includes image distortion correction processing, letterbox scaling adjustment processing, and color channel sequence conversion processing.

5. The method for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection according to claim 4 is characterized in that: In the process of obtaining the real-time ratio of the area of ​​the target object rectangular frame entering the EWR area to the area of ​​the target object rectangular frame through real-time calculation, five determination calculation strategies are set according to the position of the target object rectangular frame entering the EWR area; First position situation: when three or more of the four coordinate points of the target object rectangular frame enter the EWR area, the real-time proportion calculation is not performed, and it is determined that the target object is in the EWR area; Second position situation: when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular frame are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not, and satisfy <960, <960 or >960, When >960, the real-time proportion calculation formula is: ； ； ； In the formula, The left horizontal coordinate of the target object rectangle; is the upper ordinate of the target object’s rectangular box; is the right horizontal coordinate of the target object rectangle; is the lower ordinate of the target object’s rectangular box; is the area of ​​the target object rectangle within the EWR region; is a linear function of the two straight lines of the fan; is the total area of ​​the target object rectangle; S is the real-time proportion of the target object rectangle entering the EWR area; Third position situation: when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular frame are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not within the EWR area, and <960, When >960, the real-time proportion calculation is not performed, and the target object is determined to be in the EWR area; Fourth position situation: when the lower left corner coordinates and the lower right corner coordinates of the target object rectangular frame are within the EWR area, but the upper left corner coordinates and the upper right corner coordinates are not within the EWR area, the real-time proportion calculation formula is: ； ； ； In the formula, is the area of ​​the target object rectangle within the EWR region; is a fan-shaped function; Fifth position situation: when two or more coordinate points of the target object rectangular frame are not within the EWR area, but the coordinates of the center point are within the EWR area, the real-time proportion calculation is not performed, and the target object is determined to be within the EWR area; The calculation formula for the center point coordinates of the target object rectangular frame is: ； ； In the formula, , is the center point coordinate.

6. A device for avoiding pedestrians by a watering truck based on the detection of the proportion of visual recognition area, adopting the method for avoiding pedestrians by a watering truck based on the detection of the proportion of visual recognition area as described in any one of claims 1 to 5, characterized in that: include: The target detection model construction, training and deployment module is used to build a target detection model based on the Yolov5 target detection algorithm; The target detection model is trained by setting a data set to obtain the trained target detection model; and the trained target detection model is deployed in a sprinkler truck; The video stream acquisition and processing module is used to start the camera through the GStreamer pipeline after the sprinkler truck is started; and obtain the video stream through the camera; Convert the video format to BGR format by setting a format conversion tool; An image preprocessing module, used for preprocessing the image in the video stream to obtain an optimized image; A target object detection and information marking module, used to input the optimized image into the target detection model, perform target object detection through the target detection model, and output the position and confidence information of the target object on the optimized image; select the target object according to the position and the confidence information, and output the target object rectangular frame and the coordinate information of the rectangular frame; A working mode selection module, used to identify lane lines through the target detection model and select a working mode of the sprinkler truck according to the identification result; A real-time ratio calculation module, used for obtaining the real-time ratio of the area of ​​the target object rectangular frame entering the EWR area to the area of ​​the target object rectangular frame through real-time calculation according to the coordinate information of the rectangular frame; A first real-time proportion comparison module is used to compare the real-time proportion with a first set threshold, and if the real-time proportion is greater than the first set threshold, proceed to the next step; If the real-time proportion is not greater than the first set threshold, continue to detect the target object using the target detection model; A first control processing module, configured to, when the real-time proportion is greater than the first set threshold, send a message through a set communication protocol to notify the upper controller to close the hedging in the direction where the target object is located; A second real-time proportion comparison module is used to compare the real-time proportion with a second set threshold, and if the real-time proportion is less than the second set threshold, proceed to the next step; If the real-time proportion is not less than the second set threshold, the hedging in the direction of the target object is kept closed; The second control processing module is used to send a message through the set communication protocol to notify the upper controller to open the hedging in the direction of the target object when the real-time proportion is less than the second set threshold.

7. The device for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection according to claim 6 is characterized in that: In the target detection model construction, training and deployment module, in the process of training the target detection model through the set data set, the images in the set data set include several target objects; the images in the set data set cover several types of lighting, weather, angles and environmental backgrounds.

8. The device for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection according to claim 7 is characterized in that: In the target detection model construction, training and deployment module, in the process of deploying the trained target detection model in the sprinkler truck, the trained target detection model is used to generate a pt model file; and the pt model file is converted into a rknn model file.

9. The device for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection according to claim 8, characterized in that: In the image preprocessing module, during the process of preprocessing the image in the video stream, the preprocessing includes image distortion correction processing, letterbox scaling adjustment processing, and color channel sequence conversion processing.

10. The device for avoiding pedestrians by a sprinkler truck based on visual recognition area proportion detection according to claim 9, characterized in that: In the real-time proportion calculation module, in the process of obtaining the real-time proportion of the area of ​​the target object rectangular frame entering the EWR area to the area of ​​the target object rectangular frame through real-time calculation, five determination calculation strategies are set according to the position of the target object rectangular frame entering the EWR area; First position situation: when three or more of the four coordinate points of the target object rectangular frame enter the EWR area, the real-time proportion calculation is not performed, and it is determined that the target object is in the EWR area; Second position situation: when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular frame are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not, and satisfy <960, <960 or >960, When >960, the real-time proportion calculation formula is: ； ； ； In the formula, The left horizontal coordinate of the target object rectangle; is the upper ordinate of the target object’s rectangular box; is the right horizontal coordinate of the target object rectangle; is the lower ordinate of the target object’s rectangular box; is the area of ​​the target object rectangle within the EWR region; is a linear function of the two straight lines of the fan; is the total area of ​​the target object rectangle; S is the real-time proportion of the target object rectangle entering the EWR area; Third position situation: when the upper left corner coordinates and the upper right corner coordinates of the target object rectangular frame are within the EWR area, but the lower left corner coordinates and the lower right corner coordinates are not within the EWR area, and <960, When >960, the real-time proportion calculation is not performed, and the target object is determined to be in the EWR area; Fourth position situation: when the lower left corner coordinates and the lower right corner coordinates of the target object rectangular frame are within the EWR area, but the upper left corner coordinates and the upper right corner coordinates are not within the EWR area, the real-time proportion calculation formula is: ； ； ； In the formula, is the area of ​​the target object rectangle within the EWR region; is a fan-shaped function; Fifth position situation: when two or more coordinate points of the target object rectangular frame are not within the EWR area, but the coordinates of the center point are within the EWR area, the real-time proportion calculation is not performed, and the target object is determined to be within the EWR area; The calculation formula for the center point coordinates of the target object rectangular frame is: ； ； In the formula, , is the center point coordinate.

Images