Collision monitoring method and device and storage medium
By converting the two-dimensional image and parallax image of the video frame into three-dimensional point cloud data, and combining the target detection results of the previous frame, predicting the location of the target object that does not exist but may exist in the current frame, a vision-based collision monitoring method is solved. The problem of objects beyond the range of the image frame cannot be detected, improving the accuracy and user experience of collision monitoring.
Patent Information
- Application Number
- CN202310612372.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-11-26
AI Technical Summary
Vision-based collision monitoring methods cannot detect objects that exceed the range of the image frame but are still close to a specific object, resulting in insufficient real-time and accurate collision monitoring.
By acquiring the two-dimensional image and parallax image of the video frame, converting it into three-dimensional point cloud data, and combining the target detection results of the previous frame, predicting the location of the target object that does not exist but may exist in the current frame, thereby performing collision monitoring and early warning .
It improves the accuracy of collision monitoring, can provide early warnings for situations that exceed the range of the image frame but are close to a specific object, and improves the user experience.
Smart Images

Figure CN119027864A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a collision monitoring method, device and computer-readable storage medium. Background Art
[0002] In the application scenario of collision monitoring for objects, the following two technologies can generally be used to implement collision monitoring for movable objects: one is a sensor-based collision monitoring method, which usually uses sensor data to monitor the position of objects and obtain the possibility of collision of objects. However, this method is difficult to provide a visual interface, and usually an alarm is issued only after the collision has occurred. The other is a vision-based collision monitoring method, which detects and locates objects in image frames, and then performs dynamic collision monitoring and outputs results based on the position of the object or the distance between it and other objects.
[0003] However, in vision-based collision monitoring methods, objects that are not currently in the image frame are usually not detected, making it impossible to perform collision monitoring and warnings for situations that are beyond the shooting range of the image frame but are still close to specific objects, resulting in the collision monitoring method not being real-time and accurate enough.
[0004] Therefore, a collision monitoring method is needed to improve the collision monitoring accuracy and user experience. Summary of the invention
[0005] To solve the above technical problems, according to one aspect of the present invention, a collision monitoring method is provided, comprising: obtaining a two-dimensional image of a current frame in a video and performing target detection to obtain a detected two-dimensional target object; obtaining a disparity image of the current frame at the same time as the two-dimensional image and converting it into three-dimensional point cloud data; obtaining a position of the detected three-dimensional target object at the current frame time based on the detected two-dimensional target object and the three-dimensional point cloud data, and at least based on the target detection results of the three-dimensional target objects in one or more previous frames in the video, wherein, when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame time is obtained based on the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame time is obtained; collision monitoring is performed based on the position of the three-dimensional target object at the current frame time, and the collision possibility of the three-dimensional target object is output.
[0006] According to another aspect of the present invention, a collision monitoring device is provided, comprising: a detection unit, configured to obtain a two-dimensional image of a current frame in a video and perform target detection to obtain a detected two-dimensional target object; a conversion unit, configured to obtain a disparity image of the current frame at the same time as the two-dimensional image and convert it into three-dimensional point cloud data; a positioning unit, configured to obtain a position of the detected three-dimensional target object at the current frame time based on the detected two-dimensional target object and the three-dimensional point cloud data, and at least based on the target detection results of the three-dimensional target object in one or more previous frames in the video, wherein when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame time is obtained based on the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame time is obtained; a monitoring unit, configured to perform collision monitoring based on the position of the three-dimensional target object at the current frame time, and output the collision possibility of the three-dimensional target object.
[0007] According to another aspect of the present invention, a collision monitoring device is provided, comprising: a processor and a memory, wherein computer program instructions are stored in the memory, wherein when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: acquiring a two-dimensional image of a current frame in a video and performing target detection to acquire a detected two-dimensional target object; acquiring a disparity image of the current frame at the same time as the two-dimensional image and converting it into three-dimensional point cloud data; acquiring a position of the detected three-dimensional target object at the current frame time based on the detected two-dimensional target object and the three-dimensional point cloud data, and at least based on the target detection results of the three-dimensional target objects in one or more previous frames in the video, wherein when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame time is acquired based on the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame time is acquired; collision monitoring is performed based on the position of the three-dimensional target object at the current frame time, and the collision possibility of the three-dimensional target object is output.
[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: acquiring a two-dimensional image of a current frame in a video and performing target detection to acquire a detected two-dimensional target object; acquiring a disparity image of the current frame at the same time as the two-dimensional image and converting it into three-dimensional point cloud data; acquiring a position of the detected three-dimensional target object at the current frame time based on the detected two-dimensional target object and the three-dimensional point cloud data, and at least based on target detection results of three-dimensional target objects in one or more previous frames in the video, wherein, when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame time is acquired based on the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame time is acquired; collision monitoring is performed based on the position of the three-dimensional target object at the current frame time, and the collision possibility of the three-dimensional target object is output.
[0009] According to the above-mentioned collision monitoring method, device and computer-readable storage medium of the present invention, it is possible to predict the movement direction and position of the target object that only exists in the previous frame but not in the current frame by combining the results of target object detection for the current frame in the video and the target object detection for the previous frame, so as to perform collision monitoring and early warning for the situation that exceeds the shooting range of the current frame but is still close to the specific object according to the prediction results, thereby improving the accuracy of collision monitoring and the user experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The above and other objects, features and advantages of the present invention will become more apparent by describing in detail the embodiments of the present invention in conjunction with the accompanying drawings.
[0011] Figure 1 A schematic diagram of collision monitoring according to an example application scenario of an embodiment of the present invention is shown, wherein Figure 1 (a) shows a situation where there is a safe distance between the monitored objects; Figure 1 (b) showing situations where early warning is required between monitored objects; Figure 1 (c) showing the emergency warning situation between monitored objects; Figure 1 (d) indicates that the monitored objects have collided with each other;
[0012] Figure 2 It shows the situation that some of the target objects to be monitored do not exist in the captured image, where Figure 2 (a) shows the situation where the hook of the tower crane is not detected in the captured image. Figure 2(b) shows the situation where the high tower crane trolley is not detected in the captured image;
[0013] Figure 3 A flowchart showing a collision monitoring method according to an embodiment of the present invention is shown;
[0014] Figure 4 A block diagram showing a collision monitoring device according to an embodiment of the present invention;
[0015] Figure 5 A block diagram of a collision monitoring device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0016] The following will describe a collision monitoring method, device and computer readable storage medium according to an embodiment of the present invention with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements from beginning to end. It should be understood that the embodiments described herein are merely illustrative and should not be interpreted as limiting the scope of the present invention.
[0017] In the application scenario of collision monitoring, a vision-based collision monitoring method can usually be used for collision monitoring. This monitoring method can provide early warning of possible collisions between monitored objects to effectively avoid collisions. This collision monitoring method can generally be applied to the tower anti-collision system in construction sites, or can be applied to anti-collision monitoring during vehicle driving, or other scenarios that require distance measurement or collision monitoring.
[0018] Figure 1 A schematic diagram of collision monitoring according to an example application scenario of an embodiment of the present invention is shown. Figure 1 In FIG. 1 , an example of an application scenario of an embodiment of the present invention in a construction site is shown. Figure 1 As shown in (a)-(d), each rectangular box respectively frames each target object in the tower crane construction at the construction site. Specifically, the rectangular boxes respectively show the high tower crane boom (the upper left rectangular box in the figure), the high tower crane trolley (the rectangular box on the high tower crane boom), the high tower crane hook (the rectangular box under the high tower crane trolley) and the low tower crane boom (the rectangular box on the right side of the figure), wherein the line between the high tower crane trolley and the high tower crane hook represents the high tower crane rope. Figure 1 The four figures (a)-(d) represent the situations when the distance between the high tower crane rope and the low tower crane boom is different. Figure 1 (a) shows that the rope of the high tower crane is far away from the boom of the low tower crane, and the two maintain a safe distance; Figure 1 (b) shows that the distance between the high tower crane rope and the low tower crane boom is too close, and the two do not maintain a safe distance, which requires early warning; Figure 1(c) An emergency warning indicating that the distance between the rope of the high tower crane and the boom of the low tower crane is too close and the two are about to collide; Figure 1 (d) shows that the rope of the high tower crane has collided with the boom of the low tower crane.
[0019] exist Figure 1 In the application scenario of the construction site shown, due to the limitation of the shooting angle or shooting range, there may be some target objects that need to be subjected to collision monitoring and distance calculation that cannot all be present in the captured image. Figure 2 It shows the situation that some of the target objects to be monitored do not exist in the captured image. Specifically, Figure 2 (a) shows the situation where the hook of the tower crane is not detected in the captured image. Figure 2 (b) shows the situation where the tower crane trolley is not detected in the captured image. Figure 2 In the application scenario shown, although the high tower crane hook or high tower crane trolley does not exist in the image, it is still very close to the low tower crane boom and is very likely to collide. Therefore, for this scenario where the target object cannot be detected at present, and thus the distance measurement or collision monitoring cannot be performed based on the target object detected in the current image, a more robust collision monitoring method and device are needed, so that when the object cannot be detected in the current image, the position of the current target object can be predicted by the target detection result of the previous image, thereby obtaining a more accurate distance measurement and collision monitoring effect, so that the method and device in this embodiment have stronger robustness and can adapt to a wider range of applications.
[0020] certainly, Figure 1 and Figure 2 This is only an example of an application scenario of the embodiment of the present invention on a construction site. In practical applications, the embodiment of the present invention can also be applied to various scenarios that require distance measurement and collision monitoring. For example, the embodiment of the present invention can also be used for distance measurement and collision monitoring applications for other vehicles, buildings, pedestrians and other objects in the vehicle's surrounding environment when the vehicle is driving. The various application scenarios of the embodiment of the present invention are not limited here.
[0021] Figure 3 FIG. 3 is a flow chart of a collision monitoring method 300 according to an embodiment of the present invention. Figure 3 A collision monitoring method according to an embodiment of the present invention is described.
[0022] In step S301, a two-dimensional image of a current frame in a video is acquired and target detection is performed to acquire a detected two-dimensional target object.
[0023] In this step, various types of cameras may be used to obtain a two-dimensional image of the current frame in the video. Alternatively, a stereo camera, such as a binocular camera, may be used to obtain a two-dimensional image of the current frame in the video. Alternatively, other types of cameras may be used to obtain the two-dimensional image, which is not limited here.
[0024] After acquiring the two-dimensional image of the current frame, optionally, the target detection convolutional neural network CNN can be used to perform target detection on the two-dimensional image of the current frame to obtain the detected two-dimensional target object. Specifically, the convolutional neural network can be used to obtain the object in the two-dimensional image of the current frame that is desired to be measured for distance and collision warning as a two-dimensional target object, and the type of the detected two-dimensional target object can be further obtained for measurement and calculation of the subsequent collision monitoring method. Optionally, the two-dimensional target object detected in the current frame in the video can be one or more, and accordingly, the type of the two-dimensional target object can also be one or more, which is not limited here. In the subsequent collision monitoring and distance measurement process, collision monitoring and distance measurement can be performed between the multiple two-dimensional target objects detected here, or collision monitoring and distance measurement can be performed between the one or more two-dimensional target objects and other predetermined objects or positions, which depends on the specific application scenario and the implementation of the method.
[0025] In one example, for a construction site scene, the target object detected in the two-dimensional image of the current frame may be one or more two-dimensional target objects such as a high tower crane boom, a high tower crane trolley, a high tower crane hook, and a low tower crane boom. These two-dimensional target objects can be detected by, for example Figure 1 or Figure 2 Correspondingly, the types of the two-dimensional target objects can also be divided into one or more of a high tower crane boom, a high tower crane trolley, a high tower crane hook, and a low tower crane boom.
[0026] In addition, in another example, for a vehicle collision monitoring scene, the target object detected in the two-dimensional graphics of the current frame can be a vehicle, a pedestrian, a building, a guardrail, a road line, or other two-dimensional target objects for vehicle collision monitoring, and can also be represented by a rectangular frame in a similar manner. Accordingly, the types of the two-dimensional target objects can also be divided into one or more of vehicles, buildings, pedestrians, guardrails, road lines, etc., without limitation here.
[0027] In step S302, a disparity image of the current frame at the same time as the two-dimensional image is obtained and converted into three-dimensional point cloud data.
[0028] In this step, the disparity image of the current frame at the same time as the two-dimensional image can be first obtained, and then the depth image of the current frame can be calculated based on the disparity image of the current frame. Optionally, the disparity image of the current frame can be calculated by a left image and a right image obtained by a stereo camera, such as a binocular camera. After obtaining the disparity image, the corresponding depth value can be calculated for each pixel point based on the disparity image of the current frame to obtain the depth image of the current frame.
[0029] After obtaining the depth image of the current frame, the 3D point cloud data of the current frame can be obtained according to the depth image of the current frame in combination with camera parameters including, for example, the focal length of the camera. The 3D point cloud data may include the spatial coordinates of each 3D point, etc., which may show detailed geometric information of each object in the image.
[0030] In step S303, the position of the detected three-dimensional target object at the current frame is obtained based on the detected two-dimensional target object and the three-dimensional point cloud data, and at least based on the target detection results of the three-dimensional target object in one or more previous frames in the video, wherein, when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame is obtained based on the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame is obtained.
[0031] In this step, optionally, when the three-dimensional target object is detected to exist in the current frame, the coordinates of the three-dimensional points corresponding to the two-dimensional target object in the three-dimensional point cloud data can be obtained based on the two-dimensional target object detected in the current frame and the three-dimensional point cloud data, expressed as the corresponding three-dimensional target object, and the position of the three-dimensional target object at the current frame time can be obtained based on this.
[0032] In one example, optionally, the target detection results of the three-dimensional target object of one or more previous frames of the video can also be first obtained. The target detection process of the previous frame can be the same or similar to the steps of target detection and acquisition of the three-dimensional target object of the current frame. For example, the two-dimensional images of one or more previous frames in the video can also be obtained, and the target detection convolutional neural network can be used to perform target detection to obtain the two-dimensional target objects detected in the one or more previous frames; then the disparity images of the one or more previous frames at the same time as the two-dimensional images can be obtained and converted into three-dimensional point cloud data; finally, the three-dimensional target objects of the one or more previous frames can be detected according to the two-dimensional target objects detected in the one or more previous frames and the three-dimensional point cloud data, as the target detection results of the three-dimensional target objects of one or more previous frames in the video.
[0033] After acquiring the three-dimensional target object detected in the previous frame, the three-dimensional target object detected in the current frame can also be compared with the three-dimensional target object detected in the previous frame. For example, the types of the three-dimensional target objects detected in the current frame and the previous frame can be compared to see whether there are one or more three-dimensional target objects detected in the previous frame that do not exist in the current frame. If it is determined that there are one or more three-dimensional target objects detected in the previous frame that do not exist in the current frame, or further do not exist in the next frame of the current frame, then the position of the three-dimensional target object detected in the previous frame at the time of the current frame can be predicted using, for example, a long short-term memory network LSTM, to obtain the predicted position of the three-dimensional target object at the time of the current frame as the position of the three-dimensional target object at the time of the current frame. Among them, LSTM can predict the position of the target object in the next frame by inputting a series of time series data. For example, for the LSTM structure at time t, the position of the two-dimensional target object at time t-1 (i.e., the moment of obtaining the previous frame immediately before the current frame) can be used to predict the position at time t (i.e., the moment of obtaining the current frame) to obtain the two-dimensional target object predicted at the current frame at time t. In addition, for the structures in the LSTM at other moments, a similar method will be used to predict the target object in the next frame. Optionally, since the three-dimensional target object detected in the previous frame does not exist in the current frame, the position of the three-dimensional target object predicted by, for example, the LSTM network at the current frame moment does not exist within the field of view of the current frame. In this case, a corresponding coordinate system can be established based on the field of view of the current frame, and the position of the three-dimensional target object at the current frame moment is represented by the coordinate points or coordinate range in the coordinate system.
[0034] In addition, optionally, due to reasons such as the quality of the acquired image, noise may exist in the acquired three-dimensional point cloud data. In this regard, in the process of acquiring the three-dimensional target object and its position, the three-dimensional point cloud data can be denoised by, for example, clustering, and the three-dimensional target object and its position at the current frame time can be obtained based on the denoised three-dimensional point cloud data.
[0035] In step S304, collision monitoring is performed according to the position of the three-dimensional target object at the current frame time, and the collision possibility of the three-dimensional target object is output.
[0036] Optionally, when there are multiple three-dimensional target objects, collision monitoring can be performed by determining the distance between each three-dimensional target object. To this end, the distance between the position of the three-dimensional target object at the current frame time and another three-dimensional target object can be determined first; then, collision monitoring can be performed based on the distance, and the collision possibility of the three-dimensional target object can be output as a monitoring result.
[0037] In addition, optionally, when there is one or several preset specific objects or specific positions inside or outside the image of the current frame, and the collision monitoring method of this embodiment needs to monitor the distance between the above-mentioned three-dimensional target object and the one or more preset specific objects or positions, then the distance between the position of the three-dimensional target object at the current frame moment and one or more preset specific objects or positions can be first determined; then collision monitoring can be performed based on the distance, and the monitoring results can be output.
[0038] During the monitoring process, optionally, the distance between the three-dimensional target object and other objects or positions can be calculated in various ways. For example, the distance between the center or center of gravity of the three-dimensional target object and other objects or positions can be calculated. For another example, the shortest distance value or average distance value can be obtained by traversing the distances between each component structure constituting the three-dimensional target object (such as each small cube divided by the three-dimensional target object) and other objects or positions, as the distance for collision monitoring.
[0039] After calculating the distance to be monitored, the distance can be compared with a specific threshold to obtain the collision possibility of the three-dimensional target object. For example, the distance can be compared with a safety threshold to determine whether the three-dimensional target object is currently safe. For another example, the distance can also be compared with a collision threshold to determine whether the three-dimensional target object collides with other objects. The safety threshold and the collision threshold can be set and changed according to the application scenario and actual needs. Optionally, in one example, the safety threshold can be 4m and the collision threshold can be 1.8m. In addition, different collision monitoring results can be output according to different comparison results with the safety threshold and / or the collision threshold. For example, the collision monitoring result can be represented by multiple states such as safety (greater than or equal to the safety threshold)-low collision possibility, warning (less than the safety threshold but greater than or equal to the collision threshold)-high collision possibility, collision (less than the collision threshold)-collision has occurred. Of course, the above-mentioned collision monitoring judgment method and monitoring results are only examples, and the actual various parameter settings and monitoring result representations depend on the needs of the actual application scenario, which are not limited here.
[0040] According to the above-mentioned collision monitoring method of an embodiment of the present invention, it is possible to predict the movement direction and position of a target object that only exists in the previous frame but not in the current frame by combining the results of target object detection for the current frame in the video and the target object detection for the previous frame, thereby performing collision monitoring and early warning for situations that are beyond the shooting range of the current frame but are still close to a specific object based on the prediction results, thereby improving the accuracy of collision monitoring and the user experience.
[0041] Below, refer to Figure 4 A collision monitoring device according to an embodiment of the present invention will be described. Figure 4 FIG. 4 is a block diagram of a collision monitoring device 400 according to an embodiment of the present invention. Figure 4 As shown, the collision monitoring device 400 includes a detection unit 410, a conversion unit 420, a positioning unit 430 and a monitoring unit 440. In addition to these units, the collision monitoring device 400 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present invention, their illustration and description are omitted here. In addition, since the specific details of the following operations performed by the collision monitoring device 400 according to the embodiment of the present invention are the same as those described above with reference to Figure 3 The details described are the same, so repeated description of the same details is omitted here to avoid repetition.
[0042] Figure 4 The detection unit 410 of the collision monitoring device 400 obtains a two-dimensional image of a current frame in the video and performs target detection to obtain a detected two-dimensional target object.
[0043] The detection unit 410 can obtain the two-dimensional image of the current frame in the video through various types of cameras. Optionally, a stereo camera, such as a binocular camera, can be used to obtain the two-dimensional image of the current frame in the video. Alternatively, other types of cameras can also be used to obtain the two-dimensional image, which is not limited here.
[0044] After acquiring the two-dimensional image of the current frame, optionally, the detection unit 410 can use the target detection convolutional neural network CNN to perform target detection on the two-dimensional image of the current frame to obtain the detected two-dimensional target object. Specifically, the convolutional neural network can be used to obtain the object in the two-dimensional image of the current frame that is desired to be measured for distance and collision warning as a two-dimensional target object, and the type of the detected two-dimensional target object can be further obtained for measurement and calculation of the subsequent collision monitoring method. Optionally, the two-dimensional target object detected in the current frame in the video can be one or more, and accordingly, the type of the two-dimensional target object can also be one or more, which is not limited here. In the subsequent collision monitoring and distance measurement process, collision monitoring and distance measurement can be performed between the multiple two-dimensional target objects detected here, or collision monitoring and distance measurement can be performed between the one or more two-dimensional target objects and other predetermined objects or positions, which depends on the specific application scenario and the implementation of the method.
[0045] In one example, for a construction site scene, the target object detected in the two-dimensional image of the current frame may be one or more two-dimensional target objects such as a high tower crane boom, a high tower crane trolley, a high tower crane hook, and a low tower crane boom. These two-dimensional target objects can be detected by, for example Figure 1 or Figure 2 Correspondingly, the types of the two-dimensional target objects can also be divided into one or more of a high tower crane boom, a high tower crane trolley, a high tower crane hook, and a low tower crane boom.
[0046] In addition, in another example, for a vehicle collision monitoring scene, the target object detected in the two-dimensional graphics of the current frame can be a vehicle, a pedestrian, a building, a guardrail, a road line, or other two-dimensional target objects for vehicle collision monitoring, and can also be represented by a rectangular frame in a similar manner. Accordingly, the types of the two-dimensional target objects can also be divided into one or more of vehicles, buildings, pedestrians, guardrails, road lines, etc., without limitation here.
[0047] The conversion unit 420 obtains the disparity image of the current frame at the same time as the two-dimensional image and converts it into three-dimensional point cloud data.
[0048] The conversion unit 420 may first obtain a disparity image of the current frame at the same time as the two-dimensional image, and then calculate the depth image of the current frame based on the disparity image of the current frame. Optionally, the disparity image of the current frame may be calculated using a left image and a right image obtained by a stereo camera, such as a binocular camera. After obtaining the disparity image, a corresponding depth value may be calculated for each pixel point based on the disparity image of the current frame to obtain a depth image of the current frame.
[0049] After obtaining the depth image of the current frame, the conversion unit 420 can obtain the three-dimensional point cloud data of the current frame according to the depth image of the current frame in combination with camera parameters including, for example, the focal length of the camera. The three-dimensional point cloud data may include the spatial coordinates of each three-dimensional point, etc., which can show the detailed geometric information of each object in the image.
[0050] The positioning unit 430 obtains the position of the detected three-dimensional target object at the current frame time based on the detected two-dimensional target object and the three-dimensional point cloud data, and at least based on the target detection results of the three-dimensional target object in one or more previous frames in the video, wherein, when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame time is obtained based on the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame time is obtained.
[0051] Optionally, when the positioning unit 430 detects that the three-dimensional target object exists in the current frame, it can obtain the coordinates of the three-dimensional point corresponding to the two-dimensional target object in the three-dimensional point cloud data based on the two-dimensional target object detected in the current frame and the three-dimensional point cloud data, express it as the corresponding three-dimensional target object, and based on this, the position of the three-dimensional target object at the current frame moment can be obtained.
[0052] In one example, optionally, the positioning unit 430 may also first obtain the target detection results of the three-dimensional target object of one or more previous frames of the video. The target detection process of the previous frame may be the same or similar to the steps of target detection and acquisition of the three-dimensional target object of the current frame. For example, the two-dimensional images of one or more previous frames in the video may be obtained in the same manner, and the target detection convolutional neural network may be used to perform target detection to obtain the two-dimensional target objects detected in the one or more previous frames; then, the disparity images of the one or more previous frames at the same time as the two-dimensional images may be obtained and converted into three-dimensional point cloud data; finally, the three-dimensional target objects of the one or more previous frames may be detected based on the two-dimensional target objects detected in the one or more previous frames and the three-dimensional point cloud data, as the target detection results of the three-dimensional target objects of one or more previous frames in the video.
[0053] After acquiring the three-dimensional target object detected in the previous frame, the positioning unit 430 can also compare the three-dimensional target object detected in the current frame with the three-dimensional target object detected in the previous frame. For example, the types of the three-dimensional target objects detected in the current frame and the previous frame can be compared to determine whether there are one or more three-dimensional target objects detected in the previous frame that do not exist in the current frame. If it is determined that there are one or more three-dimensional target objects detected in the previous frame that do not exist in the current frame, or further do not exist in the next frame of the current frame, then the position of the three-dimensional target object detected in the previous frame at the time of the current frame can be predicted by using, for example, a long short-term memory network LSTM, and the predicted position of the three-dimensional target object at the time of the current frame is obtained as the position of the three-dimensional target object at the time of the current frame. Among them, LSTM can predict the position of the target object in the next frame by inputting a series of time series data. For example, for the LSTM structure at time t, the position of the two-dimensional target object at time t-1 (i.e., the moment of obtaining the previous frame immediately before the current frame) can be predicted to obtain the two-dimensional target object predicted at the current frame at time t. In addition, for the structures in the LSTM at other moments, a similar method will be used to predict the target object in the next frame. Optionally, since the three-dimensional target object detected in the previous frame does not exist in the current frame, the position of the three-dimensional target object predicted by, for example, the LSTM network at the current frame moment does not exist within the field of view of the current frame. In this case, a corresponding coordinate system can be established based on the field of view of the current frame, and the position of the three-dimensional target object at the current frame moment is represented by the coordinate points or coordinate range in the coordinate system.
[0054] In addition, optionally, due to reasons such as the quality of the acquired image, noise may exist in the acquired three-dimensional point cloud data. In this regard, in the process of acquiring the three-dimensional target object and its position, the three-dimensional point cloud data can be denoised by, for example, clustering, and the three-dimensional target object and its position at the current frame time can be obtained based on the denoised three-dimensional point cloud data.
[0055] The monitoring unit 440 performs collision monitoring according to the position of the three-dimensional target object at the current frame time, and outputs the collision possibility of the three-dimensional target object.
[0056] Optionally, when there are multiple three-dimensional target objects, the monitoring unit 440 can perform collision monitoring by determining the distance between each three-dimensional target object. To this end, the distance between the position of the three-dimensional target object at the current frame time and another three-dimensional target object can be determined first; then, collision monitoring can be performed based on the distance, and the collision possibility of the three-dimensional target object can be output as a monitoring result.
[0057] In addition, optionally, when there is one or several preset specific objects or specific positions inside or outside the image of the current frame, and the present embodiment needs to monitor the distance between the above-mentioned three-dimensional target object and the one or more preset specific objects or positions, then the distance between the position of the three-dimensional target object at the current frame moment and the one or more preset specific objects or positions can be first determined; then collision monitoring can be performed based on the distance, and the monitoring results can be output.
[0058] During the monitoring process, optionally, the distance between the three-dimensional target object and other objects or positions can be calculated in various ways. For example, the distance between the center or center of gravity of the three-dimensional target object and other objects or positions can be calculated. For another example, the shortest distance value or average distance value can be obtained by traversing the distances between each component structure constituting the three-dimensional target object (such as each small cube divided by the three-dimensional target object) and other objects or positions, as the distance for collision monitoring.
[0059] After calculating the distance to be monitored, the distance can be compared with a specific threshold to obtain the collision possibility of the three-dimensional target object. For example, the distance can be compared with a safety threshold to determine whether the three-dimensional target object is currently safe. For another example, the distance can also be compared with a collision threshold to determine whether the three-dimensional target object collides with other objects. The safety threshold and the collision threshold can be set and changed according to the application scenario and actual needs. Optionally, in one example, the safety threshold can be 4m and the collision threshold can be 1.8m. In addition, different collision monitoring results can be output according to different comparison results with the safety threshold and / or the collision threshold. For example, the collision monitoring result can be represented by multiple states such as safety (greater than or equal to the safety threshold)-low collision possibility, warning (less than the safety threshold but greater than or equal to the collision threshold)-high collision possibility, collision (less than the collision threshold)-collision has occurred. Of course, the above-mentioned collision monitoring judgment method and monitoring results are only examples, and the actual various parameter settings and monitoring result representations depend on the needs of the actual application scenario, which are not limited here.
[0060] The above-mentioned collision monitoring device according to the embodiment of the present invention can predict the movement direction and position of the target object that only exists in the previous frame but not in the current frame by combining the results of the target object detection for the current frame in the video and the target object detection for the previous frame, thereby performing collision monitoring and early warning for situations that are beyond the shooting range of the current frame but are still close to a specific object based on the prediction results, thereby improving the accuracy of collision monitoring and the user experience.
[0061] Below, refer to Figure 5 A collision monitoring device according to an embodiment of the present invention will be described. Figure 5 FIG. 5 shows a block diagram of a collision monitoring device 500 according to an embodiment of the present invention. Figure 5 As shown, the device 500 may be a computer or a server.
[0062] like Figure 5 As shown, the collision monitoring device 500 includes one or more processors 510 and a memory 520. Of course, in addition to this, the collision monitoring device 500 may also include an input device, an output device (not shown), etc. These components may be interconnected via a bus system and / or other forms of connection mechanisms. It should be noted that Figure 5 The components and structures of the collision monitoring device 500 shown are merely exemplary and not restrictive. The collision monitoring device 500 may also have other components and structures as required.
[0063] The processor 510 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may utilize computer program instructions stored in the memory 520 to perform desired functions, which may include: acquiring a two-dimensional image of a current frame in a video and performing target detection to acquire a detected two-dimensional target object; acquiring a disparity image of the current frame at the same time as the two-dimensional image and converting it into three-dimensional point cloud data; acquiring a position of the detected three-dimensional target object at the current frame time based on the detected two-dimensional target object and the three-dimensional point cloud data, and at least based on the target detection results of the three-dimensional target object in one or more previous frames in the video, wherein, when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame time is acquired based on the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame time is acquired; collision monitoring is performed based on the position of the three-dimensional target object at the current frame time, and the collision possibility of the three-dimensional target object is output.
[0064] The memory 520 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 510 may run the program instructions to implement the functions of the collision monitoring device of the embodiment of the present invention described above and / or other desired functions, and / or may execute the collision monitoring method according to the embodiment of the present invention. Various applications and various data may also be stored in the computer-readable storage medium.
[0065] The following describes a computer-readable storage medium according to an embodiment of the present invention, on which computer program instructions are stored, wherein the computer program instructions, when executed by a processor, implement the following steps: acquiring a two-dimensional image of a current frame in a video and performing target detection to acquire a detected two-dimensional target object; acquiring a disparity image of the current frame at the same time as the two-dimensional image and converting it into three-dimensional point cloud data; acquiring a position of the detected three-dimensional target object at the current frame time based on the detected two-dimensional target object and the three-dimensional point cloud data, and at least based on target detection results of three-dimensional target objects in one or more previous frames in the video, wherein when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame time is acquired based on the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame time is acquired; collision monitoring is performed based on the position of the three-dimensional target object at the current frame time, and the collision possibility of the three-dimensional target object is output.
[0066] Of course, the above-mentioned specific embodiments are merely examples rather than limitations, and those skilled in the art can, based on the concept of the present invention, merge and combine some steps and devices from the various embodiments described separately above to achieve the effects of the present invention. Such merged and combined embodiments are also included in the present invention, and such merges and combinations are not described one by one herein.
[0067] Note that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above invention are only for the purpose of illustration and facilitating understanding, not limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.
[0068] The block diagrams of the devices, apparatuses, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.
[0069] The step flow charts and the above method descriptions in the present invention are only illustrative examples and are not intended to require or imply that the steps of each embodiment must be performed in the order given. As will be appreciated by those skilled in the art, the order of the steps in the above embodiments can be performed in any order. Words such as "thereafter", "then", "next", etc. are not intended to limit the order of the steps; these words are only used to guide the reader through the description of these methods. In addition, any reference to a singular element, such as using the article "a", "an", or "the", is not to be construed as limiting the element to the singular.
[0070] In addition, the steps and devices in the various embodiments of this document are not limited to being implemented in a certain embodiment. In fact, based on the concept of the present invention, relevant partial steps and partial devices in the various embodiments of this document can be combined to conceive new embodiments, and these new embodiments are also included in the scope of the present invention.
[0071] Each operation of the method described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors.
[0072] The various illustrated logic blocks, modules and circuits described may be implemented or performed using a general purpose processor, digital signal processor (DSP), ASIC, field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but as an alternative, the processor may be any commercially available processor, controller, microcontroller or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors cooperating with a DSP core, or any other such configuration.
[0073] The steps of the method or algorithm described in conjunction with the present invention can be directly embedded in hardware, in a software module executed by a processor, or in a combination of the two. A software module can exist in any form of tangible storage medium. Some examples of storage media that can be used include random access memory (RAM), read-only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, etc. A storage medium can be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative manner, the storage medium can be integral with the processor. A software module can be a single instruction or many instructions, and can be distributed on several different code segments, between different programs, and across multiple storage media.
[0074] The method invented herein includes one or more actions for implementing the described method. The method and / or action can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions can be modified without departing from the scope of the claims.
[0075] The functions described can be implemented by hardware, software, firmware or any combination thereof. If implemented in software, the functions can be stored as one or more instructions on a tangible computer-readable medium. The storage medium can be any available tangible medium that can be accessed by a computer. By way of example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device or any other tangible medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, a disc includes a compact disc (CD), a laser disc, an optical disc, a digital versatile disc (DVD), a floppy disk and a blue disc.
[0076] Thus, a computer program product may perform the operations presented herein. For example, such a computer program product may be a computer-readable tangible medium having instructions tangibly stored (and / or encoded) thereon, which instructions may be executed by one or more processors to perform the operations described herein. The computer program product may include packaging materials.
[0077] Software or instructions may also be transmitted via a transmission medium. For example, the software may be transmitted from a website, server or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technology such as infrared, radio or microwave.
[0078] In addition, the module and / or other appropriate means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by the user terminal and / or base station when appropriate. For example, such a device can be coupled to a server to facilitate the transmission of the means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage component (e.g., RAM, ROM, a physical storage medium such as a CD or a floppy disk, etc.) so that the user terminal and / or base station can obtain the various methods when being coupled to the device or providing a storage component to the device. In addition, any other appropriate technology for providing the methods and techniques described herein to a device can be utilized.
[0079] Other examples and implementations are within the scope and spirit of the present invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hard wiring, or any combination of these. Features that implement the functions can also be physically located in various locations, including being distributed so that parts of the functions are implemented at different physical locations. Moreover, as used herein, including as used in the claims, "or" used in the enumeration of items beginning with "at least one" indicates a separate enumeration, so that, for example, the enumeration of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). In addition, the wording "exemplary" does not mean that the example described is preferred or better than other examples.
[0080] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present invention is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.
[0081] The above description of the invented aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features of the present invention.
[0082] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the form invented herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A collision monitoring method, comprising: Obtain a two-dimensional image of the current frame in the video and perform target detection to obtain the detected two-dimensional target object; Obtaining a disparity image of the current frame at the same time as the two-dimensional image and converting it into three-dimensional point cloud data; According to the detected two-dimensional target object and the three-dimensional point cloud data, and at least according to the target detection results of the three-dimensional target object in one or more previous frames in the video, the position of the detected three-dimensional target object at the current frame time is obtained, wherein when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame time is obtained according to the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame time is obtained; Collision monitoring is performed according to the position of the three-dimensional target object at the current frame time, and the collision possibility of the three-dimensional target object is output.
2. The method of claim 1, wherein: Obtaining a two-dimensional image of the current frame in a video and performing target detection includes: A convolutional neural network is used to perform target detection on the two-dimensional image of the current frame to obtain the detected two-dimensional target object.
3. The method of claim 1, wherein: Acquiring the disparity image of the current frame at the same time as the two-dimensional image and converting it into three-dimensional point cloud data includes: Calculating a depth image of the current frame according to the disparity image of the current frame; Acquire three-dimensional point cloud data of the current frame according to the depth image of the current frame.
4. The method of claim 1, wherein: The method further comprises: Acquire two-dimensional images of one or more previous frames in the video and perform target detection to obtain the detected two-dimensional target objects in the one or more previous frames; Acquire the disparity images of the one or more previous frames at the same time as the two-dimensional image and convert them into three-dimensional point cloud data; According to the detected two-dimensional target objects of the one or more previous frames and the three-dimensional point cloud data, the three-dimensional target objects of the one or more previous frames are detected as target detection results of the three-dimensional target objects of the one or more previous frames in the video.
5. The method of claim 1, wherein: Acquiring the position of the three-dimensional target object at the current frame time according to the detected two-dimensional target object and the three-dimensional point cloud data includes: According to the detected two-dimensional target object and the three-dimensional point cloud data, a three-dimensional target object corresponding to the two-dimensional target object in the three-dimensional point cloud data is obtained, and the position of the three-dimensional target object at the current frame time is obtained.
6. The method of claim 4, wherein, based on the detected two-dimensional target object and the three-dimensional point cloud data, and at least based on the target detection results of the three-dimensional target object in one or more previous frames in the video, obtaining the position of the detected three-dimensional target object at the current frame time comprises: When the three-dimensional target object detected in the one or more previous frames does not exist in the current frame, the position of the three-dimensional target object detected in the previous frame at the current frame time is predicted to obtain the predicted position of the three-dimensional target object at the current frame time.
7. The method of claim 6, wherein: Predicting the position of the three-dimensional target object detected in the previous frame at the current frame time to obtain the predicted position of the three-dimensional target object at the current frame time includes: A long short-term memory network (LSTM) is used to predict the position of the three-dimensional target object detected in the previous frame at the current frame time, so as to obtain the predicted position of the three-dimensional target object at the current frame time.
8. The method of claim 5, wherein: According to the detected two-dimensional target object and the three-dimensional point cloud data, acquiring a three-dimensional target object corresponding to the two-dimensional target object in the three-dimensional point cloud data, and obtaining the position of the three-dimensional target object at the current frame time comprises: De-noising the acquired three-dimensional point cloud data; The position of the three-dimensional target object at the current frame time is obtained according to the three-dimensional point cloud data after noise reduction.
9. The method of claim 1, wherein: Performing collision monitoring according to the position of the three-dimensional target object at the current frame time and outputting the collision possibility of the three-dimensional target object includes: Determine the distance between the position of the three-dimensional target object at the current frame time and another three-dimensional target object or a preset position; Collision monitoring is performed according to the distance, and the collision possibility of the three-dimensional target object is output.
10. A collision monitoring device, comprising: A detection unit, configured to obtain a two-dimensional image of a current frame in the video and perform target detection to obtain a detected two-dimensional target object; A conversion unit, configured to obtain a disparity image of the current frame at the same time as the two-dimensional image and convert it into three-dimensional point cloud data; a positioning unit, configured to obtain a position of the detected three-dimensional target object at the current frame time according to the detected two-dimensional target object and the three-dimensional point cloud data, and at least according to the target detection results of the three-dimensional target object in one or more previous frames in the video, wherein when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame time is obtained according to the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame time is obtained; The monitoring unit is configured to perform collision monitoring according to the position of the three-dimensional target object at the current frame time, and output the collision possibility of the three-dimensional target object.
11. A collision monitoring device, comprising: processor; and a memory having computer program instructions stored therein, Wherein, when the computer program instructions are executed by the processor, the processor is caused to perform the following steps: Obtain a two-dimensional image of the current frame in the video and perform target detection to obtain the detected two-dimensional target object; Obtaining a disparity image of the current frame at the same time as the two-dimensional image and converting it into three-dimensional point cloud data; According to the detected two-dimensional target object and the three-dimensional point cloud data, and at least according to the target detection results of the three-dimensional target object in one or more previous frames in the video, the position of the detected three-dimensional target object at the current frame time is obtained, wherein when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame time is obtained according to the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame time is obtained; Collision monitoring is performed according to the position of the three-dimensional target object at the current frame time, and the collision possibility of the three-dimensional target object is output.
12. A computer-readable storage medium having computer program instructions stored thereon, wherein: When the computer program instructions are executed by a processor, the following steps are implemented: Obtain a two-dimensional image of the current frame in the video and perform target detection to obtain the detected two-dimensional target object; Obtaining a disparity image of the current frame at the same time as the two-dimensional image and converting it into three-dimensional point cloud data; According to the detected two-dimensional target object and the three-dimensional point cloud data, and at least according to the target detection results of the three-dimensional target object in one or more previous frames in the video, the position of the detected three-dimensional target object at the current frame time is obtained, wherein when the three-dimensional target object is detected to exist in the current frame, the position of the three-dimensional target object at the current frame time is obtained according to the detected two-dimensional target object and the three-dimensional point cloud data, and when the three-dimensional target object is detected to exist only in the previous frame but not in the current frame, the predicted position of the three-dimensional target object at the current frame time is obtained; Collision monitoring is performed according to the position of the three-dimensional target object at the current frame time, and the collision possibility of the three-dimensional target object is output.
Citation Information
Cited By
Collision detection method and device, storage medium and electronic equipment
CN121040944A
