A method, device and equipment for obstacle detection in a blind area of a binocular camera

Abstract

Embodiments of the present application disclose a method, device and equipment for obstacle detection in a blind area of a binocular camera. Embodiments of the present application automatically detect whether a target obstacle enters the blind area of the binocular camera, and in the case that the blind area of the binocular camera enters the target obstacle, adaptively complete the obstacle information of the target obstacle in the current depth map, so as to perform obstacle detection based on the obstacle information in the current depth map. This realizes that even if there is a target obstacle in the blind area of the binocular camera, the target obstacle can be detected, and the risk of collision is avoided.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a method, apparatus, and device for detecting obstacles in the blind zone of a binocular camera. Background Technology

[0002] Binocular cameras are currently widely used in mobile robots. Mobile robots use binocular cameras to perceive obstacles and can detect and locate obstacles in a timely manner to avoid being obstructed during movement.

[0003] However, in applications, due to the limitations of the implementation principle of stereo cameras, they often have a certain blind spot (also known as the stereo camera blind zone) at close range. The larger the baseline distance of the stereo camera, the larger the blind spot at close range. When an obstacle enters the stereo camera blind zone, the two cameras (e.g., the left and right cameras) cannot simultaneously acquire images of the obstacle, and therefore cannot obtain the corresponding depth information. This means that when performing obstacle perception based on stereo depth information, obstacles within the stereo camera blind zone cannot be detected, creating a risk of collision. Summary of the Invention

[0004] This application discloses a method, apparatus, and device for detecting obstacles in the blind zone of a binocular camera, enabling the detection of obstacles entering the blind zone of a binocular camera.

[0005] According to a first aspect of the embodiments of this application, a method for detecting obstacles in the blind zone of a binocular camera is provided, the method comprising:

[0006] Obtain the current frame image and the current depth map corresponding to the current frame image from any camera in the binocular camera system;

[0007] Check whether the same target obstacle exists in the current frame image and the previous frame image captured by the camera; if so, detect whether the target obstacle is currently in the blind spot of the stereo camera based on the current depth map;

[0008] If the target obstacle is detected to be in the blind spot of the binocular camera, the obstacle information of the target obstacle is supplemented in the current depth map based on the current obstacle region of the target obstacle in the current frame image and the predicted trajectory information of the target obstacle in the current frame image, so as to perform obstacle detection based on the obstacle information in the current depth map; wherein, the obstacle information includes at least the depth information corresponding to the target obstacle.

[0009] According to a second aspect of the present application, an obstacle detection device for blind spots of a binocular camera is provided, the device comprising:

[0010] The acquisition unit is used to acquire the current frame image and the current depth map corresponding to the current frame image from any camera in the binocular camera system.

[0011] The target obstacle determination unit is used to check whether the same target obstacle exists in the current frame image and the previous frame image acquired by the camera; if so, it detects whether the target obstacle is currently in the blind zone of the binocular camera based on the current depth map.

[0012] The detection unit is configured to, when it is detected that the target obstacle is currently in the blind zone of the binocular camera, supplement the obstacle information of the target obstacle in the current depth map based on the current obstacle region of the target obstacle in the current frame image and the predicted trajectory information of the target obstacle in the current frame image, so as to perform obstacle detection based on the obstacle information in the current depth map; wherein, the obstacle information includes at least the depth information corresponding to the target obstacle.

[0013] According to a third aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a processor and a memory;

[0014] The memory is used to store machine-executable instructions;

[0015] The processor is used to read and execute machine-executable instructions stored in the memory to implement the obstacle detection method in the blind zone of the binocular camera as described above.

[0016] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0017] As can be seen from the above technical solutions, the embodiments of this application will automatically detect whether the blind spot of the binocular camera has entered the target obstacle. When the blind spot of the binocular camera is detected to have entered the target obstacle, the obstacle information of the target obstacle will be automatically supplemented in the current depth map so as to perform obstacle detection based on the obstacle information in the current depth map. This enables the detection of the target obstacle even if there is a target obstacle in the blind spot of the binocular camera, thus avoiding the risk of collision.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0020] Figure 1 A flowchart illustrating the method provided in this application embodiment;

[0021] Figure 2 A flowchart illustrating the implementation of step 102 provided in this application embodiment;

[0022] Figure 3 Another implementation flowchart of step 102 provided in the embodiments of this application;

[0023] Figure 4 A flowchart of trajectory tracking and prediction provided for embodiments of this application;

[0024] Figure 5 This is a structural diagram of the device provided in the embodiments of this application;

[0025] Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0028] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0029] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0030] See Figure 1, Figure 1 This is a flowchart illustrating a method provided in an embodiment of this application. As one embodiment, Figure 1 The illustrated process can be applied to mobile robots that include at least two cameras. A binocular camera consists of at least two cameras (e.g., labeled as the left camera and the right camera). Figure 1 As shown, the process may include the following steps:

[0031] Step 101: Obtain the current frame image and the current depth map corresponding to the current frame image from any camera in the binocular camera system.

[0032] In this embodiment, any of the aforementioned cameras can be either the left-eye camera or the right-eye camera. Optionally, in a specific implementation, the left-eye camera and the right-eye camera in the binocular camera will synchronously acquire images according to a preset period. Then, based on the images synchronously acquired by the left-eye camera and the right-eye camera, the corresponding depth map (denoted as the current depth map) can be obtained.

[0033] Step 102: Check if the same target obstacle exists in the current frame image and the previous frame image acquired by the camera. If so, based on the current depth map, detect whether the target obstacle is currently in the blind zone of the binocular camera.

[0034] In this embodiment, determining whether the same target obstacle exists in the current frame image and the previous frame image captured by the camera can rely on historical obstacle regions in the obtained previous frame image. For example, iterate through the current frame image to see if there is an obstacle region that matches a historical obstacle region. If so, it is determined that the same target obstacle exists in the current frame image and the previous frame image captured by the camera; otherwise, it is determined that the same target obstacle does not exist in the current frame image and the previous frame image. The following... Figure 2 , Figure 3 Two implementation methods are illustrated here, but will not be elaborated upon further.

[0035] In this embodiment, if it is determined whether the same target obstacle exists in the current frame image and the previous frame image, then it is possible to detect whether the target obstacle is currently in the blind spot of the binocular camera based on the current depth map.

[0036] Based on the principle of depth map acquisition by a stereo camera, if an obstacle exists in the current frame image, but there is no corresponding pixel in the current depth map, or if the pixel exists but has no depth information, it indicates that the obstacle has entered the stereo camera's blind zone (denoted as the target obstacle entering the stereo camera's blind zone). Therefore, in this embodiment, when detecting whether a target obstacle is currently in the stereo camera's blind zone based on the current depth map, it can first detect whether the target obstacle's depth information exists in the current depth map. If the target obstacle's depth information is not present in the current depth map, it is determined that the target obstacle is currently in the stereo camera's blind zone; otherwise, it is determined that the target obstacle is not currently in the stereo camera's blind zone.

[0037] It can be observed that once the blind spot of the binocular camera is determined to have entered the target obstacle, it means that the target obstacle may exist in the current frame image, but the depth information of the target obstacle is not present in the current depth map. To facilitate obstacle perception based on binocular depth information, the following step 103 needs to be performed.

[0038] Step 103: If the target obstacle is detected to be in the blind zone of the binocular camera, the obstacle information of the target obstacle is supplemented in the current depth map based on the current obstacle area of ​​the target obstacle in the current frame image and the predicted trajectory information of the target obstacle in the current frame image, so as to perform obstacle detection based on the obstacle information in the current depth map; wherein, the obstacle information includes at least the depth information corresponding to the target obstacle.

[0039] Step 103 is performed on the premise that step 102 detected the existence of the same target obstacle in the current frame image and the previous frame image. If the same target obstacle exists in the current frame image and the previous frame image, then the obstacle area in the current frame image that matches the historical obstacle area in the previous frame image is the current obstacle area.

[0040] Optionally, in this embodiment, based on the current obstacle region of the target obstacle in the current frame image and the predicted trajectory information of the target obstacle in the current frame image, supplementing the obstacle information of the target obstacle in the current depth map may include:

[0041] Step a1: Map the current obstacle region in the current frame image to the relevant region in the current depth map.

[0042] By step a1, the pixel corresponding to the aforementioned target obstacle will be present in the current depth map.

[0043] Step a2: Based on the predicted trajectory information of the target obstacle in the current frame image, set the corresponding depth information for each pixel in the relevant region.

[0044] Optionally, in this embodiment, the trajectory information includes at least the pose and velocity of the target obstacle relative to the binocular camera when the current frame image is acquired. Based on this, as an embodiment, step a2 can set corresponding depth information for each pixel in the relevant region according to the pose and velocity, and the depth information of the target obstacle in the previous frame depth map corresponding to the previous frame image. For example, step a2 can obtain the 3D point cloud pose corresponding to each pixel in the previous frame depth map based on the 3D point cloud position information of the target obstacle, combined with the obstacle's trajectory information. For any 3D point pose (xc, yc, zc) in the 3D point cloud pose, according to the camera imaging model u=fx*xc / zc+u0, v=fy*yc / zc+v0, where fx and fy are the equivalent focal lengths of the camera, and u0 and v0 are the center points, the corresponding pixel point on the current depth map can be obtained. If the pixel point belongs to the relevant region in step a1 above, the zc value in the 3D point pose is used as the depth information of the pixel point in the current depth map. This process is repeated to complete the depth information of the obstacle in the current depth map. Other methods can also be used, and this embodiment is not specifically limited. Ultimately, the obstacle information of the target obstacle is completed in the current depth map.

[0045] Finally, through steps a1 to a2, obstacle information of the target obstacle is completed in the current depth map based on the current obstacle region in the current frame image and the predicted trajectory information of the target obstacle in the current frame image. Once the obstacle information of the target obstacle is completed in the current depth map, obstacle perception can be performed based on the binocular depth information, enabling the detection of the target obstacle even if it is in the blind spot of the binocular camera, thus avoiding the risk of collision.

[0046] This concludes the process. Figure 1 The process is shown below.

[0047] pass Figure 1 As can be seen from the process shown, the embodiments of this application automatically detect whether the blind spot of the binocular camera has entered the target obstacle. When the blind spot of the binocular camera is detected to have entered the target obstacle, the obstacle information of the target obstacle will be automatically supplemented in the current depth map so as to perform obstacle detection based on the obstacle information in the current depth map. This enables the detection of the target obstacle even if there is a target obstacle in the blind spot of the binocular camera, thus avoiding the risk of collision.

[0048] The following is through Figure 2 Example of how to check if the same target obstacle exists in the current frame and the previous frame:

[0049] See Figure 2 , Figure 2 A flowchart illustrating the implementation of step 102 provided in an embodiment of this application. Figure 2 As shown, the process may include the following steps:

[0050] Step 201: Determine whether there is a target obstacle region in the current frame image that matches the historical obstacle region in the previous frame image. If yes, proceed to step 202; otherwise, proceed to step 203.

[0051] Optionally, in this embodiment, feature data corresponding to a specified feature attribute of a historical obstacle region can be obtained first. Taking grayscale as an example, the feature data is grayscale value. The target obstacle region matching the above feature data is traversed in the current frame image. If the target obstacle region is found in the current frame image, and the feature data corresponding to the specified feature attribute of the target obstacle region matches (e.g., identical or approximately) the feature data corresponding to the specified feature attribute of the historical obstacle region, then step 202 is executed, i.e., it is determined that the same target obstacle exists in the current frame image and the previous frame image captured by the camera. Otherwise, step 203 is executed, i.e., it is determined that the same target obstacle does not exist in the current frame image and the previous frame image captured by the camera.

[0052] Step 202: Determine that the same target obstacle exists in the current frame image and the previous frame image captured by the camera.

[0053] Step 203: Determine that there is no target obstacle in the current frame image and the previous frame image captured by the camera.

[0054] Finally, the method of matching historical obstacle regions with the current frame image is used as an example to illustrate how to check whether the same target obstacle exists in the current frame image and the previous frame image captured by the camera.

[0055] This concludes the process. Figure 2 The process is shown below.

[0056] The following is about Figure 3 The process is described as follows:

[0057] See Figure 3 , Figure 3 Another implementation flowchart of step 102 provided in the embodiments of this application. For example... Figure 3 As shown, the process may include the following steps:

[0058] Step 301: Input the current frame image into the trained instance segmentation model to extract obstacle feature information from the current frame image through the instance segmentation model.

[0059] The instance segmentation model here is pre-trained using deep learning, which will not be elaborated further. The obstacle feature information, such as pose information, is not specifically limited in this embodiment.

[0060] Step 302: Check whether the trajectory information of the obstacle predicted based on the historical obstacle area in the previous frame image matches the obstacle feature information mentioned above. If they match, it is determined that the current frame image and the previous frame image captured by the camera contain the same target obstacle. Otherwise, it is determined that the current frame image and the previous frame image do not contain the same target obstacle.

[0061] For example, the trajectory information mentioned above includes the pose of the obstacle relative to the stereo camera when the current frame image is captured, and the obstacle feature information is pose information. Then, the trajectory information is compared with the obstacle feature information to see if they are consistent or similar. If they are, the trajectory information is determined to match the obstacle feature information; otherwise, the trajectory information is determined not to match the obstacle feature information. This is just an example and not intended to be limiting.

[0062] This concludes the process. Figure 3 The flowchart of the method is shown.

[0063] pass Figure 3 This method enables the determination of whether the same target obstacle exists in the current frame image and the previous frame image by using deep learning instances and combining historical obstacle regions and predicted trajectories from the previous frame image.

[0064] It should be noted that in this embodiment, if it is determined in step 102 that the target obstacle is not currently in the blind spot of the binocular camera, then the current depth map is the binocular depth map. As an example, this embodiment can perform obstacle trajectory tracking and prediction based on this binocular depth map; see details below. Figure 4 The process is shown below.

[0065] See Figure 4 , Figure 4 A flowchart illustrating the trajectory tracking and prediction process provided in this application embodiment. Figure 4 As shown, the process may include the following steps:

[0066] Step 401: Convert the current depth map into a 3D point cloud.

[0067] In applications, a depth map can be viewed as an image, where each pixel's coordinates contain its own depth information (such as a depth value). As an example, for any pixel (u, v) in the depth map, combining its depth information (such as the depth value z) with the camera's intrinsic parameters fx, fy, u0, and v0 (where fx and fy are the camera's equivalent focal lengths, and u0 and v0 are the center points), the 3D coordinates (xc, yc, zc) of that point in the camera coordinate system can be obtained using the following formula:

[0068] zc=z, xc=zc*(u-u0) / fx, yc=zc*(v-v0) / fy.

[0069] By performing the above transformation on the pixels in the depth map, a 3D point cloud can be obtained.

[0070] It should be noted that this is just an example to illustrate how to convert the current depth map into a 3D point cloud, and is not intended to be limiting.

[0071] Step 402: Based on the above 3D point cloud, cluster the obstacle positions to obtain the obstacle position information corresponding to the current depth map.

[0072] Optionally, in this embodiment, step 402 can be implemented by first projecting the 3D point cloud onto a 2D grid (the size of each grid is the minimum resolution, which can be set to, for example, 0.01m). That is, without considering the z-coordinate, the grid where the x and y coordinates fall is considered to be projected onto that grid, and that grid is also considered to correspond to a pixel in the 3D point cloud (also called a 3D projection point). Thus, for each grid in the 2D grid, if there is a corresponding 3D projection point, the process extends to its neighboring grids in all directions. If neighboring grids also have 3D projection points, they are grouped into the same class, and this process of extending neighboring grids is repeated until none of its neighboring grids have 3D projection points. This completes one obstacle clustering process, and the resulting class can be called a candidate obstacle class. Then, based on the number of grids occupied by each candidate obstacle class, it is determined whether it is a target obstacle class. For example, if the number of grids occupied is greater than a threshold, the candidate obstacle class is considered a target obstacle class. Ultimately, at least one target obstacle class will be obtained. Since each grid cell occupied by the target obstacle class corresponds to a 3D projection point in the 3D point cloud, the 3D projection points corresponding to each grid cell occupied by the target obstacle class are combined to form the obstacle position information corresponding to the current depth map.

[0073] Step 403: Map the obstacle location information onto the current frame image to obtain the obstacle region.

[0074] In a binocular camera system, the left frame image captured by the left eye camera and the right frame image captured by the right eye camera both relate to depth. Figure 1With this one-to-one correspondence, it is easy to map the obstacle location information corresponding to the current depth map onto the current frame image, such as the left or right frame image, to obtain the obstacle region.

[0075] Step 404: Track and predict the trajectory of obstacles based on the obstacle area.

[0076] Taking a stationary mobile robot as an example, if the current value is t... n At any given moment, based on the obstacle region, the location information (x) of the center of the obstacle region can be determined. n ,y n ,z n Similarly, for the previous t1~t... n-1 The historical obstacle regions obtained at each time point have their center location information as (x1, y1, z1)...(x n-1 ,y n-1 ,z n-1 Based on the location information of each center, the trajectory information of the obstacle can be obtained through parameter fitting (v). x ,v y ,v z This means that obstacle trajectory tracking has been achieved.

[0077] Furthermore, in this embodiment, in conjunction with t n The center location information of the obstacle region at time t and the aforementioned trajectory information can be used to predict the trajectory at time t. n+1 The trajectory information of obstacles is acquired at all times while capturing image frames (v n+1 , v n+1 , v n+1 Of course, if the robot is in motion, its motion state needs to be taken into account and converted to a stationary state for tracking and prediction. Ultimately, obstacle trajectory tracking and prediction are achieved through the above description.

[0078] This concludes the process. Figure 4 The process is shown below.

[0079] pass Figure 4 The process shown enables obstacle trajectory tracking and prediction based on obstacle areas.

[0080] The methods provided in the embodiments of this application have been described above. The apparatus provided in the embodiments of this application is described below:

[0081] See Figure 5 , Figure 5 This is a structural diagram of the device provided in an embodiment of this application. Figure 5 As shown, the device includes:

[0082] The acquisition unit is used to acquire the current frame image and the current depth map corresponding to the current frame image from any camera in the binocular camera system.

[0083] The target obstacle determination unit is used to check whether the same target obstacle exists in the current frame image and the previous frame image acquired by the camera; if so, it detects whether the target obstacle is currently in the blind zone of the binocular camera based on the current depth map.

[0084] The detection unit is configured to, when it is detected that the target obstacle is currently in the blind zone of the binocular camera, supplement the obstacle information of the target obstacle in the current depth map based on the current obstacle region of the target obstacle in the current frame image and the predicted trajectory information of the target obstacle in the current frame image, so as to perform obstacle detection based on the obstacle information in the current depth map; wherein, the obstacle information includes at least the depth information corresponding to the target obstacle.

[0085] Optionally, the target obstacle determination unit checks whether the current frame image and the previous frame image captured by the camera have the same target obstacle, including: determining whether there is a target obstacle region in the current frame image that matches a historical obstacle region in the previous frame image, wherein, if the feature data corresponding to the specified feature attribute of the historical obstacle region matches the feature data corresponding to the feature attribute of the target obstacle region, then the historical obstacle region matches the target obstacle region; if so, it is determined that the current frame image and the previous frame image captured by the camera have the same target obstacle; otherwise, it is determined that the current frame image and the previous frame image do not have the same target obstacle.

[0086] Optionally, the target obstacle determination unit checks whether the same target obstacle exists in the current frame image and the previous frame image acquired by the camera, including: inputting the current frame image into a trained instance segmentation model to extract obstacle feature information from the current frame image through the instance segmentation model; checking whether the trajectory information of the obstacle predicted based on the historical obstacle region in the previous frame image matches the obstacle feature information; if they match, it is determined that the same target obstacle exists in the current frame image and the previous frame image acquired by the camera; otherwise, it is determined that the same target obstacle does not exist in the current frame image and the previous frame image.

[0087] Optionally, the target obstacle determination unit detects whether the target obstacle is currently in the blind zone of the binocular camera based on the current depth map, including: detecting whether the depth information of the target obstacle exists in the current depth map; if the depth information of the target obstacle does not exist in the current depth map, then it is determined that the target obstacle is currently in the blind zone of the binocular camera; otherwise, it is determined that the target obstacle is not currently in the blind zone of the binocular camera.

[0088] The detection unit completes the obstacle information of the target obstacle in the current depth map based on the current obstacle region of the target obstacle in the current frame image and the predicted trajectory information of the target obstacle in the current frame image. This includes: mapping the current obstacle region in the current frame image to the relevant region of the current depth map; and setting corresponding depth information for each pixel in the relevant region based on the predicted trajectory information of the target obstacle in the current frame image.

[0089] Optionally, in this embodiment, the trajectory information includes at least: the pose and velocity of the target obstacle relative to the binocular camera when the current frame image is acquired; based on this, the detection unit sets corresponding depth information for each pixel in the relevant region according to the predicted trajectory information of the target obstacle in the current frame image, including: setting corresponding depth information for each pixel in the relevant region according to the pose and velocity, and the depth information of the target obstacle in the previous frame depth map corresponding to the previous frame image.

[0090] Optionally, when the detection unit detects that the target obstacle is not currently in the blind zone of the binocular camera, it further converts the current depth map into a 3D point cloud; based on the 3D point cloud, it clusters the obstacle positions to obtain obstacle position information corresponding to the current depth map; it maps the obstacle position information onto the current frame image to obtain the obstacle region; and it performs obstacle trajectory tracking and prediction based on the obstacle region.

[0091] This concludes the process. Figure 5 Structural description of the device shown.

[0092] Correspondingly, embodiments of this application also provide a hardware structure diagram of an electronic device, specifically as follows: Figure 6 As shown, the electronic device can be a device implementing the above-described method. Figure 6 As shown, the hardware architecture includes a processor and memory.

[0093] The memory is used to store machine-executable instructions;

[0094] The processor is used to read and execute the machine-executable instructions stored in the memory to implement the obstacle detection method embodiment in the blind zone of the binocular camera as shown above.

[0095] As one embodiment, the memory can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, the memory can be volatile memory, non-volatile memory, or similar storage media. Specifically, the memory can be RAM (Random Access Memory), flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0096] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for obstacle detection in the blind zone of a binocular camera, characterized in that, The method includes: Obtain the current frame image and the current depth map corresponding to the current frame image from any camera in the binocular camera system; Check whether the same target obstacle exists in the current frame image and the previous frame image captured by the camera; if so, detect whether the target obstacle is currently in the blind spot of the stereo camera based on the current depth map; If the target obstacle is detected to be in the blind spot of the binocular camera, the obstacle information of the target obstacle is supplemented in the current depth map based on the current obstacle region of the target obstacle in the current frame image and the predicted trajectory information of the target obstacle in the current frame image, so as to perform obstacle detection based on the obstacle information in the current depth map; wherein, the obstacle information includes at least the depth information corresponding to the target obstacle.

2. The method according to claim 1, characterized in that, The step of checking whether the current frame image and the previous frame image captured by the camera contain the same target obstacle includes: Determine whether there is a target obstacle region in the current frame image that matches the historical obstacle region in the previous frame image. If the feature data corresponding to the specified feature attribute of the historical obstacle region matches the feature data corresponding to the feature attribute of the target obstacle region, then the historical obstacle region matches the target obstacle region. If so, it is determined that the current frame image and the previous frame image captured by the camera contain the same target obstacle; otherwise, it is determined that the current frame image and the previous frame image do not contain the same target obstacle.

3. The method according to claim 1, characterized in that, The step of checking whether the current frame image and the previous frame image captured by the camera contain the same target obstacle includes: The current frame image is input into a trained instance segmentation model to extract obstacle feature information from the current frame image through the instance segmentation model; Check whether the trajectory information of the obstacle predicted based on the historical obstacle region in the previous frame image matches the obstacle feature information in the current frame image. If they match, it is determined that the current frame image and the previous frame image captured by the camera contain the same target obstacle. Otherwise, it is determined that the current frame image and the previous frame image do not contain the same target obstacle.

4. The method according to claim 1, characterized in that, The step of detecting whether the target obstacle is currently in the blind spot of the binocular camera based on the current depth map includes: The depth information of the target obstacle is detected in the current depth map. If the depth information of the target obstacle is not found in the current depth map, it is determined that the target obstacle is currently in the blind spot of the binocular camera; otherwise, it is determined that the target obstacle is not currently in the blind spot of the binocular camera.

5. The method according to claim 1, characterized in that, The step of completing the obstacle information of the target obstacle in the current depth map based on the current obstacle region of the target obstacle in the current frame image and the predicted trajectory information of the target obstacle in the current frame image includes: Map the current obstacle region in the current frame image to the relevant region of the current depth map; Based on the predicted trajectory information of the target obstacle in the current frame image, corresponding depth information is set for each pixel in the relevant region.

6. The method according to claim 5, characterized in that, The trajectory information includes at least: the pose and velocity of the target obstacle relative to the binocular camera when the current frame image is acquired; The step of setting corresponding depth information for each pixel in the relevant region based on the predicted trajectory information of the target obstacle in the current frame image includes: Based on the pose, the velocity, and the depth information of the target obstacle in the previous frame depth map corresponding to the previous frame image, set the corresponding depth information for each pixel in the relevant region.

7. The method according to claim 1, characterized in that, When it is detected that the target obstacle is not currently in the blind zone of the binocular camera, the method further includes: Convert the current depth map into a 3D point cloud; Based on the 3D point cloud, the obstacle positions are clustered to obtain the obstacle position information corresponding to the current depth map; The obstacle location information is mapped onto the current frame image to obtain the obstacle region; Based on the obstacle area, the trajectory of the obstacle is tracked and predicted.

8. An obstacle detection device in the blind zone of a binocular camera, characterized in that, The device includes: The acquisition unit is used to acquire the current frame image and the current depth map corresponding to the current frame image from any camera in the binocular camera system. The target obstacle determination unit is used to check whether the same target obstacle exists in the current frame image and the previous frame image acquired by the camera; if so, it detects whether the target obstacle is currently in the blind zone of the binocular camera based on the current depth map. The detection unit is configured to, when it is detected that the target obstacle is currently in the blind zone of the binocular camera, supplement the obstacle information of the target obstacle in the current depth map based on the current obstacle region of the target obstacle in the current frame image and the predicted trajectory information of the target obstacle in the current frame image, so as to perform obstacle detection based on the obstacle information in the current depth map; wherein, the obstacle information includes at least the depth information corresponding to the target obstacle.

9. The apparatus according to claim 8, characterized in that, The step of checking whether the current frame image and the previous frame image captured by the camera contain the same target obstacle includes: determining whether there is a target obstacle region in the current frame image that matches a historical obstacle region in the previous frame image, wherein, if the feature data corresponding to a specified feature attribute of the historical obstacle region matches the feature data corresponding to the feature attribute of the target obstacle region, then the historical obstacle region matches the target obstacle region; if so, it is determined that the current frame image and the previous frame image captured by the camera contain the same target obstacle; otherwise, it is determined that the current frame image and the previous frame image do not contain the same target obstacle; and / or, The step of checking whether the current frame image and the previous frame image captured by the camera contain the same target obstacle includes: inputting the current frame image into a trained instance segmentation model to extract obstacle feature information from the current frame image through the instance segmentation model; checking whether the trajectory information of the obstacle predicted based on the historical obstacle region in the previous frame image matches the obstacle feature information; if they match, it is determined that the current frame image and the previous frame image captured by the camera contain the same target obstacle; otherwise, it is determined that the current frame image and the previous frame image do not contain the same target obstacle; and / or, The step of detecting whether the target obstacle is currently in the blind spot of the binocular camera based on the current depth map includes: detecting whether depth information of the target obstacle exists in the current depth map; if the depth information of the target obstacle does not exist in the current depth map, then determining that the target obstacle is currently in the blind spot of the binocular camera; otherwise, determining that the target obstacle is currently not in the blind spot of the binocular camera; and / or, The step of supplementing the obstacle information of the target obstacle in the current depth map based on the current obstacle region of the target obstacle in the current frame image and the predicted trajectory information of the target obstacle in the current frame image includes: mapping the current obstacle region in the current frame image to the relevant region of the current depth map; setting corresponding depth information for each pixel in the relevant region based on the predicted trajectory information of the target obstacle in the current frame image; and / or, The trajectory information includes at least: the pose and velocity of the target obstacle relative to the binocular camera when the current frame image is acquired; setting corresponding depth information for each pixel in the relevant region based on the predicted trajectory information of the target obstacle in the current frame image includes: setting corresponding depth information for each pixel in the relevant region based on the pose and velocity, and the depth information of the target obstacle in the previous frame depth map corresponding to the previous frame image; and / or, When the target obstacle is detected to be outside the blind zone of the binocular camera, the detection unit further converts the current depth map into a 3D point cloud; based on the 3D point cloud, the obstacle positions are clustered to obtain obstacle position information corresponding to the current depth map; the obstacle position information is mapped onto the current frame image to obtain the obstacle region; and obstacle trajectory tracking and prediction are performed based on the obstacle region.

10. An electronic device, characterized in that, The electronic device includes: a processor and memory; The memory is used to store machine-executable instructions; The processor is configured to read and execute machine-executable instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

Images