Shielding detection method and device and electronic equipment

Abstract

The invention relates to a shielding detection method and device and electronic equipment. The method comprises the steps that first depth information of at least one pixel point in a target image frame under a camera coordinate system is acquired; the target image frame is acquired by an image acquisition assembly in a three-dimensional scene at a target moment; obtaining a three-dimensional frame of the first object in the target scene frame; determining second depth information of at least one surface point of the three-dimensional frame in the camera coordinate system; and determining a shielding detection result of the first object based on the first depth information of the at least one pixel point in the camera coordinate system and the second depth information of the at least one surface point in the camera coordinate system. The shielding detection result of the first object can be automatically acquired in different three-dimensional scenes, the acquisition efficiency of the shielding detection result data is improved, and the accuracy of shielding detection of the first object is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving algorithms, and specifically to an occlusion detection method, device, and electronic device. Background Art

[0002] Three-dimensional (3D) visual perception is one of the important perception capabilities in the field of autonomous driving. With the continuous optimization of the performance of autonomous driving processing models, the demand for annotating 3D data has become more and more. In the process of annotating 3D data, the annotation information of objects includes attribute information related to the visibility of the objects, such as whether the objects are occluded, the occlusion rate, and other information. Manually judging visibility and annotating will result in inaccurate judgments and low efficiency. Summary of the invention

[0003] The embodiments of the present application provide an occlusion detection method, device, and electronic device, which can improve the problems of inaccurate judgment and low efficiency in manual judgment of object visibility.

[0004] In order to achieve the above purpose, the technical solution adopted in this application is as follows:

[0005] An occlusion detection method includes obtaining first depth information of at least one pixel point in a target image frame in a camera coordinate system; the target image frame is acquired by an image acquisition component in a three-dimensional scene at a target time, and the camera coordinate system is a camera coordinate system of the image acquisition component at the target time;

[0006] Acquire a three-dimensional frame of the first object in a target scene frame; the target scene frame is a scene frame of the three-dimensional scene at a target time, and the first object is located within the acquisition viewing angle range of the image acquisition component in the target scene frame;

[0007] Determine second depth information of at least one surface point of the three-dimensional frame in the camera coordinate system;

[0008] An occlusion detection result of the first object is determined based on first depth information of at least one pixel point in the camera coordinate system and second depth information of at least one surface point in the camera coordinate system.

[0009] According to the above-mentioned technical means, the occlusion detection result of the first object can be obtained by acquiring the first depth information of at least one pixel point in the target image frame captured by the image acquisition component in the camera coordinate system, and the second depth information of at least one surface point of the three-dimensional box of the first object in the target scene frame in the camera coordinate system. On the one hand, it is possible to automatically acquire the occlusion detection result of the first object in different three-dimensional scenes, thereby improving the efficiency of acquiring the occlusion detection result data; on the other hand, determining the occlusion detection result of the first object based on the first depth information and the second depth information in the same camera coordinate system can improve the accuracy of occlusion detection.

[0010] In some embodiments, determining second depth information of at least one surface point of the three-dimensional frame in a camera coordinate system includes:

[0011] Acquire first position information of at least one surface point of the three-dimensional frame in a world coordinate system of the three-dimensional scene, and second position information of the image acquisition component in the world coordinate system;

[0012] For each surface point, second depth information of the surface point in the camera coordinate system is determined based on the first position information, the second position information of the surface point and the camera parameters of the image acquisition component.

[0013] According to the above-mentioned technical means, based on the first position information of at least one surface point of the acquired three-dimensional frame in the world coordinate system of the three-dimensional scene, the second position information of the image acquisition component in the world coordinate system and the camera parameters of the image acquisition component, the second depth information of each surface point in the camera coordinate system can be determined, and the conversion of the three-dimensional coordinates of the surface point from the world coordinate system to the camera coordinate system can be completed, thereby improving the accuracy of the second depth information of the surface point relative to the image acquisition component.

[0014] In some embodiments, for each surface point, determining second depth information of the surface point in a camera coordinate system based on the first position information and the second position information of the surface point and a camera parameter of the image acquisition component includes:

[0015] For each surface point, based on the first position information, the second position information and the camera parameters of the image acquisition component, the surface point is mapped to the camera coordinate system to obtain the mapping coordinates corresponding to the surface point in the camera coordinate system; the camera coordinate system is constructed based on the second position information and the camera parameters of the image acquisition component;

[0016] Based on the mapped coordinates, second depth information of the surface point in the camera coordinate system is determined.

[0017] According to the above-mentioned technical means, for each surface point, the surface point can be mapped to the camera coordinate system to obtain the mapping coordinates, completing the conversion of the surface point coordinates from the world coordinate system to the camera coordinate system, and then based on the mapping coordinates of the surface point in the camera coordinate system, the second depth information of the surface point in the camera coordinate system is determined, thereby improving the accuracy of the second depth information of the surface point obtained after the coordinate system conversion relative to the image acquisition component.

[0018] In some embodiments, the second depth information includes a second depth value; and determining an occlusion detection result of the first object based on the first depth information of at least one pixel point in the camera coordinate system and the second depth information of at least one surface point in the camera coordinate system includes:

[0019] Determine the projection point of each surface point on the imaging plane of the image acquisition component;

[0020] Determine at least one target surface point from each surface point based on the distribution position of each projection point on the imaging plane and the second depth value of each surface point in the camera coordinate system;

[0021] An occlusion detection result of the first object is determined based on first depth information of at least one pixel point in the camera coordinate system and a second depth value of each target surface point in the camera coordinate system.

[0022] According to the above-mentioned technical means, at least one target surface point can be determined from each surface point based on the distribution position of the projection point of each surface point on the imaging plane of the image acquisition component and the second depth value of each surface point in the camera coordinate system, so as to realize flexible adjustment of the processing data used to determine the occlusion detection result of the first object according to different three-dimensional scenes and detection accuracy requirements; and based on the first depth information of at least one pixel point in the camera coordinate system and the second depth value of each target surface point in the camera coordinate system, the accuracy of the occlusion detection result of the first object is further improved.

[0023] In some embodiments, the imaging plane includes multiple imaging areas; based on the distribution position of each projection point on the imaging plane and the second depth value of each surface point in the camera coordinate system, at least one target surface point is determined from each surface point, including: for each imaging area, the surface point with the smallest second depth value among the surface points corresponding to each projection point in the imaging area is determined as the target surface point.

[0024] According to the above technical means, after obtaining each projection point, for each imaging area, the surface point with the smallest second depth value among the surface points corresponding to each projection point in the imaging area can be determined as the target surface point, thereby saving computing resources for subsequent steps and improving processing efficiency.

[0025] In some embodiments, the first depth information includes a first depth value; and determining the occlusion detection result of the first object based on the first depth information of at least one pixel in the camera coordinate system and the second depth value of each target surface point in the camera coordinate system includes:

[0026] For each target surface point, a second depth value of the target surface point in the camera coordinate system and a first depth value of a pixel point in the target image frame that matches the target surface point in the camera coordinate system are compared to obtain an occlusion state of the target surface point; the pixel point that matches the target surface point coincides with a projection point of the target surface point on the imaging plane of the image acquisition component;

[0027] An occlusion detection result of the first object is determined based on an occlusion state of the at least one target surface point.

[0028] According to the above technical means, the occlusion state of each target surface point can be obtained based on the second depth value and the first depth value, and the accuracy of the occlusion detection result of the first object can be improved based on the occlusion state of at least one target surface point.

[0029] In some implementations, comparing the second depth value of the target surface point in the camera coordinate system and the first depth value of the pixel point matching the target surface point in the target image frame in the camera coordinate system to obtain the occlusion state of the target surface point includes at least one of the following:

[0030] When the difference between the second depth value corresponding to the target surface point and the first depth value is not greater than the difference threshold, determining that the target surface point is not blocked;

[0031] When the difference between the first depth value and the second depth value corresponding to the target surface point is greater than the difference threshold, it is determined that the target surface point is blocked.

[0032] According to the above technical means, the occlusion state of the target surface point can be determined according to the relationship between the difference between the second depth value and the first depth value corresponding to the target surface point and the difference threshold, thereby further improving the accuracy of the determined occlusion state of the target surface point.

[0033] In some embodiments, the occlusion detection result includes an occlusion rate; and determining the occlusion detection result of the first object based on the occlusion state of at least one target surface point includes:

[0034] determining a target number of occluded surface points among the target surface points based on an occlusion state of at least one target surface point;

[0035] An occlusion rate of the first object is determined based on the number of objects and the total number of object surface points.

[0036] According to the above technical means, based on the occlusion state of at least one target surface point, the number of target obscured surface points in each target surface point can be determined, and then based on the number of targets and the total number of target surface points, the occlusion rate of the first object can be determined, thereby further improving the accuracy of the occlusion detection result for the first object.

[0037] An occlusion detection device includes a first acquisition module, which is used to acquire first depth information of at least one pixel point in a target image frame in a camera coordinate system; the target image frame is acquired by an image acquisition component in a three-dimensional scene at a target time, and the camera coordinate system is the camera coordinate system of the image acquisition component at the target time;

[0038] A second acquisition module is used to acquire a three-dimensional frame of the first object in a target scene frame; the target scene frame is a scene frame of the three-dimensional scene at a target time, and the first object is located within the acquisition viewing angle range of the image acquisition component in the target scene frame;

[0039] A first determination module, used to determine second depth information of at least one surface point of the three-dimensional frame in a camera coordinate system;

[0040] The second determination module is used to determine the occlusion detection result of the first object based on the first depth information of at least one pixel point in the camera coordinate system and the second depth information of at least one surface point in the camera coordinate system.

[0041] According to the above-mentioned technical means, the occlusion detection result of the first object can be obtained by acquiring the first depth information of at least one pixel point in the target image frame captured by the image acquisition component in the camera coordinate system, and the second depth information of at least one surface point of the three-dimensional box of the first object in the target scene frame in the camera coordinate system. On the one hand, it is possible to automatically acquire the occlusion detection result of the first object in different three-dimensional scenes, thereby improving the efficiency of acquiring the occlusion detection result data; on the other hand, determining the occlusion detection result of the first object based on the first depth information and the second depth information in the same camera coordinate system can improve the accuracy of occlusion detection.

[0042] An electronic device comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the above-mentioned occlusion detection method is implemented when the processor executes the program.

[0043] Beneficial effects of this application:

[0044] (1) Obtaining the first depth information of at least one pixel point in the target image frame acquired by the image acquisition component in the camera coordinate system and the second depth information of at least one surface point of the three-dimensional frame of the first object in the target scene frame in the camera coordinate system, thereby obtaining the occlusion detection result of the first object, and automatically obtaining the occlusion detection result of the first object in different three-dimensional scenes, thereby improving the efficiency of obtaining the occlusion detection result data;

[0045] (2) Determining the occlusion detection result of the first object based on the first depth information and the second depth information in the same camera coordinate system can improve the accuracy of the obtained visibility information of the first object;

[0046] (3) By completing the conversion of the three-dimensional coordinates of the surface point from the world coordinate system to the camera coordinate system, the accuracy of the second depth information of the surface point relative to the image acquisition component can be improved;

[0047] (4) based on the distribution position of the projection point of each surface point on the imaging plane of the image acquisition component and the second depth value of each surface point in the camera coordinate system, at least one target surface point is determined from each surface point, so that the processing data used to determine the occlusion detection result of the first object can be flexibly adjusted according to different three-dimensional scenes and detection accuracy requirements;

[0048] (5) After obtaining each projection point, for each imaging area, the surface point with the smallest second depth value among the surface points corresponding to each projection point in the imaging area is determined as the target surface point, thereby saving computing resources for subsequent steps and improving processing efficiency;

[0049] (6) determining an occlusion state of each target surface point based on the second depth value and the first depth value of each target surface point, wherein the accuracy of the occlusion detection result of the determined first object can be improved based on the occlusion state of the target surface point;

[0050] (7) determining the occlusion state of the target surface point according to the relationship between the difference between the second depth value and the first depth value corresponding to the target surface point and the difference threshold, which can further improve the accuracy of the determined occlusion state of the target surface point;

[0051] (8) Based on the target number of occluded surface points in each target surface point and the total number of target surface points, the occlusion rate of the first object is determined. The accuracy of the occlusion detection result for the first object can be further improved based on the occlusion rate of the first object. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1A A schematic diagram of a first implementation flow of an occlusion detection method provided in an embodiment of the present application;

[0053] Figure 1BA schematic diagram of the structure of a surface point provided in an embodiment of the present application;

[0054] Figure 2A A second schematic diagram of an implementation flow of an occlusion detection method provided in an embodiment of the present application;

[0055] Figure 2B A schematic diagram of a process for screening depth values ​​of projected points provided in an embodiment of the present application;

[0056] Figure 3 A schematic diagram of the structure of an occlusion detection device provided in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0059] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0060] In the process of annotating 3D data, the visibility information of real data is usually annotated manually. Manually judging visibility and annotating data may result in inaccurate judgment and low efficiency.

[0061] On this basis, the embodiment of the present application proposes an occlusion detection method, which can be performed by an electronic device. The electronic device may refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device) and other devices with data processing capabilities. Figure 1A As shown, the occlusion detection method includes the following steps S101 to S104:

[0062] Step S101, obtaining first depth information of at least one pixel in a target image frame in a camera coordinate system; the target image frame is acquired by an image acquisition component in a three-dimensional scene at a target time, and the camera coordinate system is the camera coordinate system of the image acquisition component at the target time.

[0063] Here, the three-dimensional scene may include a space scene of a three-dimensional structure, which may be a real scene or a virtual scene. For example, the three-dimensional scene may include but is not limited to a real road scene, a virtual simulated indoor scene, a virtual game scene, etc.

[0064] The target time may include a time corresponding to a data frame of a collected 3D scene, and may include a current time or a historical time. The data frame of the 3D scene may include but is not limited to at least one of an image frame of the 3D scene, 3D snapshot information, and the like.

[0065] For example, the image frames of the three-dimensional scene can be acquired by the image acquisition component.

[0066] For another example, the snapshot information of the three-dimensional scene at each moment can be recorded by simulation software. The snapshot information may include information such as the position and size of the elements in the three-dimensional scene; the position and size of the elements may include but are not limited to the relative position, three-dimensional coordinates, and rotation angle of the elements in the three-dimensional scene; the size of the elements may include but are not limited to at least one of the volume of the elements, the surface area of ​​the elements, and the surface point of the three-dimensional box corresponding to the elements.

[0067] The image acquisition component may be a device capable of acquiring information to generate images, such as at least one of a camera, an infrared camera, a thermal imager, a depth camera, and the like.

[0068] In some embodiments, different types of image acquisition components acquire different types of image frames.

[0069] For example, when the image acquisition component is a red-green-blue color mode (RGB color mode) camera, the target image frame acquired at the target moment may be an RGB (Red-Green-Blue color mode) image.

[0070] For another example, when the image acquisition component is a depth camera, the target image frame acquired at the target moment may be a depth map.

[0071] The first depth information may be information that can characterize the distance between a corresponding point of a pixel point in a target image frame in a three-dimensional scene and the image acquisition component. For example, the first depth information may be a depth value.

[0072] In some implementations, the first depth information may be determined based on a target depth image acquired by a depth image acquisition component, for example, based on a depth map acquired by a depth camera, to obtain depth values ​​of pixels in the depth map.

[0073] In some embodiments, the three-dimensional information of the element in the two-dimensional image may be determined based on multiple image frames by using a tracking algorithm, and the first depth information may be determined based on the three-dimensional information of the element. The three-dimensional information of the element may include but is not limited to information such as the position and size of the element.

[0074] In some implementations, the camera coordinate system corresponding to the image acquisition component at the target moment may be determined based on the parameters of the image acquisition component at the target moment.

[0075] For example, the camera coordinate system corresponding to the RGB camera at the target time can be determined based on the internal parameters and external parameters of the RGB camera. The internal parameters may include but are not limited to focal length, principal point coordinates, distortion coefficients, etc., and the external parameters may be parameters that characterize the position information of the RGB camera, such as rotation matrix, translation vector, etc.

[0076] Step S102, obtaining a three-dimensional frame of the first object in a target scene frame; the target scene frame is a scene frame of a three-dimensional scene at a target moment, and the first object in the target scene frame is located within the capture viewing angle of the image capture component.

[0077] Here, the three-dimensional box (ie, 3D box, stereo box) may be a three-dimensional boundary box marked on a two-dimensional image or three-dimensional data, and the three-dimensional box may simulate the shape and position of an object in a three-dimensional space.

[0078] The first object may include an object within the capture viewing angle of the image capture component at the target moment, and may be static and / or dynamic.

[0079] For example, when the three-dimensional scene is an indoor scene, the first object may include static items in the indoor scene, such as furniture, plants, utensils, etc.

[0080] For another example, when the three-dimensional scene is an outdoor scene, the first object may include static objects and traffic participants in the road traffic scene, such as road signs, green belts, pedestrians, vehicles, and the like.

[0081] The target scene frame may be a data frame corresponding to a target time, including three-dimensional information of each object in the three-dimensional scene at the target time.

[0082] For example, the target scene frame may include snapshot information of the three-dimensional scene.

[0083] In some implementations, the three-dimensional frame of the first object in the target scene frame may be obtained by acquiring snapshot information recorded by simulation software.

[0084] In some implementations, the capture viewing angle range of the image capture component can be determined based on the posture information of the image capture component; based on the capture viewing angle range of the image capture component, the first object captured by the image capture component at the target moment can be determined from the three-dimensional scene.

[0085] Step S103: Determine second depth information of at least one surface point of the three-dimensional frame in the camera coordinate system.

[0086] Here, the surface points of the 3D frame can be vertices or pixels located on the boundary of the 3D frame. The surface points of the 3D frame constitute the 3D form of the 3D frame and define the shape and size of the 3D frame. The surface points of the 3D frame not only have plane coordinates in the 2D image, but also have coordinates representing the distance in the 3D space, that is, depth information.

[0087] The second depth information may be information capable of representing the distance between at least one surface point of the three-dimensional frame and the image acquisition component. For example, the second depth information may be a depth value.

[0088] In some implementations, point cloud data of surface points of the three-dimensional frame may be generated at preset intervals on each side of the three-dimensional frame based on the position and posture of the first object and the size of the three-dimensional frame.

[0089] For example, Figure 1B As shown, Figure 1B It can be a rectangle corresponding to one of the planes of the three-dimensional box of the first object composed of rectangles corresponding to six planes, and the rectangle can generate a three-dimensional point cloud of the surface points of the three-dimensional box of the first object at an interval of 0.05m*0.05m. Figure 1B The rectangle in the figure is composed of 8*7 squares, a total of 56 squares, as shown in square 10. The specification of each square can be 0.05m*0.05m. The three-dimensional point cloud 11 in the square 10 is the first object in Figure 1B The three-dimensional point cloud 11 may be one or more.

[0090] By determining the second depth information of at least one surface point of the three-dimensional frame in the camera coordinate system, the distance between the image acquisition component and a point corresponding to the surface of at least one first object in the three-dimensional space can be determined.

[0091] Step S104: determine an occlusion detection result of the first object based on first depth information of at least one pixel point in the camera coordinate system and second depth information of at least one surface point in the camera coordinate system.

[0092] Here, the occlusion detection result may be visibility information at the target moment when the first object in the target scene is within the capture viewing angle of the image acquisition component, and may include but is not limited to at least one of the occlusion state, occlusion rate, occluder, etc.

[0093] In some implementations, the occlusion result of the first object may be determined based on the difference between the first depth information and the second depth information.

[0094] It can be understood that the first depth information of the pixel point in the target image frame in the camera coordinate system is the actual acquisition information of the image acquisition component, that is, the depth information of the pixel point corresponding to the point with the smallest distance from the image acquisition component in the three-dimensional scene, and the second depth information of the surface point of the first object in the camera coordinate system is the information characterizing the distance between the first object and the image acquisition component in the three-dimensional scene; by comparing the difference between the first depth information and the second depth information, it can be determined whether the surface point of the first object is the point with the smallest distance from the image acquisition component within the acquisition viewing angle of the image acquisition component in the three-dimensional scene, so as to determine the occlusion detection result of the first object.

[0095] According to the above-mentioned technical means, the occlusion detection result of the first object can be obtained by acquiring the first depth information of at least one pixel point in the target image frame captured by the image acquisition component in the camera coordinate system, and the second depth information of at least one surface point of the three-dimensional box of the first object in the target scene frame in the camera coordinate system. On the one hand, it is possible to automatically acquire the occlusion detection result of the first object in different three-dimensional scenes, thereby improving the efficiency of acquiring the occlusion detection result data; on the other hand, determining the occlusion detection result of the first object based on the first depth information and the second depth information in the same camera coordinate system can improve the accuracy of occlusion detection.

[0096] In some embodiments, the image acquisition component includes a depth image acquisition component, the target image frame includes a target depth image, the first depth information includes a first depth value, and the above step S101 may include the following steps S111 to S112:

[0097] Step S111: Acquire a target depth image acquired by a depth image acquisition component.

[0098] In some embodiments, the image acquisition component may include a color image acquisition component and a depth image acquisition component, and a target depth image corresponding to the color image acquired by the depth image acquisition component may be acquired while acquiring the color image acquired by the color image acquisition component.

[0099] Step S112: Obtain a first depth value of at least one pixel in the target depth image in the camera coordinate system.

[0100] According to the above technical means, the efficiency of obtaining the first depth value can be improved through the depth image acquisition component.

[0101] In some embodiments, the above step S102 may include the following steps S121:

[0102] Step S121: Determine a three-dimensional frame of the first object in the target scene frame based on the position and size of the first object in the target scene.

[0103] In some embodiments, before determining the three-dimensional frame of the first object in the target scene frame based on the position and size of the first object in the target scene, the target scene frame matching the target moment can be determined from a set of data frames corresponding to the three-dimensional scene based on time information of the target image frame.

[0104] For example, a target scene frame corresponding to a target time at which an RGB image and / or a depth map is collected may be determined from a set of data frames corresponding to the three-dimensional scene.

[0105] In some embodiments, the above step S103 may include the following steps S131 to S132:

[0106] Step S131, obtaining first position information of at least one surface point of the three-dimensional frame in the world coordinate system of the three-dimensional scene, and second position information of the image acquisition component in the world coordinate system.

[0107] Here, the first position information may be information capable of representing the position of at least one surface point of the three-dimensional frame in the world coordinate system of the three-dimensional scene. For example, the first position information may include the three-dimensional coordinates of the surface point in the world coordinate system corresponding to the three-dimensional scene.

[0108] The second position information may be information representing the position of the image acquisition component in the world coordinate system of the three-dimensional scene. For example, the second position information may include the three-dimensional coordinates of the image acquisition component in the world coordinate system corresponding to the three-dimensional scene.

[0109] In some embodiments, when the image acquisition component is located on a movable acquisition device, the three-dimensional coordinates of the image acquisition component in the world coordinate system corresponding to the three-dimensional scene can be the same as the three-dimensional coordinates of the movable acquisition device in the world coordinate system corresponding to the three-dimensional scene.

[0110] Step S132: for each surface point, based on the first position information and the second position information of the surface point and the camera parameters of the image acquisition component, determine the second depth information of the surface point in the camera coordinate system.

[0111] Here, the camera parameters may include internal parameters as well as external parameters of the image acquisition component.

[0112] In some embodiments, for each surface point, the distance between each surface point and the image acquisition component in the world coordinate system can be determined based on the first position information of the surface point and the second position information of the image acquisition component in the world coordinate system; based on the distance between each surface point and the image acquisition component in the world coordinate system and the camera parameters of the image acquisition component, the coordinate information of each surface point can be converted from the world coordinate system to the camera coordinate system to obtain the second depth information of the surface point in the camera coordinate system.

[0113] According to the above-mentioned technical means, based on the first position information of at least one surface point of the acquired three-dimensional frame in the world coordinate system of the three-dimensional scene, the second position information of the image acquisition component in the world coordinate system and the camera parameters of the image acquisition component, the second depth information of each surface point in the camera coordinate system can be determined, and the conversion of the three-dimensional coordinates of the surface point from the world coordinate system to the camera coordinate system can be completed, thereby improving the accuracy of the second depth information of the surface point relative to the image acquisition component.

[0114] In some embodiments, the above step S132 may include the following steps S141 to S142:

[0115] Step S141: for each surface point, based on the first position information, the second position information of the surface point and the camera parameters of the image acquisition component, map the surface point to the camera coordinate system to obtain the mapping coordinates corresponding to the surface point in the camera coordinate system; the camera coordinate system is constructed based on the second position information and the camera parameters of the image acquisition component.

[0116] In some embodiments, the mapping coordinates of the surface point in the camera coordinate system can be determined based on the three-dimensional coordinates of the surface point and the image acquisition component in the world coordinate system and the camera parameters of the image acquisition component, wherein the plane where the x-axis and the y-axis in the camera coordinate system are located can correspond to the plane where the image acquired by the image acquisition component is located.

[0117] In some embodiments, the mapping coordinates of the same surface point in the camera coordinate system at different times may be different. For example, when the viewing angle of the image acquisition component changes, the mapping coordinates of the same surface point from the world coordinate system to the camera coordinate system may be different.

[0118] Step S142: Determine second depth information of the surface point in the camera coordinate system based on the mapped coordinates.

[0119] Here, the second depth information of the surface point in the camera coordinate system may be determined based on the coordinates representing the distance between the surface point and the image acquisition component in the mapping coordinates.

[0120] In some embodiments, the coordinates of the camera coordinate system on the z-axis in the mapping coordinates may correspond to the distance between the surface point and the image acquisition component, and the second depth information may be determined based on the coordinate values ​​of the z-axis in the mapping coordinates.

[0121] According to the above-mentioned technical means, for each surface point, the surface point can be mapped to the camera coordinate system to obtain the mapping coordinates, completing the conversion of the surface point coordinates from the world coordinate system to the camera coordinate system, and then based on the mapping coordinates of the surface point in the camera coordinate system, the second depth information of the surface point in the camera coordinate system is determined, thereby improving the accuracy of the second depth information of the surface point obtained after the coordinate system conversion relative to the image acquisition component.

[0122] In some embodiments, the second depth information includes a second depth value, and the above step S104 may include the following steps S151 to S153:

[0123] Step S151, determining the projection point of each surface point on the imaging plane of the image acquisition component.

[0124] Here, the imaging plane of the image acquisition component may include the plane where the x-axis and the y-axis in the camera coordinate system are located, that is, it may correspond to the plane where the image acquired by the image acquisition component is located.

[0125] Step S152: determine at least one target surface point from each surface point based on the distribution position of each projection point on the imaging plane and the second depth value of each surface point in the camera coordinate system.

[0126] Here, the target surface point may include at least one surface point among all surface points corresponding to all projection points.

[0127] In some embodiments, there may be multiple first objects, and the surface points of each first object correspond to respective projection points; after determining the projection point of each surface point on the imaging plane, the target surface point corresponding to each first object may be determined respectively.

[0128] For example, after obtaining the projection points of the multiple first objects, the imaging plane can be divided into squares with preset distance intervals based on the distribution positions of the projection points on the imaging plane, and for each first object, the surface points in each square whose second depth value is within the preset range are used to determine the target surface point of the first object. In this way, the processing data used to determine the occlusion detection result of the first object can be flexibly adjusted according to different three-dimensional scenes and detection accuracy requirements.

[0129] Step S153: determine an occlusion detection result of the first object based on first depth information of at least one pixel in the camera coordinate system and a second depth value of each target surface point in the camera coordinate system.

[0130] In some embodiments, based on the distribution position of the projection points corresponding to the target surface points on the imaging plane, the first depth information of at least one pixel point corresponding to the distribution position in the target image frame can be determined, and based on the difference between the second depth value of each target surface point and the first depth information, the occlusion detection result of the first object can be determined.

[0131] According to the above-mentioned technical means, at least one target surface point can be determined from each surface point based on the distribution position of the projection point of each surface point on the imaging plane of the image acquisition component and the second depth value of each surface point in the camera coordinate system, so as to realize flexible adjustment of the processing data used to determine the occlusion detection result of the first object according to different three-dimensional scenes and detection accuracy requirements; and based on the first depth information of at least one pixel point in the camera coordinate system and the second depth value of each target surface point in the camera coordinate system, the accuracy of the occlusion detection result of the first object is further improved.

[0132] In some embodiments, the imaging plane includes multiple imaging areas, and the above step S152 may include the following steps S161:

[0133] Step S161 : for each imaging area, determine the surface point with the smallest second depth value among the surface points corresponding to the projection points in the imaging area as the target surface point.

[0134] Here, the surface point with the smallest second depth value among the surface points corresponding to the projection points in the imaging area may correspond to the surface point closest to the image acquisition component.

[0135] In some embodiments, after the imaging plane is divided into multiple imaging areas based on the distribution positions of the projection points on the imaging plane, for each imaging area, the surface point with the smallest second depth value is retained as the target surface point, and other surface points except the target surface point are deleted.

[0136] According to the above technical means, after obtaining each projection point, for each imaging area, the surface point with the smallest second depth value among the surface points corresponding to each projection point in the imaging area can be determined as the target surface point, thereby saving computing resources for subsequent steps and improving processing efficiency.

[0137] In some embodiments, the first depth information includes a first depth value, and the above step S153 may include the following steps S171 to S172:

[0138] Step S171: for each target surface point, compare the second depth value of the target surface point in the camera coordinate system and the first depth value of the pixel point matching the target surface point in the target image frame in the camera coordinate system to obtain the occlusion state of the target surface point; the pixel point matching the target surface point coincides with the projection point of the target surface point on the imaging plane of the image acquisition component.

[0139] Here, by comparing the first depth value of the target surface point with the first depth value of the pixel at the corresponding position of the target surface point in the target image frame, it can be determined whether the depth value of the target surface point is the same or similar to the depth value of the pixel captured by the image acquisition component at the corresponding position: if they are the same or similar, it can be determined that the target surface point is the content displayed by the pixel in the target image frame captured by the image acquisition component, that is, the target surface point is not obscured; if they are different or dissimilar, it can be determined that the target surface point is not the content displayed by the pixel in the target image frame captured by the image acquisition component, that is, the target surface point is obscured.

[0140] Step S172: Determine an occlusion detection result of the first object based on an occlusion state of at least one target surface point.

[0141] In some implementations, when the occlusion status of all target surface points is not occluded, the occlusion detection result of the first object may be determined as the first object is not occluded.

[0142] In some implementations, when the occlusion state of at least one target surface point is occluded, the occlusion detection result of the first object may be determined as the first object being occluded.

[0143] In some embodiments, when the occlusion detection result of the first object determines that the first object is occluded, the occlusion parameter of the first object can be determined based on at least one occluded target surface point. The occlusion parameter may include but is not limited to the occlusion rate of the first object, the number and type of occluders of the first object, etc.

[0144] According to the above technical means, the occlusion state of each target surface point can be obtained based on the second depth value and the first depth value, and the accuracy of the occlusion detection result of the first object can be improved based on the occlusion state of at least one target surface point.

[0145] In some embodiments, the above step S171 may include at least one of the following steps S181 to S182:

[0146] Step S181: When the difference between the second depth value and the first depth value corresponding to the target surface point is not greater than a difference threshold, determine that the target surface point is not blocked.

[0147] Here, the difference threshold may include a preset minimum difference.

[0148] In some embodiments, when the difference between the second depth value and the first depth value is not greater than a difference threshold, it can be determined that the second depth value is similar to or identical to the first depth value, that is, the target surface point is identical to the content displayed by the pixel in the target image frame, and the target surface point is not obscured.

[0149] For example, the difference threshold is 0 m. When the difference between the second depth value and the first depth value is 0 m, it can be determined that the second depth value is the same as the first depth value, and the target surface point is not blocked.

[0150] For another example, taking into account the possible error between the calculated second depth information and the actually collected first depth information, the difference threshold can be set to 0.5m. When the difference between the second depth value and the first depth value is 0.4m, it can be determined that the second depth value is similar to the first depth value and the target surface point is not obscured.

[0151] Step S182: When the difference between the first depth value and the second depth value corresponding to the target surface point is greater than a difference threshold, it is determined that the target surface point is blocked.

[0152] In some embodiments, when the difference between the second depth value and the first depth value is greater than a difference threshold, it can be determined that the second depth value is different from the first depth value, that is, the target surface point is different from the content displayed by the pixel point in the target image frame, and the target surface point is blocked.

[0153] According to the above technical means, the occlusion state of the target surface point can be determined according to the relationship between the difference between the second depth value and the first depth value corresponding to the target surface point and the difference threshold, thereby further improving the accuracy of the determined occlusion state of the target surface point.

[0154] In some embodiments, the occlusion detection result includes an occlusion rate, and the above step S172 may include the following steps S191 to S192:

[0155] Step S191: Determine the target number of obscured surface points in each target surface point based on the obstruction state of at least one target surface point.

[0156] In some implementations, a target surface point for which the difference between the corresponding first depth value and the second depth value is greater than a difference threshold may be determined as an obscured surface point.

[0157] For example, the number of occluded surface points can be expressed as p occ .

[0158] Step S192: Determine the occlusion rate of the first object based on the number of targets and the total number of target surface points.

[0159] In some implementations, the occlusion rate of the first object may be determined based on a ratio between the number of occluded surface points and the total number of target surface points.

[0160] For example, the total number of target surface points can be expressed as p all , the occlusion rate can be expressed as occ ratio , occ ratio =p occ / p all For example, the number of target surface points p that are occluded by the six planes of the first object is occ is 200, the total number of target surface points in the 6 planes p all When the occlusion rate is 250, ratio It can be 80%,

[0161] According to the above technical means, based on the occlusion state of at least one target surface point, the number of target obscured surface points in each target surface point can be determined, and then based on the number of targets and the total number of target surface points, the occlusion rate of the first object can be determined, thereby further improving the accuracy of the occlusion detection result for the first object.

[0162] The present application also proposes an occlusion detection method that can be performed by an electronic device. Taking the occlusion detection result of traffic participants in a simulation scene of autonomous driving as an example, Figure 2A As shown, the method includes the following steps S201 to S206:

[0163] Step S201: Acquire a depth map according to the viewing angle of a camera.

[0164] Here, the camera corresponds to the image acquisition component in the above-mentioned occlusion detection method.

[0165] During implementation, an RGB camera and a depth camera for capturing the simulation scene may be set. In the simulation scene, the positions of the two cameras may be the same.

[0166] During implementation, a simulation scene can be built by importing static environment materials and traffic participant models.

[0167] During implementation, the traffic participants in the simulation scene can be placed at reasonable positions and their routes can be set, for example, pedestrians on the sidewalk and vehicles on the lane.

[0168] During implementation, when the RGB camera and the depth camera are placed on the collection vehicle, after the driving route of the collection vehicle is set, the snapshot information of each frame of the simulation scene can be recorded through the simulation software, such as the position of the traffic participants, the time information of each frame, etc., and the RGB image and the depth map corresponding to the RGB image are collected through the RGB camera and the depth camera. Among them, the RGB image and the depth map corresponding to the RGB image can correspond to the target image frame, and the snapshot information of the simulation scene can correspond to the target scene frame.

[0169] Step S202: Obtain the position and posture of each traffic participant and the size of the three-dimensional frame.

[0170] During implementation, the simulation scene snapshot information matching the time information may be searched according to the time information of the RGB image and the depth map corresponding to the RGB image. The time information may correspond to the target moment in the above-mentioned occlusion detection method.

[0171] During implementation, the three-dimensional coordinates, rotation angle, and three-dimensional frame size of each traffic participant model in the world coordinate system of the simulation scene can be obtained from the snapshot information of the simulation scene.

[0172] In some implementations, the three-dimensional frame surface points may be placed in the simulation scene at the position and rotation angle of each traffic participant model to obtain the three-dimensional coordinates of each three-dimensional frame surface point in the world coordinate system corresponding to the simulation scene.

[0173] Step S203: Generate three-dimensional frame surface points for each traffic participant model.

[0174] Here, three-dimensional frame surface points corresponding to the three-dimensional frame of each traffic participant model are generated.

[0175] During implementation, the traffic participants within the acquisition angle of view of the RGB camera at the target moment can be screened out based on the acquisition angle of view of the RGB camera for the data corresponding to the target moment.

[0176] Step S204: Project the surface points of the three-dimensional frame according to the camera viewing angle.

[0177] During implementation, after mapping the three-dimensional coordinates of each surface point in the simulation scene corresponding to the world coordinate system to the camera coordinate system of the RGB camera, the projection point of the three-dimensional frame surface point on the imaging plane of the camera can be obtained.

[0178] Step S205: screening the depth values ​​of the projected surface points to remove the obscured surface points.

[0179] During implementation, all projection points may be screened based on the depth values ​​of the surface points corresponding to the projection points, the obscured surface points may be eliminated, and the surface points finally retained may be determined as the target surface points.

[0180] Step S206: Calculate the visibility of the object using the depth map and the depth of the target surface point.

[0181] Here, the depth map can correspond to the target image frame in the above-mentioned occlusion detection method; the depth of the target surface point is the depth value of the target surface point in the camera coordinate system, which can correspond to the second depth value and the second depth information in the above-mentioned occlusion detection method; the object visibility corresponds to the visibility information of each traffic participant, that is, the occlusion detection result of the first object in the above-mentioned occlusion detection method.

[0182] During implementation, the occlusion state of the target surface point may be determined based on the depth value of the target surface point and the first depth value of the pixel point corresponding to the target surface point in the depth map, and the object visibility may be further calculated.

[0183] like Figure 2B As shown, the above step S205 may include the following steps S211 to S218:

[0184] Step S211, obtaining the projected point set.

[0185] Here, a point set consisting of the projected three-dimensional frame surface points of each traffic participant obtained in step S204 is obtained.

[0186] Step S212: Apply the hash function divided by the grid to each point in the point set.

[0187] Here, the grid may correspond to the imaging area in the above-mentioned occlusion detection method, and a hash function is used to generate a hash value of the grid.

[0188] For example, the projected three-dimensional frame surface points can be parameterized into a two-dimensional plane according to each traffic participant. The imaging plane is divided into multiple plane grids of 0.05m*0.05m, each grid can have a projection point of at least one surface point of the traffic participant, and each grid has a different hash value.

[0189] Step S213: Determine whether the obtained hash value already exists.

[0190] Here, each hash value corresponds to a grid. If the hash value of the surface point already exists, it is determined that the grid corresponding to the hash value already exists, that is, there are other projection points of the surface point in the imaging area, and step S214 is executed; if the hash value does not exist, it is determined that there are no other projection points of the surface point in the imaging area, and step S217 is directly executed.

[0191] Step S214: Compare the depth values ​​of the surface points of the existing hash grid with the depth values ​​of the calculated points.

[0192] Here, the calculated point corresponds to the surface point of the hash grid for which it is not yet determined whether a corresponding point already exists; the hash grid is a grid with a hash value; the surface point of the existing hash grid is a surface point with a projection point in the imaging area corresponding to the grid. Among them, the surface points of the existing hash grid and the calculated point can both correspond to the surface points of the three-dimensional box projected on the imaging plane of the image acquisition component in the above-mentioned occlusion detection method.

[0193] In the case where a hash grid already exists, it indicates that there are projection points of other surface points in the imaging area corresponding to the hash grid. The depth values ​​of the surface points of the existing hash grid and the depth values ​​of the calculated points can both correspond to the second depth values ​​in the above occlusion detection method.

[0194] Step S215: Determine whether the depth value of the calculated point is greater than the depth value of the existing surface point in the hash grid.

[0195] Here, if the depth value of the calculated point is greater than the depth value of the existing surface point in the hash grid, it is determined that the calculated point is blocked and step S216 is executed; if the depth value of the calculated point is not greater than the depth value of the existing surface point in the hash grid, it is determined that the calculated point is not blocked and step S217 is executed.

[0196] Step S216: discard the calculation point.

[0197] Here, after executing step S216 , step S213 may continue to be executed until there is no surface point for which it is not determined whether a corresponding hash grid already exists.

[0198] Step S217: Fill the calculated points into the grid corresponding to the hash value, and delete other points in the grid.

[0199] Step S218: Obtain the unobstructed point set.

[0200] Here, the unobstructed point set corresponds to the set of target surface points in the above-mentioned occlusion detection method, that is, the point set of target surface points of each traffic participant within the acquisition viewing angle of the camera at the current moment.

[0201] According to the above technical means, by acquiring the position and posture of each traffic participant and the size of the three-dimensional box, a three-dimensional box corresponding to each traffic participant is generated, the surface points of the three-dimensional box in the world coordinate system are projected into the camera coordinate system, and the depth value of the projected target surface point is compared with the depth value of the pixel point in the depth map in the camera coordinate system to calculate the visibility of each traffic participant. On the one hand, it is possible to automatically complete the visibility annotation of traffic participants in the simulation scene of autonomous driving; on the other hand, the surface points of the projected traffic participant are screened based on the depth value, and the visibility of the object is calculated based on the target surface points retained after screening, which can improve the calculation efficiency and the accuracy of the calculation results.

[0202] The present application also proposes an occlusion detection device, such as Figure 3 As shown, the device 300 includes:

[0203] A first acquisition module 301 is used to acquire first depth information of at least one pixel in a target image frame in a camera coordinate system; the target image frame is acquired by an image acquisition component in a three-dimensional scene at a target time, and the camera coordinate system is a camera coordinate system of the image acquisition component at the target time;

[0204] A second acquisition module 302 is used to acquire a three-dimensional frame of the first object in a target scene frame; the target scene frame is a scene frame of the three-dimensional scene at the target moment, and the first object is located within the acquisition viewing angle of the image acquisition component in the target scene frame;

[0205] A first determining module 303 is used to determine second depth information of at least one surface point of the three-dimensional frame in the camera coordinate system;

[0206] The second determination module 304 is configured to determine an occlusion detection result of the first object based on the first depth information of the at least one pixel point in the camera coordinate system and the second depth information of the at least one surface point in the camera coordinate system.

[0207] In some embodiments, the first determination module can also be used to: obtain first position information of at least one surface point of the three-dimensional box in the world coordinate system of the three-dimensional scene, and second position information of the image acquisition component in the world coordinate system; for each of the surface points, determine the second depth information of the surface point in the camera coordinate system based on the first position information of the surface point, the second position information and the camera parameters of the image acquisition component.

[0208] In some embodiments, the first determination module can also be used to: for each of the surface points, based on the first position information of the surface point, the second position information and the camera parameters of the image acquisition component, map the surface point to the camera coordinate system to obtain the mapping coordinates corresponding to the surface point in the camera coordinate system; the camera coordinate system is constructed based on the second position information and the camera parameters of the image acquisition component; based on the mapping coordinates, determine the second depth information of the surface point in the camera coordinate system.

[0209] In some embodiments, the second depth information includes a second depth value, and the second determination module can also be used to: determine the projection point of each of the surface points on the imaging plane of the image acquisition component; determine at least one target surface point from each of the surface points based on the distribution position of each of the projection points on the imaging plane and the second depth value of each of the surface points in the camera coordinate system; determine the occlusion detection result of the first object based on the first depth information of the at least one pixel point in the camera coordinate system and the second depth value of each of the target surface points in the camera coordinate system.

[0210] In some embodiments, the imaging plane includes multiple imaging areas, and the second determination module can also be used to: for each of the imaging areas, determine the surface point with the smallest second depth value among the surface points corresponding to the projection points in the imaging area as the target surface point.

[0211] In some embodiments, the first depth information includes a first depth value, and the second determination module can also be used to: for each of the target surface points, compare the second depth value of the target surface point in the camera coordinate system and the first depth value of the pixel point in the target image frame that matches the target surface point in the camera coordinate system to obtain the occlusion state of the target surface point; the pixel point that matches the target surface point coincides with the projection point of the target surface point on the imaging plane of the image acquisition component; and determine the occlusion detection result of the first object based on the occlusion state of at least one of the target surface points.

[0212] In some embodiments, the second determination module can also be used for at least one of the following: when the difference between the second depth value corresponding to the target surface point and the first depth value is not greater than a difference threshold, determining that the target surface point is not obscured; when the difference between the first depth value corresponding to the target surface point and the second depth value is greater than the difference threshold, determining that the target surface point is obscured.

[0213] In some embodiments, the occlusion detection result includes an occlusion rate, and the second determination module can also be used to: determine the target number of occluded surface points in each of the target surface points based on the occlusion state of at least one of the target surface points; and determine the occlusion rate of the first object based on the target number and the total number of the target surface points.

[0214] The present application also provides an electronic device, including a memory and a processor. Figure 4 As shown, the electronic device 400 includes:

[0215] Memory 401, used to store computer programs that can be run on processor 402;

[0216] The processor 402 is used to execute the program stored in the memory 401 to implement the above-mentioned occlusion detection method.

[0217] The embodiment of the present application further proposes a computer program, including a computer-readable code. When the computer-readable code is executed in a computer device, a processor in the computer device executes part or all of the steps for implementing the above-mentioned occlusion detection method.

[0218] The embodiment of the present application further proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements some or all of the steps in the above-mentioned occlusion detection method.

[0219] The embodiment of the present application further proposes a computer-readable storage medium, which stores a computer program. The computer program can be executed by a processor to implement the above-mentioned occlusion detection method.

[0220] The present application is described with reference to the flowchart and / or block diagram of the method, device and equipment according to the embodiment of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, and the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0221] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0222] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0223] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed in the present application should be included in the protection scope of the present application.

[0224] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0225] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0226] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0227] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0228] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0229] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0230] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0231] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0232] The above embodiments are only preferred embodiments for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Any equivalent substitution or change made by a person skilled in the art based on the present application is within the protection scope of the present application.

Claims

1. A method for detecting occlusion, characterized in that: include: Acquire first depth information of at least one pixel in the target image frame in the camera coordinate system; The target image frame is acquired by an image acquisition component in a three-dimensional scene at a target time, and the camera coordinate system is the camera coordinate system of the image acquisition component at the target time; Obtaining a three-dimensional frame of the first object in the target scene frame; The target scene frame is a scene frame of the three-dimensional scene at the target time, and the first object is located within the acquisition viewing angle of the image acquisition component in the target scene frame; Determine second depth information of at least one surface point of the three-dimensional frame in the camera coordinate system; An occlusion detection result of the first object is determined based on the first depth information of the at least one pixel point in the camera coordinate system and the second depth information of the at least one surface point in the camera coordinate system.

2. The method according to claim 1, characterized in that The determining second depth information of at least one surface point of the three-dimensional frame in the camera coordinate system includes: Acquire first position information of at least one surface point of the three-dimensional frame in a world coordinate system of the three-dimensional scene, and second position information of the image acquisition component in the world coordinate system; For each of the surface points, second depth information of the surface point in the camera coordinate system is determined based on the first position information of the surface point, the second position information and the camera parameters of the image acquisition component.

3. The method according to claim 2, characterized in that The determining, for each of the surface points, second depth information of the surface point in the camera coordinate system based on the first position information of the surface point, the second position information, and a camera parameter of the image acquisition component, comprises: For each of the surface points, based on the first position information of the surface point, the second position information and the camera parameters of the image acquisition component, the surface point is mapped to the camera coordinate system to obtain the mapping coordinates corresponding to the surface point in the camera coordinate system; the camera coordinate system is constructed based on the second position information and the camera parameters of the image acquisition component; Based on the mapped coordinates, the second depth information of the surface point in the camera coordinate system is determined.

4. The method according to any one of claims 1 to 3, characterized in that The second depth information includes a second depth value; and determining the occlusion detection result of the first object based on the first depth information of the at least one pixel point in the camera coordinate system and the second depth information of the at least one surface point in the camera coordinate system includes: Determine a projection point of each of the surface points on an imaging plane of the image acquisition component; Determine at least one target surface point from each of the surface points based on the distribution position of each of the projection points on the imaging plane and the second depth value of each of the surface points in the camera coordinate system; An occlusion detection result of the first object is determined based on the first depth information of the at least one pixel in the camera coordinate system and the second depth value of each of the target surface points in the camera coordinate system.

5. The method according to claim 4, characterized in that The imaging plane includes a plurality of imaging areas; The determining at least one target surface point from each of the surface points based on the distribution position of each of the projection points on the imaging plane and the second depth value of each of the surface points in the camera coordinate system comprises: For each of the imaging areas, a surface point with the smallest second depth value among the surface points corresponding to the projection points in the imaging area is determined as a target surface point.

6. The method according to claim 4, characterized in that The first depth information includes a first depth value; The determining, based on the first depth information of the at least one pixel point in the camera coordinate system and the second depth value of each of the target surface points in the camera coordinate system, an occlusion detection result of the first object comprises: For each of the target surface points, a second depth value of the target surface point in the camera coordinate system and a first depth value of a pixel point in the target image frame that matches the target surface point in the camera coordinate system are compared to obtain an occlusion state of the target surface point; the pixel point that matches the target surface point coincides with a projection point of the target surface point on the imaging plane of the image acquisition component; An occlusion detection result of the first object is determined based on an occlusion state of at least one of the target surface points.

7. The method according to claim 6, characterized in that The comparing the second depth value of the target surface point in the camera coordinate system and the first depth value of the pixel point matching the target surface point in the target image frame in the camera coordinate system to obtain the occlusion state of the target surface point includes at least one of the following: When the difference between the second depth value corresponding to the target surface point and the first depth value is not greater than a difference threshold, determining that the target surface point is not blocked; When the difference between the first depth value and the second depth value corresponding to the target surface point is greater than the difference threshold, it is determined that the target surface point is blocked.

8. The method according to claim 6, characterized in that The occlusion detection result includes an occlusion rate; The determining the occlusion detection result of the first object based on the occlusion state of at least one of the target surface points comprises: determining the target number of occluded surface points in each of the target surface points based on the occlusion state of at least one of the target surface points; An occlusion rate of the first object is determined based on the number of targets and the total number of target surface points.

9. An occlusion detection device, characterized in that: include: A first acquisition module, used to acquire first depth information of at least one pixel point in a target image frame in a camera coordinate system; The target image frame is acquired by an image acquisition component in a three-dimensional scene at a target time, and the camera coordinate system is the camera coordinate system of the image acquisition component at the target time; A second acquisition module, used to acquire a three-dimensional frame of the first object in the target scene frame; The target scene frame is a scene frame of the three-dimensional scene at the target time, and the first object is located within the acquisition viewing angle of the image acquisition component in the target scene frame; A first determination module, configured to determine second depth information of at least one surface point of the three-dimensional frame in the camera coordinate system; The second determination module is used to determine the occlusion detection result of the first object based on the first depth information of the at least one pixel point in the camera coordinate system and the second depth information of the at least one surface point in the camera coordinate system.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the method according to any one of claims 1 to 8 are implemented.

Images