Object detection method and apparatus

Abstract

An object detection method and apparatus. The method comprises: performing depth calculation on a first image to obtain a first relative depth image, wherein the first image is an image obtained by a camera photographing a scene comprising at least one target object; converting the first relative depth image into a second relative depth image on the basis of a scale correction parameter associated with the camera; and determining position information of an obstacle in the at least one target object on the basis of the second relative depth image. In the method, the scale correction parameter associated with the camera can be used to perform scale conversion on the first relative depth image provided by a general model to obtain the second relative depth image, so that point cloud data corresponding to the second relative depth image can be used to perform obstacle detection, thereby reducing the degree to which detection results are affected by intrinsic and extrinsic parameters of cameras, and improving the accuracy and universality of detection of objects in images.

Description

Object detection method and device

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. 202411292839.7, filed on September 13, 2024, and entitled "Object detection method and device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of detection, and in particular, to an object detection method and device. BACKGROUND

[0004] In the field of intelligent driving such as assisted driving and automatic driving, during the driving process, the current driving road conditions usually need to be detected and judged by an automatic driving system or an advanced driver-assistance system (ADAS) to realize the detection of general obstacles (or general obstacles) and effectively avoid obstacles to ensure the safety of vehicle driving.

[0005] One commonly used algorithm for obstacle detection is a monocular vision detection algorithm, which is an algorithm for general obstacle recognition based on depth estimation algorithm, so that the automatic driving system or the ADAS can make correct avoidance decisions based on the obstacle recognition result. The current monocular vision detection algorithm mainly relies on absolute depth information, for example, obstacle extraction based on absolute depth information to determine the position of the obstacle.

[0006] When applying the monocular vision detection algorithm across models / cross models, the camera deployed for different models usually produces supervised data and model training separately, which is very inefficient and redundant. Moreover, the obstacle recognition effect is also highly dependent on the quality of the produced supervised data. Since the absolute depth information is very sensitive to the camera internal and external parameters, a set of absolute depth estimation model cannot be applied across models / cross models. During the process of obstacle extraction and position determination based on absolute depth information, the error of the absolute depth information has a great impact on the calculation of the obstacle position, which seriously affects the accuracy of the position estimation of the obstacle.

[0007] Therefore, how to reduce the influence of the camera internal and external parameters on the monocular vision detection algorithm is still an important problem to be solved. SUMMARY

[0008] The present application provides an object detection method and device for improving the accuracy and universality of detecting objects in an image.

[0009] In a first aspect, the present application provides an object detection method, which can include: performing depth calculation on a first image to obtain a first relative depth image, the first image being an image captured by a camera for a scene containing at least one target object; converting the first relative depth image into a second relative depth image according to a scale correction parameter associated with the camera; and determining position information of an obstacle in the at least one target object according to the second relative depth image. Exemplarily, a computing node can perform depth calculation on the first image through a monocular depth estimation model, which can be trained based on a variety of crowd-sourced data.

[0010] Through the above method, the first relative depth image provided by a general model can be converted into a second relative depth image through a scale correction parameter associated with the camera, so that obstacle detection can be performed on the point cloud data corresponding to the second relative depth image, the degree of influence of the camera internal and external parameters on the detection result can be reduced, and the accuracy and universality of object detection in the image can be improved.

[0011] In a possible implementation, the method can further include: performing depth calculation on a second image to obtain a third relative depth image, the second image being an image captured by the camera for a scene containing at least one target object; and determining the scale correction parameter associated with the camera according to supervised data corresponding to the third relative depth image and the second image. Exemplarily, the supervised data corresponding to the second image can include absolute depth information corresponding to the second image. That is, the absolute depth information corresponding to the second image captured by the camera is used as supervised data (or true value data) to obtain the scale correction parameter associated with the camera. In a specific implementation, the supervised data can be a small amount of data, for example, absolute depth information corresponding to tens to hundreds of frames of the second image. The present application does not limit the obtaining manner of the supervised data.

[0012] Through the above method, the scale correction parameter associated with the camera can be obtained through the supervised data, so that the scaleless relative depth image can be converted into a measurable relative depth image through the scale correction parameter, and obstacle recognition can be performed in the point cloud data corresponding to the measurable relative depth image.

[0013] In an example, the determining of the position information of the obstacle in the at least one target object according to the second relative depth image includes: performing obstacle extraction in the point cloud data corresponding to the second relative depth image to determine a grounding point of the obstacle in the at least one target object; determining a pixel point corresponding to the grounding point of the obstacle in the first image; and determining the position information of the obstacle according to a position point of the pixel point after being mapped to a target imaging plane.

[0014] In another example, the determining the position information of the obstacle in the at least one target object according to the second relative depth image comprises: performing obstacle extraction on point cloud data corresponding to the second relative depth image to determine an equivalent vertex of a height of the obstacle in the at least one target object in the point cloud data; determining a pixel point corresponding to the equivalent vertex in the first image; and determining the position information of the obstacle according to a position point of the pixel point after being mapped to a target imaging plane.

[0015] In another example, the determining the position information of the obstacle in the at least one target object according to the second relative depth image comprises: performing obstacle extraction on point cloud data corresponding to the second relative depth image to determine a height variation of the at least one target object; and determining the position information of the obstacle according to the height variation of the at least one target object.

[0016] In a possible implementation, the performing obstacle extraction on the point cloud data corresponding to the second relative depth image comprises: rasterizing the point cloud data corresponding to the second relative depth image, and analyzing a height variation trend of point cloud of each grid and a height variation trend of point cloud of adjacent grids to perform obstacle extraction.

[0017] In a possible implementation, the rasterizing the point cloud data corresponding to the second relative depth image comprises: rasterizing the point cloud data corresponding to the second relative depth image according to a first resolution.

[0018] In a possible implementation, the determining the position information of the obstacle in the at least one target object according to the second relative depth image comprises: converting the second relative depth image into an absolute depth image; and performing obstacle extraction on point cloud data corresponding to the absolute depth image to determine the position information of the obstacle.

[0019] In a possible implementation, the performing depth calculation on the first image comprises:

[0020] The monocular depth estimation model is used to perform depth calculation on the first image.

[0021] In a second aspect, the present application provides an object detection device, comprising: a depth estimation unit configured to perform depth calculation on a first image to obtain a first relative depth image, the first image being an image captured by a camera on a scene containing at least one target object; a correction unit configured to convert the first relative depth image into a second relative depth image according to a scale correction parameter associated with the camera; and a detection unit configured to determine position information of an obstacle in the at least one target object according to the second relative depth image.

[0022] In a possible implementation, the depth estimation unit is further configured to: perform depth calculation on a second image to obtain a third relative depth image, the second image being an image captured by the camera for a scene containing at least one target object; and determine a scale correction parameter associated with the camera according to the third relative depth image and supervision data corresponding to the second image.

[0023] In a possible implementation, the detection unit is specifically configured to: perform obstacle extraction in point cloud data corresponding to the second relative depth image to determine a grounding point of an obstacle in the at least one target object; determine, in the second relative depth image, a pixel point corresponding to the grounding point of the obstacle; and determine position information of the obstacle according to a position point of the pixel point after being mapped to a target imaging plane.

[0024] In a possible implementation, the detection unit is specifically configured to: perform obstacle extraction in point cloud data corresponding to the second relative depth image to determine an equivalent vertex of a height of an obstacle in the at least one target object in the point cloud data; determine, in the second relative depth image, a pixel point corresponding to the equivalent vertex; and determine position information of the obstacle according to a position point of the pixel point after being mapped to a target imaging plane.

[0025] In a possible implementation, the detection unit is specifically configured to: perform obstacle extraction in point cloud data corresponding to the second relative depth image to determine a height variation of the at least one target object; and determine position information of the obstacle according to the height variation of the at least one target object.

[0026] In a possible implementation, the detection unit is specifically configured to: rasterize the point cloud data corresponding to the second relative depth image, and analyze a height variation trend of point cloud of each grid and a height variation trend of point cloud of adjacent grids to perform obstacle extraction.

[0027] In a possible implementation, the detection unit is specifically configured to: rasterize the point cloud data corresponding to the second relative depth image at a first resolution.

[0028] In a possible implementation, the detection unit is specifically configured to: convert the second relative depth image into an absolute depth image; and perform obstacle extraction based on point cloud data corresponding to the absolute depth image to determine position information of the obstacle.

[0029] In a possible implementation, the depth estimation unit is specifically configured to: perform depth calculation on the first image by using a monocular depth estimation model.

[0030] In a third aspect, the present application provides a communication apparatus, comprising at least one processor and an interface circuit, the interface circuit being configured to provide data or code instructions for the at least one processor, and the at least one processor being configured to implement the method according to the first aspect and any possible implementation of the first aspect.

[0031] In a fourth aspect, the present application provides a vehicle, comprising a unit configured to implement the method according to the first aspect and any possible implementation of the first aspect.

[0032] In a fifth aspect, the present application provides a computer readable storage medium, which stores program codes, and the program codes, when executed on a computer, cause the computer to perform the method according to the first aspect and any possible implementation of the first aspect.

[0033] In a sixth aspect, the present application provides a computer program product, which, when executed on a computer, cause the computer to perform the method according to the first aspect and any possible implementation of the first aspect.

[0034] In a seventh aspect, the present application provides a terminal device, comprising a unit configured to implement the method according to the first aspect and any possible implementation of the first aspect. Exemplarily, the terminal device includes, but is not limited to, intelligent transportation devices (such as cars, ships, drones, trains, trucks, etc.), intelligent manufacturing devices (such as robots, industrial devices, intelligent logistics, intelligent factories, etc.), intelligent terminals (mobile phones, computers, tablets, palmtop computers, desktop computers, earphones, sound systems, wearable devices, vehicle-mounted devices, etc.). Based on the implementation provided in the above aspects, the present application can be further combined to provide more implementations.

[0035] The technical effects that can be achieved by any possible implementation of any one of the second aspect to the seventh aspect can be described with reference to the technical effects that can be achieved by any possible implementation of the first aspect, and the repeated parts will not be discussed. BRIEF DESCRIPTION OF DRAWINGS

[0036] FIG. 1 shows a schematic diagram of a communication system to which the embodiments of the present application are applicable;

[0037] FIG. 2 shows a flowchart of an object detection method according to an embodiment of the present application;

[0038] FIG. 3 shows a schematic diagram of a principle of obtaining a scale correction parameter according to an embodiment of the present application;

[0039] FIGS. 4-5 show schematic diagrams of an object detection principle according to an embodiment of the present application;

[0040] FIG. 6 and FIG. 7 show schematic diagrams of different examples of obstacle detection according to embodiments of the present application;

[0041] FIG. 8 shows a flowchart of an object detection method according to embodiments of the present application;

[0042] FIG. 9 shows a schematic diagram of a communication device according to embodiments of the present application;

[0043] FIG. 10 shows a schematic diagram of another communication device according to embodiments of the present application. DETAILED DESCRIPTION

[0044] The object detection method provided by the embodiments of the present application can be applied to various target detection fields, such as security, image processing, video processing, land transportation, sea transportation, unmanned aerial vehicle monitoring, navigation, smart home, and autonomous driving, etc. The following will be described taking the autonomous driving scenario as an example.

[0045] In the field of autonomous driving, the driving safety of autonomous vehicles is very important. In the driving process, the autonomous vehicle generally needs to use an autonomous driving system or an ADAS system to detect and judge the current road conditions in a timely manner, so as to effectively avoid obstacles and ensure driving safety. However, due to the complexity of the road conditions and the diversity of the participants in the road, it is currently difficult to achieve high-precision detection of general obstacles by a set of solutions. Unreliable detection results can easily cause great safety hazards to autonomous driving. For example, false detection can affect the decision of the ADAS system, and then cause the frequent triggering of the vehicle's forward collision warning system (FCWS), resulting in reduced driving efficiency, and missed detection can easily cause serious vehicle collision accidents.

[0046] The existing obstacle detection methods in the field of autonomous driving mainly include detection methods based on monocular cameras, binocular cameras, radars, lidars, etc. Taking a monocular vision detection method based on a monocular camera as an example, the following two methods are included:

[0047] Method 1: An absolute depth estimation network model is used to obtain absolute depth three-dimensional point cloud data, and a three-dimensional point cloud data processing algorithm is used to extract obstacles to obtain the position information of the obstacles.

[0048] The method relies on absolute depth information, which is very sensitive to camera internal and external parameters. In cross-model / cross-vehicle applications, production supervision data and model training are usually performed separately according to the cameras deployed in different vehicle models, which is very inefficient and redundant. Moreover, the obstacle recognition effect is also highly dependent on the quality of the production supervision data. Therefore, a set of absolute depth estimation model cannot realize cross-model / cross-vehicle applications. Moreover, in the process of obstacle extraction and position determination through absolute depth information, the error of absolute depth information has a great influence on the calculation of obstacle position, which seriously affects the accuracy of the estimation of obstacle position.

[0049] Method 2: Obtain a relative depth image by using a relative depth estimation network model, obtain the scale and translation parameters of the relative depth image to the absolute depth image by using the ground part in the relative depth image, convert the relative depth image into an absolute depth image, and then obtain an absolute depth three-dimensional point cloud, extract an obstacle based on the absolute depth three-dimensional point cloud, and obtain the position of the obstacle.

[0050] In this method, additional techniques are needed for ground extraction. Moreover, using the ground part of the relative depth image to estimate the scale and translation parameters will introduce new errors, and the relative depth image still needs to be converted into the corresponding point cloud of the absolute depth image, and the position of the obstacle is greatly affected by the error of the absolute depth point cloud.

[0051] In summary, the existing technical solutions have the problems of poor universality and low accuracy in obstacle detection.

[0052] To solve the above problems, the embodiments of the present application provide an object detection method and device, which are used to improve the accuracy and universality of object detection in images. The method and the device are based on the same technical concept. Since the principles of the method and the device for solving problems are similar, the implementation of the device and the method can be mutually referred to, and the repeated parts will not be described again. Moreover, in each embodiment of the present application, if there is no special description and logical conflict, the terms and / or descriptions of each embodiment are consistent and can be mutually referred to. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0053] It should be noted that the object detection scheme in the embodiments of the present application can be applied to the Internet of Vehicles, such as vehicle to everything (V2X), long term evolution-vehicle (LTE-V), vehicle to vehicle (V2V), etc. For example, it can be applied to a vehicle with a driving mobile function, or other devices with a driving mobile function in the vehicle. The other devices include, but are not limited to, a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip, a vehicle-mounted unit, a vehicle-mounted radar, or a vehicle-mounted camera and other sensors. The vehicle can implement the perception method provided by the present application through the vehicle-mounted terminal, vehicle-mounted controller, vehicle-mounted module, vehicle-mounted module, vehicle-mounted component, vehicle-mounted chip, vehicle-mounted unit, vehicle-mounted radar, or vehicle-mounted camera. Of course, the control scheme in the embodiments of the present application can also be used for other intelligent terminals with a mobile control function other than vehicles, or be arranged in other intelligent terminals with a mobile control function other than vehicles, or be arranged in components of the intelligent terminals. The intelligent terminal can be an intelligent transportation device, an intelligent home device, a robot, etc. For example, it includes, but is not limited to, an intelligent terminal or a controller, a chip, a radar, or a camera and other sensors, and other components in the intelligent terminal.

[0054] It should be noted that "at least one" in the embodiments of the present application means one or more, and "multiple" means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the following cases: A exists alone, A and B exist together, B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including single item or any combination of multiple items. For example, at least one of a, b, or c, can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0055] In addition, unless otherwise specified, the ordinal numbers "first", "second", etc. mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the priority or importance of the multiple objects. For example, the first image and the second image are only used to distinguish different images, and do not represent the difference in priority or importance of the two images.

[0056] For ease of understanding, the following will be introduced in conjunction with the drawings and embodiments.

[0057] FIG. 1 shows a schematic diagram of a communication system to which embodiments of the present application are applicable. In the communication system, a sensor 110 and a computing node 120 can be included. The computing node can also be referred to as a computing platform, which can include at least one processor 121 that can execute instructions 123 stored in a non-transitory computer readable medium such as a memory 122. In addition to the instructions 123, the memory 122 can also store data.

[0058] The sensor 110 can be, for example, an image acquisition device such as a camera. The computing node 120 can be an object detection device with data processing capability, which can perform object detection in combination with images acquired by the image acquisition device. The image acquisition device can be integrated into the object detection device, or the image acquisition device can transmit images to the object detection device in real time. In addition, the image acquisition device can also detect images in multimedia files stored in the memory 122.

[0059] In different application scenarios, the product form or deployment manner of the sensor 110 and the computing node 120 can be different.

[0060] For example, the sensor 110 can be an image acquisition device such as a camera, a monocular camera, a binocular camera, a near-infrared camera, a video camera, a cabin camera, a vehicle event data recorder (i.e., a video terminal), a reversing image camera, etc., which is not specifically limited in the embodiments of the present application. For example, in the field of intelligent driving such as autonomous driving and assisted driving, the image acquisition device can capture an external environment of a vehicle. For example, when the vehicle is moving forward, the capturing area can be the area in front of the vehicle head; when the vehicle is reversing, the capturing area can be the area behind the vehicle tail; when the image acquisition device is a 360-degree multi-angle camera, the capturing area can be a 360-degree area around the vehicle, etc.

[0061] For example, the computing node 120 can be a standalone device such as a roadside unit or a cloud server.

[0062] For example, the computing node 120 can be a chip or a component in a vehicle, which can also be a software module, and can be deployed on a related vehicle-mounted device of the vehicle, or can be deployed on other types of terminal devices.

[0063] As an example but not limitation, the computing node 120 can be a vehicle with data processing function, or a vehicle-mounted device with data processing function in the vehicle, or be arranged in a sensor with image acquisition and processing function. The vehicle-mounted device can include but is not limited to a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip, a vehicle-mounted unit, an electronic control unit (ECU), a domain controller (DC), and the like.

[0064] In some embodiments, the computing node can be implemented as a computing platform of a vehicle, which can also be a plurality of computing devices that control individual components or subsystems of the vehicle in a distributed manner. The processor 121 can be any conventional processor, such as a central processing unit (CPU). Alternatively, the processor 121 can also include a graphic process unit (GPU), a field programmable gate array (FPGA), a system on chip (SoC), an application specific integrated circuit (ASIC), or a combination thereof. The sensor 110 can be arranged on the same vehicle as the computing node 120, or on a different vehicle.

[0065] Optionally, the data stored in the memory 122 can include, for example, a road map, route information, a position, a direction, a speed of the vehicle, and other such vehicle data, and other information. Such information can be used by the vehicle and the computing platform during operation of the vehicle in an autonomous mode, a semi-autonomous mode, and / or a manual mode to assist the vehicle in achieving autonomous driving or intelligent driving.

[0066] In addition, the computing node 120 can also be other electronic devices with data processing function, including but not limited to smart home devices (such as televisions, etc.), smart robots, mobile terminals (such as mobile phones, tablets, etc.), wearable devices (such as smart watches, etc.), and other smart devices. The object detection device can also be a controller, a chip, or other devices in the smart device.

[0067] The object detection method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It can be understood that the following described embodiments are only some of the embodiments of the present application, not all the embodiments.

[0068] FIG. 2 is a schematic diagram of an object detection method according to an embodiment of the present application. With reference to FIG. 2, the object detection method according to an embodiment of the present application will be described in detail. The object detection method can be implemented by the computing node 120 in FIG. 1, and can include the following steps:

[0069] S210: The computing node performs depth calculation on the first image to obtain a first relative depth image.

[0070] In the embodiment of the present application, when S210 is implemented, the computing node can perform depth calculation on the first image by using a depth estimation model to obtain the first relative depth image. The depth estimation model is a machine vision model, for example, a deep learning model, which can be used to estimate the distance information from the surface of an object in a scene to a camera from a single frame of image. The distance information is represented in the form of a depth map, in which the value of each point represents the depth value of the corresponding pixel. For example, the depth estimation model can be a neural network model, such as a convolutional neural network (CNN) or a feedforward neural network, and the specific implementation of the model is not limited in the embodiment of the present application.

[0071] In a specific implementation, for example, the depth estimation model described above can be a monocular depth estimation model, including a monocular relative depth estimation model or a monocular absolute depth estimation model. The monocular relative depth estimation model can be used to obtain relative depth estimation information corresponding to an image, and the monocular absolute depth estimation model can be used to obtain absolute depth estimation information corresponding to an image. In one example, when S210 is implemented, the computing node can perform depth calculation on the first image by using a monocular relative depth estimation model to obtain the first relative depth image. The monocular depth models described below are all monocular relative depth estimation models, which will not be distinguished and described in detail.

[0072] The monocular depth estimation model can be pre-installed on the computing node, and the monocular depth estimation model can be obtained by training according to the business data set and the public data set of the cameras of various vehicle models or various types. The source of the training data is various. For example, the training data can be obtained from different vehicle manufacturers or different individual users after authorization. The monocular depth estimation model obtained by training is not limited by the camera internal and external parameters, and the relative depth image or the absolute depth image obtained based on the monocular depth estimation model is not sensitive to the camera internal and external parameters. The training process of the monocular depth estimation model is not limited in the embodiment of the present application.

[0073] The first image can be an image, such as an RGB image, captured by a camera pair of a scene containing at least one target object. As an example, the camera can be deployed on a vehicle, and the first image can include a captured region of the camera. For example, if the camera is deployed at the front of the vehicle, the first image can be a captured image of a region in front of the vehicle. Or for example, if the camera is deployed at the rear of the vehicle, the first image can be a captured image of a region behind the vehicle. If the camera is a 360-degree multi-angle camera on the vehicle, the first image can be a captured image of a region in a 360-degree range around the vehicle.

[0074] If the camera and the computing node are deployed on the same vehicle, before S210 is implemented, the camera can transmit the first image to the computing node in real time, and accordingly, the computing node can receive the first image from the camera. If the camera and the computing node are deployed on different vehicles, or the computing node is deployed on a roadside unit or a cloud server, before S210 is implemented, the camera can transmit the first image to the computing node through wireless communication technology. Accordingly, the computing node can receive the first image from the camera. In another example, the first image can also be stored in a storage medium accessible to the computing node, and before S210 is implemented, the computing node can read the first image from the storage medium. Embodiments of the present application do not make specific limitations on the manner of obtaining the first image.

[0075] S220: The computing node converts the first relative depth image into a second relative depth image according to the scale correction parameter associated with the camera.

[0076] In embodiments of the present application, the scale correction parameter associated with the camera can be used to represent the scale and translation parameters of the relative depth information of the image provided by the camera relative to the absolute depth information.

[0077] Both the relative depth image and the absolute depth image can be converted into corresponding point cloud data. The point cloud data corresponding to the relative depth image obtained by depth estimation based on the original captured image is scale-free, and cannot be directly applied to the absolute depth point cloud processing algorithm with fixed scale. The point cloud data corresponding to the absolute depth image obtained by depth estimation based on the original captured image is scale-dependent, but sensitive to the internal and external parameters of the camera. By using the scale correction parameter associated with the camera, the relative depth image obtained by the monocular depth estimation model can be corrected in scale and translation, and further, the scale-free relative depth point cloud can be converted into a relative depth point cloud with a scale similar to that of the absolute depth point cloud, which is called a measurable relative depth point cloud. Although there is an error between the measurable relative depth point cloud and the absolute depth point cloud, it is still applicable to various algorithms for processing the absolute depth point cloud. When obstacle detection is performed based on the measurable relative depth point cloud, the error of the scale correction parameter is not sensitive, which helps to ensure the accuracy of the obstacle detection result.

[0078] Before implementing S220, the computing node can obtain supervision data corresponding to the second image. As an example, the supervision data corresponding to the second image can include absolute depth information corresponding to the second image. When implementing S220, the computing node can use the absolute depth information corresponding to the second image acquired by the camera as supervision data (or true value data) to obtain the scale correction parameter associated with the camera. In a specific implementation, the second image can include an image, such as an RGB image, taken by the camera of a scene containing at least one target object. The number of second images can be a small data set, such as tens to hundreds of frames of images. The supervision data can be a small data set with true value of absolute depth information, such as tens to hundreds of frames of absolute depth information corresponding to the second image. The supervision data can label the position information of the obstacle in the at least one target object. The computing node can perform depth calculation on the second image to obtain a third relative depth image, and determine the scale correction parameter associated with the camera according to the third relative depth image and the supervision data corresponding to the second image. The scale correction parameter can also be referred to as a scale balancing parameter between the monocular depth estimation model and the camera.

[0079] As an example but not limitation, the computing node can obtain the scale balancing parameter between the monocular depth estimation model and the camera by using a least square method or the like. As shown in FIG. 3, M0 represents the supervision data corresponding to the second image. The second image can obtain a third relative depth image Mr after being input into the monocular depth estimation model. The mapping relationship between M0 and Mr can satisfy the following expression (1) by using the least square method: M0=a*Mr+b; (1)

[0080] Wherein, the scale correction parameter associated with the camera includes: a, b. a and b respectively represent the scale parameter and the translation parameter between the third relative depth image and the supervision data corresponding to the second image.

[0081] The computing node can initialize the monocular depth estimation model by using the scale correction parameter associated with the camera. Subsequently, after the monocular depth estimation model normally runs, the scale correction parameter can be used for the model output to obtain the measurable point cloud data of the relative depth image. As shown in FIG. 4, the first relative depth image can be obtained after the monocular depth estimation model is used to perform depth calculation on the first image. The second relative depth image can be obtained after the first relative depth image is processed by using the scale correction parameter associated with the camera. The point cloud data corresponding to the second relative depth image is the measurable relative depth point cloud data.

[0082] It should be understood that the monocular depth estimation model of the embodiments of the present application is a general model suitable for the computing device or equipment associated with the cameras of various types or models. The training data is more abundant, and better point cloud effects can be obtained. When the monocular depth estimation model is applied to other computing devices or equipment, the monocular depth estimation model can be initialized in the same way, and details are not described herein.

[0083] S230: The computing node determines the position information of the obstacle in the at least one target object according to the second relative depth image.

[0084] In the embodiments of the present application, when S230 is implemented, the second relative depth image can be converted into corresponding point cloud data, i.e., the relative depth point cloud data can be measured. Since the measurable relative depth point cloud data is close to the absolute depth point cloud data in the order of magnitude scale, the same processing method as the absolute depth point cloud data can be used for obstacle detection.

[0085] For example, the computing node can extract the obstacle contact point and the vertical contact point of the floating obstacle in the measurable relative depth point cloud data, and then find the corresponding pixels of the obstacle contact point and the vertical contact point of the floating obstacle in the first image. As shown in FIG. 5, the corresponding pixels of the obstacle contact point in the first image are represented by a dashed line. Further, the computing node can calculate the position of the obstacle contact point in the ego vehicle coordinate system by using the contact point assumption method, i.e., obtain the position information of the obstacle.

[0086] When S230 is implemented, the computing node extracts the obstacle contact point and the vertical contact point of the floating obstacle in the point cloud data corresponding to the second relative depth image and realizes the positioning detection of the obstacle, which can specifically include the following examples:

[0087] Example 1: The computing node extracts the obstacle in the point cloud data corresponding to the second relative depth image to determine the contact point of the obstacle in the at least one target object; in the first image, determines the pixel point corresponding to the contact point of the obstacle; according to the position point of the pixel point after mapping to the target imaging plane, determines the position information of the obstacle.

[0088] Let B' represent the obstacle contact point or the vertical contact point of the floating obstacle in the point cloud data corresponding to the second relative depth image, and the pixel point corresponding to the contact point can be found in the first image, and converted into a point A(X A ,Y A ,Z A ) on the normalized imaging plane. As shown in FIG. 6, taking the camera optical axis as the origin, it is represented as the camera optical center O(X O ,Y O ,Z O), the actual grounding point corresponding to A in the ego-vehicle coordinate system is represented as B(X B ,Y B ,0). The following expression (2) is obtained by using the triangle similarity relationship, and the coordinates (X B ,Y B ) of the grounding point B in the ego-vehicle coordinate system are further obtained.

[0089] Based on the coordinates (X B ,Y B ) of the grounding point B in the ego-vehicle coordinate system, the computing node can know whether there is an obstacle in the front or rear of the vehicle.

[0090] Example 2: The computing node performs obstacle extraction in the point cloud data corresponding to the second relative depth image to determine the equivalent vertex of the obstacle in the at least one target object in the point cloud data; in the first image, a pixel point corresponding to the equivalent vertex is determined; and according to a position point of the pixel point after being mapped to a target imaging plane, position information of the obstacle is determined.

[0091] The equivalent vertex of the obstacle in the point cloud data corresponding to the second relative depth image is represented as C', and Zc represents the obstacle height. The physical meaning of the equivalent vertex is consistent with the grounding point position. In the first image, a pixel point corresponding to the equivalent vertex can be found, and the pixel point is converted into a point D(X D ,Y D ,Z D ) on a normalized imaging plane. As shown in FIG. 7, taking the camera optical axis as the origin, represented as the camera optical center O(X O ,Y O ,Z O ), the equivalent vertex of the obstacle corresponding to D in the ego-vehicle coordinate system is C(X C ,Y C ,Z C ). The following expression (3) is obtained by using the triangle similarity relationship, and the height Z C of the obstacle in the ego-vehicle coordinate system is further obtained.

[0092] Based on the height Z C of the obstacle in the ego-vehicle coordinate system, the computing node can know whether the obstacle in the front or rear of the vehicle will affect driving.

[0093] Example 3: The computing node can simultaneously determine the grounding point and the equivalent vertex of the obstacle in the point cloud data corresponding to the second relative depth image, so as to determine, based on the coordinates (X B ,Y B ) of the grounding point B of the obstacle in the ego-vehicle coordinate system and the height Z C, determine whether the front or rear obstacle will affect driving. The method of obstacle detection based on the ground point and the equivalent vertex can be referred to the introduction of example 1 and example 2, and will not be repeated here.

[0094] Example 4: The computing node performs obstacle extraction in the point cloud data corresponding to the second relative depth image to determine the height change of at least one target object; and determines the position information of the obstacle according to the height change of the at least one target object.

[0095] In example 4, the height change of the at least one target object can be referred to the introduction of analyzing the height of the obstacle in the ego vehicle coordinate system in example 2, and will not be repeated here.

[0096] In an optional implementation, in examples 1-4 above, before performing obstacle extraction on the point cloud data corresponding to the second relative depth image, the computing node can also rasterize the point cloud data and analyze the height change trend of the point cloud of each grid and the change trend of the point cloud of adjacent grids to perform obstacle extraction. Specifically, the computing node can rasterize the point cloud data corresponding to the second relative depth image according to a first resolution (for example, 10). The resolution used for rasterization in the embodiments of the present application is not specifically limited.

[0097] Example 5: The computing node converts the second relative depth image into an absolute depth image; and performs obstacle extraction based on the point cloud data corresponding to the absolute depth image to determine the position information of the obstacle. For example, the ground point, the equivalent vertex, the height change and other information of the obstacle are extracted in the point cloud data corresponding to the absolute depth image, and then the corresponding pixel points in the first image are found, so as to obtain the position information of the obstacle.

[0098] In example 5, by performing obstacle detection on the absolute depth image converted from the relative depth image, the association with the camera internal and external parameters can be eliminated, and the accuracy of obstacle detection can be improved.

[0099] Therefore, by using the above method, the monocular depth estimation module and the scale correction parameter associated with the camera can obtain measurable relative depth point cloud data, and the obstacle detection based on the relative depth point cloud data can reduce the influence of the detection result on the camera internal and external parameters, and can improve the accuracy and universality of the object detection in the image.

[0100] In order to facilitate understanding, the following describes a complete process of the object detection method of the embodiments of the present application by taking the obstacle detection method of example 1 as an example.

[0101] As shown in FIG. 8, the method can include the following steps:

[0102] S801: The computing node obtains a first image, inputs the first image as input information into a monocular depth estimation model, performs depth calculation on the first image through the monocular depth estimation model, and obtains a first relative depth image, denoted as M1.

[0103] S802: The computing node converts the first relative depth image into a second relative depth image, denoted as M2, through the monocular depth estimation model according to the scale correction parameters associated with the relative depth image. M1 and M2 satisfy the following expression (4): M2 = a * M1 + b; (4)

[0104] Wherein, a and b are scale correction parameters associated with the relative depth image.

[0105] S803: The computing node converts the second relative depth image into corresponding point cloud data through the monocular depth estimation model.

[0106] S804: The computing node rasterizes the point cloud data corresponding to the second relative depth image according to a first resolution (for example, 10), analyzes the height variation trend of each grid of point cloud and the variation trend of adjacent grid point cloud, extracts the obstacle contact point or the vertical contact point of the suspended obstacle, and denotes it as B'.

[0107] S805: The computing node can perform contact point ranging and positioning based on the first image to obtain obstacle position information.

[0108] For example, the computing node can find the pixel point corresponding to the contact point in the first image, and convert it into a point A (X A ,Y A ,Z A ) on the normalized imaging plane. Taking the camera optical axis as the origin, denoted as the camera optical center O (X O ,Y O ,Z O ), the actual contact point corresponding to A in the ego vehicle coordinate system is denoted as B (X B ,Y B ,0). Using the triangle similarity relationship and the above expression (2), the coordinates (X B ,Y B ) of the contact point B in the ego vehicle coordinate system are further obtained.

[0109] Optionally, between S804 and S805, the computing node can also determine the height of the obstacle in the point cloud data to obtain the equivalent vertex of the obstacle. The pixel point corresponding to the equivalent vertex in the first image can be found, and it is converted into a point D (X D ,Y D ,Z D ) on the normalized imaging plane. Taking the camera optical axis as the origin, denoted as the camera optical center O (X OY O Z O ), the equivalent vertex of the obstacle corresponding to D in the ego-vehicle coordinate system is C(X C Y C Z C ). By using the triangle similarity relationship and the above expression (3), the height Z C of the obstacle in the ego-vehicle coordinate system is further obtained. The computing node can combine the coordinates (X B Y B ) of the grounding point B in the ego-vehicle coordinate system and the height Z C of the obstacle in the ego-vehicle coordinate system to analyze whether the obstacle in front of or behind the vehicle will affect the driving of the vehicle.

[0110] Therefore, by using the monocular depth estimation module and the scale correction parameter associated with the camera, the above method can obtain measurable relative depth point cloud data. The obstacle detection based on the relative depth point cloud data can reduce the influence of the camera internal and external parameters on the detection result, and can improve the accuracy and universality of the detection of objects in the image.

[0111] The embodiment of the application further provides a communication device for executing the method performed by the computing node in the above method embodiment. The related features can be referred to the above method embodiment, and will not be described here.

[0112] As shown in FIG. 9, the communication device 900 can include a depth estimation unit 901 configured to perform depth calculation on a first image to obtain a first relative depth image, the first image being an image of a scene containing at least one target object taken by a camera; a correction unit 902 configured to convert the first relative depth image into a second relative depth image according to a scale correction parameter associated with the camera; and a detection unit 903 configured to determine position information of an obstacle in the at least one target object according to the second relative depth image. For specific implementation, please refer to the method steps implemented by the computing node in the above method embodiment, which will not be described here.

[0113] It should be understood that the division of units in the above apparatus is only a logical function division, and all or part of them can be integrated into a physical entity or physically separated when actually implemented. In addition, the units in the apparatus can be implemented in the form of processor calling software; for example, the apparatus includes a processor, the processor is connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the units of the apparatus, wherein the processor is, for example, a general processor such as a central processing unit (CPU) or a microprocessor, and the memory is a memory in the apparatus or a memory outside the apparatus. Alternatively, the units in the apparatus can be implemented in the form of hardware circuit, and the functions of part or all of the units can be realized by the design of the hardware circuit, which can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of part or all of the units are realized by the design of the logical relationship of elements in the circuit; for example, in another implementation, the hardware circuit is realized by a programmable logic device (PLD), and a field programmable gate array (FPGA) is taken as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of part or all of the units. All units of the above apparatus can be realized in the form of processor calling software, or all units can be realized in the form of hardware circuit, or part of the units can be realized in the form of processor calling software, and the remaining part can be realized in the form of hardware circuit.

[0114] In embodiments of the present application, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a CPU, a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), a digital signal processor (DSP), etc. In another implementation, the processor can implement certain functions through a logical relationship of hardware circuit, which is fixed or reconfigurable, such as an ASIC or a PLD implemented hardware circuit, such as an FPGA. In a reconfigurable hardware circuit, the processor loads a configuration document to implement hardware circuit configuration. It can be understood that the processor loads instructions to implement the functions of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0115] It can be seen that each unit in the above apparatus can be one or more processors (or processing circuits) configured to implement the above methods, such as a CPU, a GPU, an NPU, a TPU, a DPU, a microprocessor, a DSP, an ASIC, an FPGA, or a combination of at least two of these processor forms.

[0116] In addition, each unit in the above apparatus can be integrated together or can be independently implemented. In one implementation, these units are integrated together to form a system-on-a-chip (SOC). The SOC can include at least one processor for implementing any of the above methods or functions of the units of the apparatus. The at least one processor can be of different types, such as a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0117] In a simple embodiment, those skilled in the art can conceive that the communication apparatus in the above embodiments can all adopt the form shown in FIG. 10.

[0118] As shown in FIG. 10, the apparatus 1000 includes at least one processor 1010 and a communication interface 1030. In an optional design, it can also include a memory 1020.

[0119] The specific connection medium between the processor 1010 and the memory 1020 is not limited in the embodiments of the present application.

[0120] In the apparatus as shown in FIG. 10, the processor 1010 can perform data transmission through the communication interface 1030 when communicating with other devices.

[0121] When the communication apparatus adopts the form as shown in FIG. 10, the processor 1010 in FIG. 10 can invoke the computer-executable instructions stored in the memory 1020, so that the apparatus 1000 can perform any of the above method embodiments.

[0122] The embodiments of the present application also relate to a chip system, which includes a processor for invoking computer programs or computer instructions stored in a memory, so that the processor performs the method of any of the above embodiments.

[0123] In a possible implementation, the processor can be coupled with the memory through an interface.

[0124] In a possible implementation, the chip system can also directly include the memory, and the memory stores computer programs or computer instructions.

[0125] By way of example, the memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example and not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0126] The embodiments of the present application also relate to a processor configured to invoke a computer program or computer instructions stored in a memory to cause the processor to perform the method of any of the above embodiments.

[0127] For example, in the embodiments of the present application, the processor is an integrated circuit chip with processing capability of signals. For example, the processor can be an FPGA, a general purpose processor, a DSP, an ASIC or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, a system on chip (SoC), a CPU, a network processor (NP), a micro controller unit (MCU), a PLD or other integrated circuits, which can realize or execute the methods, steps and logical block diagrams disclosed in the embodiments of the present application. The general purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly embodied in hardware code of the processor, or a combination of hardware and software modules in the processor. The software modules can reside in the random memory, the flash memory, the read-only memory, the programmable read-only memory, the electrically programmable read-only memory, the register, or other forms of the storage medium in the art. The storage medium is located in the storage memory, and the processor reads information in the storage memory and combines the hardware to execute the steps of the above methods.

[0128] It should be understood that the embodiments of the present application can be provided as a method, a system, or a computer program product.

[0129] In a possible implementation, the embodiments of the present application provide a computer readable storage medium, which stores program codes, and when the program codes are run on the computer, the computer executes the above method embodiments.

[0130] In a possible implementation, the embodiments of the present application provide a computer program product, and when the computer program product is run on the computer, the computer executes the above method embodiments.

[0131] Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer usable program code.

[0132] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flowsheets and / or block or blocks of the block diagrams.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheets and / or block or blocks of the block diagrams.

[0134] Obviously, persons having ordinary skill in the art can make various modifications and variations to the embodiments of the present application without departing from the scope of the present application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents. In the various embodiments of the present application, the terms and / or descriptions of the various embodiments are consistent unless specifically stated otherwise and in conflict with logic, and the terms and / or descriptions of the various embodiments can be referenced mutually, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

Claims

1. An object detection method characterized by, The method comprises: performing depth calculation on a first image to obtain a first relative depth image, the first image being an image captured by a camera for a scene containing at least one target object; converting the first relative depth image into a second relative depth image according to a scale correction parameter associated with the camera; determining position information of an obstacle in the at least one target object according to the second relative depth image.

2. The method of claim 1, wherein, The method further comprises: performing depth calculation on a second image to obtain a third relative depth image, the second image being an image captured by a camera for a scene containing at least one target object; determining a scale correction parameter associated with the camera according to supervised data corresponding to the third relative depth image and the second image.

3. The method according to claim 1 or 2, characterized in that, The determination of the position information of the obstacle in the at least one target object according to the second relative depth image comprises: obstacle extraction in point cloud data corresponding to the second relative depth image to determine a ground point of the obstacle in the at least one target object; determination of a pixel point corresponding to the ground point of the obstacle in the first image; determination of the position information of the obstacle according to a position point of the pixel point after mapping to a target imaging plane.

4. The method according to claim 1 or 2, characterized in that, The determination of the position information of the obstacle in the at least one target object according to the second relative depth image comprises: obstacle extraction in point cloud data corresponding to the second relative depth image to determine an equivalent vertex of a height of the obstacle in the at least one target object in the point cloud data; determination of a pixel point corresponding to the equivalent vertex in the first image; determination of the position information of the obstacle according to a position point of the pixel point after mapping to a target imaging plane.

5. The method according to claim 1 or 2, characterized in that, The determination of the position information of the obstacle in the at least one target object according to the second relative depth image comprises: obstacle extraction in point cloud data corresponding to the second relative depth image to determine a height variation of the at least one target object; determination of the position information of the obstacle according to the height variation of the at least one target object.

6. The method according to any one of claims 3-5, characterized in that, The obstacle extraction in the point cloud data corresponding to the second relative depth image comprises: rasterization of the point cloud data corresponding to the second relative depth image, and analysis of a height variation trend of point cloud of each grid and a height variation trend of point cloud of adjacent grids to perform obstacle extraction.

7. The method of claim 6, wherein, The rasterization of the point cloud data corresponding to the second relative depth image comprises: rasterization of the point cloud data corresponding to the second relative depth image according to a first resolution.

8. The method of claim 1 or 2, wherein, The determination of the position information of the obstacle in the at least one target object according to the second relative depth image comprises: conversion of the second relative depth image into an absolute depth image; obstacle extraction based on point cloud data corresponding to the absolute depth image to determine the position information of the obstacle.

9. The method according to any one of claims 1-8, characterized in that, The depth calculation on the first image comprises: depth calculation on the first image by a monocular depth estimation model.

10. An object detection device, characterized by, It comprises: a depth estimation unit configured to perform depth calculation on the first image to obtain a first relative depth image, the first image being an image of a scene containing at least one target object taken by a camera; a correction unit configured to convert the first relative depth image into a second relative depth image according to a scale correction parameter associated with the camera; a detection unit configured to determine position information of an obstacle in the at least one target object according to the second relative depth image.

11. A communications device, characterized by A computer program product comprising a computer readable medium storing program code which, when executed on a computer, causes the computer to carry out the method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer readable medium stores program code which, when executed on a computer, causes the computer to carry out the method of any one of claims 1-9.

13. A computer program product, characterised in that, The computer program product, when executed on a computer, causes the computer to carry out the method of any one of claims 1-9.

14. A vehicle characterized by comprising: comprising means for carrying out the method of any one of claims 1-9.

Images