Object detection method and device

Abstract

The invention discloses an object detection method and device, and the method comprises the steps: carrying out the depth calculation of a first image, obtaining a first relative depth image, and enabling a camera to shoot a scene containing at least one target object; converting the first relative depth image into a second relative depth image according to the scale correction parameter associated with the camera; and determining position information of an obstacle in at least one target object according to the second relative depth image. According to the method, scale conversion can be performed on the first relative depth image provided by the general model by using the scale correction parameter associated with the camera to obtain the second relative depth image, so that obstacle detection can be performed by using the point cloud data corresponding to the second relative depth image, and the degree of influence of internal and external parameters of the camera on a detection result can be reduced; the accuracy and universality of detecting the object in the image can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of detection, and in particular to an object detection method and device. BACKGROUND

[0002] In the field of intelligent driving such as assisted driving and automatic driving, during driving, the current driving road condition usually needs to be detected and judged by an automatic driving system or an advanced driver-assistance system (ADAS), so as to realize detection of general obstacles (or general obstructions), thereby effectively avoiding the obstacles and ensuring the safety of vehicle driving.

[0003] One commonly used algorithm for realizing obstacle detection is a monocular vision detection algorithm, which is an algorithm for general obstacle recognition based on depth estimation, so as to enable the automatic driving system or the ADAS to make correct avoidance decisions based on the obstacle recognition result. The current monocular vision detection algorithm mainly relies on absolute depth information, for example, obstacle extraction based on absolute depth information to determine the position of the obstacle.

[0004] When the monocular vision detection algorithm is applied across models / cross models, the camera deployed according to different models is usually used to produce supervised data and model training, which is very inefficient and redundant. Moreover, the obstacle recognition effect is also highly dependent on the quality of the produced supervised data. Since the absolute depth information is very sensitive to the camera internal and external parameters, a set of absolute depth estimation model cannot be applied across models / cross models. During the process of obstacle extraction and position determination by using absolute depth information, the error of the absolute depth information has a great influence on the calculation of the obstacle position, which seriously affects the accuracy of the position estimation of the obstacle.

[0005] Therefore, how to reduce the influence of the camera internal and external parameters on the monocular vision detection algorithm is still an important problem to be solved. SUMMARY

[0006] The present application provides an object detection method and device, which are used to improve the accuracy and universality of detecting objects in an image.

[0007] In a first aspect, the present application provides an object detection method, which can include: performing depth calculation on a first image to obtain a first relative depth image, the first image being an image captured by a camera for a scene containing at least one target object; converting the first relative depth image into a second relative depth image according to a scale correction parameter associated with the camera; and determining position information of an obstacle in the at least one target object according to the second relative depth image. Exemplarily, a computing node can perform depth calculation on the first image through a monocular depth estimation model, which can be trained based on a variety of crowd-sourced data.

[0008] Through the above method, the first relative depth image provided by a general model can be converted into a second relative depth image through a scale correction parameter associated with the camera, so that obstacle detection can be performed on the point cloud data corresponding to the second relative depth image, the degree of influence of the camera internal and external parameters on the detection result can be reduced, and the accuracy and universality of object detection in the image can be improved.

[0009] In a possible implementation, the method can further include: performing depth calculation on a second image to obtain a third relative depth image, the second image being an image captured by the camera for a scene containing at least one target object; and determining the scale correction parameter associated with the camera according to supervised data corresponding to the third relative depth image and the second image. Exemplarily, the supervised data corresponding to the second image can include absolute depth information corresponding to the second image. That is, the absolute depth information corresponding to the second image captured by the camera is used as supervised data (or true value data) to obtain the scale correction parameter associated with the camera. In a specific implementation, the supervised data can be a small amount of data, for example, absolute depth information corresponding to tens to hundreds of frames of the second image. The present application does not limit the obtaining manner of the supervised data.

[0010] Through the above method, the scale correction parameter associated with the camera can be obtained through the supervised data, so that the scaleless relative depth image can be converted into a measurable relative depth image through the scale correction parameter, and obstacle recognition can be performed in the point cloud data corresponding to the measurable relative depth image.

[0011] In an example, the determining of the position information of the obstacle in the at least one target object according to the second relative depth image includes: performing obstacle extraction in the point cloud data corresponding to the second relative depth image to determine a grounding point of the obstacle in the at least one target object; determining a pixel point corresponding to the grounding point of the obstacle in the first image; and determining the position information of the obstacle according to a position point of the pixel point after being mapped to a target imaging plane.

[0012] In another example, the determining the position information of the obstacle in the at least one target object according to the second relative depth image comprises: performing obstacle extraction on point cloud data corresponding to the second relative depth image to determine an equivalent vertex of a height of the obstacle in the at least one target object in the point cloud data; determining a pixel point corresponding to the equivalent vertex in the first image; and determining the position information of the obstacle according to a position point of the pixel point after being mapped to a target imaging plane.

[0013] In another example, the determining the position information of the obstacle in the at least one target object according to the second relative depth image comprises: performing obstacle extraction on point cloud data corresponding to the second relative depth image to determine a height variation of the at least one target object; and determining the position information of the obstacle according to the height variation of the at least one target object.

[0014] In a possible implementation, the performing obstacle extraction on the point cloud data corresponding to the second relative depth image comprises: rasterizing the point cloud data corresponding to the second relative depth image, and analyzing a height variation trend of point cloud of each grid and a height variation trend of point cloud of adjacent grids to perform obstacle extraction.

[0015] In a possible implementation, the rasterizing the point cloud data corresponding to the second relative depth image comprises: rasterizing the point cloud data corresponding to the second relative depth image according to a first resolution.

[0016] In a possible implementation, the determining the position information of the obstacle in the at least one target object according to the second relative depth image comprises: converting the second relative depth image into an absolute depth image; and performing obstacle extraction on point cloud data corresponding to the absolute depth image to determine the position information of the obstacle.

[0017] In a possible implementation, the performing depth calculation on the first image comprises:

[0018] The monocular depth estimation model is used to perform depth calculation on the first image.

[0019] In a second aspect, the present application provides an object detection device, comprising: a depth estimation unit configured to perform depth calculation on a first image to obtain a first relative depth image, the first image being an image captured by a camera on a scene containing at least one target object; a correction unit configured to convert the first relative depth image into a second relative depth image according to a scale correction parameter associated with the camera; and a detection unit configured to determine position information of an obstacle in the at least one target object according to the second relative depth image.

[0020] In a possible implementation, the depth estimation unit is further configured to: perform depth calculation on a second image to obtain a third relative depth image, the second image being an image captured by the camera for a scene containing at least one target object; and determine a scale correction parameter associated with the camera according to the third relative depth image and supervision data corresponding to the second image.

[0021] In a possible implementation, the detection unit is specifically configured to: perform obstacle extraction in point cloud data corresponding to the second relative depth image to determine a grounding point of an obstacle in the at least one target object; determine a pixel point corresponding to the grounding point of the obstacle in the second relative depth image; and determine the position information of the obstacle according to a position point of the pixel point after being mapped to a target imaging plane.

[0022] In a possible implementation, the detection unit is specifically configured to: perform obstacle extraction in point cloud data corresponding to the second relative depth image to determine an equivalent vertex of a height of an obstacle in the at least one target object in the point cloud data; determine a pixel point corresponding to the equivalent vertex in the second relative depth image; and determine the position information of the obstacle according to a position point of the pixel point after being mapped to a target imaging plane.

[0023] In a possible implementation, the detection unit is specifically configured to: perform obstacle extraction in point cloud data corresponding to the second relative depth image to determine a height variation of the at least one target object; and determine the position information of the obstacle according to the height variation of the at least one target object.

[0024] In a possible implementation, the detection unit is specifically configured to: rasterize the point cloud data corresponding to the second relative depth image, and analyze a height variation trend of point cloud of each grid and a height variation trend of point cloud of adjacent grids to perform obstacle extraction.

[0025] In a possible implementation, the detection unit is specifically configured to: rasterize the point cloud data corresponding to the second relative depth image at a first resolution.

[0026] In a possible implementation, the detection unit is specifically configured to: convert the second relative depth image into an absolute depth image; and perform obstacle extraction based on point cloud data corresponding to the absolute depth image to determine the position information of the obstacle.

[0027] In a possible implementation, the depth estimation unit is specifically configured to: perform depth calculation on the first image by using a monocular depth estimation model.

[0028] In a third aspect, the present application provides a communication apparatus, comprising at least one processor and an interface circuit, the interface circuit being configured to provide data or code instructions for the at least one processor, and the at least one processor being configured to implement the method according to the first aspect and any possible implementation of the first aspect.

[0029] In a fourth aspect, the present application provides a vehicle, comprising a unit configured to implement the method according to the first aspect and any possible implementation of the first aspect.

[0030] In a fifth aspect, the present application provides a computer readable storage medium, which stores program codes, and when the program codes are run on a computer, the computer is caused to execute the method according to the first aspect and any possible implementation of the first aspect.

[0031] In a sixth aspect, the present application provides a computer program product, and when the computer program product is run on a computer, the computer is caused to execute the method according to the first aspect and any possible implementation of the first aspect.

[0032] In a seventh aspect, the present application provides a terminal device, comprising a unit configured to implement the method according to the first aspect and any possible implementation of the first aspect. Exemplarily, the terminal device includes, but is not limited to, intelligent transportation devices (such as cars, ships, drones, trains, trucks, etc.), intelligent manufacturing devices (such as robots, industrial devices, intelligent logistics, intelligent factories, etc.), intelligent terminals (mobile phones, computers, tablets, palmtop computers, desktop computers, earphones, sound systems, wearable devices, vehicle-mounted devices, etc.). Based on the implementation provided in the above aspects, the present application can be further combined to provide more implementations.

[0033] The technical effects that can be achieved by any possible implementation of any one of the second aspect to the seventh aspect can be described with reference to the technical effects that can be achieved by any possible implementation of the first aspect, and the repeated parts will not be discussed. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A schematic diagram of a communication system to which embodiments of the present application are applicable is shown;

[0035] Figure 2 A flowchart of an object detection method according to an embodiment of the present application is shown;

[0036] Figure 3 An acquisition principle diagram of a scale correction parameter according to an embodiment of the present application is shown;

[0037] Figures 4-5 FIG. 1 shows a schematic diagram illustrating the principle of object detection according to an embodiment of the present application;

[0038] Figure 6 and Figure 7 FIG. 2 shows a schematic diagram illustrating the principle of different examples of obstacle detection according to an embodiment of the present application;

[0039] Figure 8 FIG. 3 shows a flowchart illustrating the method of object detection according to an embodiment of the present application;

[0040] Figure 9 FIG. 4 shows a schematic diagram illustrating the structure of a communication device according to an embodiment of the present application;

[0041] Figure 10 FIG. 5 shows a schematic diagram illustrating the structure of another communication device according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] The object detection method provided by the embodiments of the present application can be applied to various target detection fields, such as security, image processing, video processing, land transportation, sea transportation, unmanned aerial vehicle monitoring, navigation, smart home, and autonomous driving, etc. The following will be described taking the autonomous driving scenario as an example.

[0043] In the field of autonomous driving, the driving safety of autonomous vehicles is very important. In the driving process, the autonomous vehicle generally needs to use an autonomous driving system or an ADAS system to detect and judge the current road conditions in a timely manner, so as to effectively avoid obstacles and ensure driving safety. However, due to the complexity of the road conditions and the diversity of the participants in the road, it is currently difficult to achieve high-precision detection of general obstacles by a set of solutions. Unreliable detection results can easily cause great safety hazards to autonomous driving. For example, false detection can affect the decision of the ADAS system, and then cause the frequent triggering of the vehicle's forward collision warning system (FCWS), resulting in reduced driving efficiency, and missed detection can easily cause serious vehicle collision accidents.

[0044] The existing obstacle detection methods in the field of autonomous driving mainly include detection methods based on monocular cameras, binocular cameras, radars, lidars, etc. Taking the monocular vision detection method based on a monocular camera as an example, the following two methods are included:

[0045] Method 1: obtaining absolute depth three-dimensional point cloud data by using an absolute depth estimation network model, and extracting obstacles by using a three-dimensional point cloud data processing algorithm to obtain the position information of the obstacles.

[0046] The method relies on absolute depth information, which is very sensitive to camera internal and external parameters. In cross-model / cross-vehicle applications, production supervision data and model training are usually performed separately according to the cameras deployed in different vehicle models, which is very inefficient and redundant. Moreover, the obstacle recognition effect is also highly dependent on the quality of the production supervision data. Therefore, a set of absolute depth estimation model cannot realize cross-model / cross-vehicle applications. Moreover, in the process of obstacle extraction and position determination through absolute depth information, the error of absolute depth information has a great influence on the calculation of obstacle position, which seriously affects the accuracy of the estimation of obstacle position.

[0047] Method 2: Obtain a relative depth image by using a relative depth estimation network model, obtain the scale and translation parameters of the relative depth image to the absolute depth image by using the ground part in the relative depth image, convert the relative depth image into an absolute depth image, and then obtain an absolute depth three-dimensional point cloud, extract an obstacle based on the absolute depth three-dimensional point cloud, and obtain the position of the obstacle.

[0048] In this method, additional techniques are needed for ground extraction. Moreover, using the ground part of the relative depth image to estimate the scale and translation parameters will introduce new errors, and the relative depth image still needs to be converted into the corresponding point cloud of the absolute depth image, and the position of the obstacle is greatly affected by the error of the absolute depth point cloud.

[0049] In summary, the existing technical solutions have the problems of poor universality and low accuracy in obstacle detection.

[0050] To solve the above problems, the embodiments of the present application provide an object detection method and device, which are used to improve the accuracy and universality of object detection in images. The method and the device are based on the same technical concept. Since the principles of the method and the device for solving problems are similar, the implementation of the device and the method can be mutually referred to, and the repeated parts will not be described again. Moreover, in each embodiment of the present application, if there is no special description and logical conflict, the terms and / or descriptions of each embodiment are consistent and can be mutually referred to. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0051] It should be noted that the object detection scheme in the embodiments of the present application can be applied to the Internet of Vehicles, such as vehicle to everything (V2X), long term evolution-vehicle (LTE-V), vehicle to vehicle (V2V), etc. For example, it can be applied to a vehicle with a driving mobile function, or other devices with a driving mobile function in the vehicle. The other devices include, but are not limited to, a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip, a vehicle-mounted unit, a vehicle-mounted radar, or a vehicle-mounted camera and other sensors. The vehicle can implement the perception method provided by the present application through the vehicle-mounted terminal, vehicle-mounted controller, vehicle-mounted module, vehicle-mounted module, vehicle-mounted component, vehicle-mounted chip, vehicle-mounted unit, vehicle-mounted radar, or vehicle-mounted camera. Of course, the control scheme in the embodiments of the present application can also be used for other intelligent terminals with a mobile control function other than vehicles, or be arranged in other intelligent terminals with a mobile control function other than vehicles, or be arranged in components of the intelligent terminal. The intelligent terminal can be an intelligent transportation device, an intelligent home device, a robot, etc. For example, it includes, but is not limited to, an intelligent terminal or a controller, a chip, a radar, or a camera and other sensors, and other components in the intelligent terminal.

[0052] It should be noted that in the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship between the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the following cases: A exists alone, A and B exist together, B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including single item or any combination of multiple items. For example, at least one of a, b, or c, can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0053] In addition, unless otherwise specified, the ordinal numbers mentioned in the embodiments of the present application are used to distinguish a plurality of objects, and are not used to limit the priority or importance of the plurality of objects. For example, the first image and the second image are only used to distinguish different images, and do not represent the difference in priority or importance of the two images.

[0054] For ease of understanding, the following will be introduced in conjunction with the drawings and embodiments.

[0055] Figure 1A schematic diagram of a communication system to which embodiments of the present application are applicable is shown. In the communication system, a sensor 110 and a computing node 120 can be included. The computing node can also be referred to as a computing platform, which can include at least one processor 121 that can execute instructions 123 stored in a non-transitory computer readable medium such as a memory 122. In addition to the instructions 123, the memory 122 can also store data.

[0056] The sensor 110 can be, for example, an image acquisition device such as a camera. The computing node 120 can be an object detection device with data processing capability, which can perform object detection in combination with images acquired by the image acquisition device. The image acquisition device can be integrated into the object detection device, or the image acquisition device can transmit images to the object detection device in real time. In addition, the image acquisition device can also detect images in multimedia files stored in the memory 122.

[0057] In different application scenarios, the product form or deployment manner of the sensor 110 and the computing node 120 can be different.

[0058] For example, the sensor 110 can be an image acquisition device such as a camera, a monocular camera, a binocular camera, a near-infrared camera, a video camera, a cabin camera, a vehicle event data recorder (i.e., a video terminal), a reversing image camera, etc., which is not specifically limited in the embodiments of the present application. For example, in the field of intelligent driving such as autonomous driving and assisted driving, the image acquisition device can capture an external environment of a vehicle. For example, when the vehicle is moving forward, the capturing area can be the area in front of the vehicle head; when the vehicle is reversing, the capturing area can be the area behind the vehicle tail; when the image acquisition device is a 360-degree multi-angle camera, the capturing area can be a 360-degree area around the vehicle, etc.

[0059] For example, the computing node 120 can be a standalone device such as a roadside unit or a cloud server.

[0060] For example, the computing node 120 can be a chip or a component in a vehicle, which can also be a software module, and can be deployed on a related vehicle-mounted device of the vehicle, or can be deployed on other types of terminal devices.

[0061] As an example but not limitation, the computing node 120 can be a vehicle with data processing function, or a vehicle-mounted device with data processing function in the vehicle, or be arranged in a sensor with image acquisition and processing function. The vehicle-mounted device can include but is not limited to a vehicle terminal, a vehicle controller, a vehicle module, a vehicle module, a vehicle component, a vehicle chip, a vehicle unit, an electronic control unit (ECU), a domain controller (DC), and the like.

[0062] In some embodiments, the computing node can be implemented as a computing platform of a vehicle, which can also be a plurality of computing devices that control individual components or subsystems of the vehicle in a distributed manner. The processor 121 can be any conventional processor, such as a central processing unit (CPU). Alternatively, the processor 121 can also include a graphic process unit (GPU), a field programmable gate array (FPGA), a system on chip (SoC), an application specific integrated circuit (ASIC), or a combination thereof. The sensor 110 can be arranged on the same vehicle as the computing node 120, or on a different vehicle.

[0063] Optionally, the data stored in the memory 122 can include, for example, a road map, route information, a position, a direction, a speed of the vehicle, and other such vehicle data, and other information. Such information can be used by the vehicle and the computing platform during operation of the vehicle in an autonomous mode, a semi-autonomous mode, and / or a manual mode to assist the vehicle in achieving autonomous driving or intelligent driving.

[0064] In addition, the computing node 120 can also be other electronic devices with data processing function, including but not limited to smart home devices (such as televisions, etc.), smart robots, mobile terminals (such as mobile phones, tablets, etc.), wearable devices (such as smart watches, etc.), and other smart devices. The object detection device can also be a controller, a chip, or other devices in the smart device.

[0065] The object detection method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It can be understood that the following described embodiments are only some of the embodiments of the present application, not all the embodiments.

[0066] Figure 2A schematic diagram of an object detection method provided by an embodiment of the present application is shown in FIG. 1. The object detection method provided by an embodiment of the present application is described below with reference to Figure 2 The object detection method provided by an embodiment of the present application is described in detail. The object detection method can be implemented by the computing node 120 in the system 100, and can include the following steps. Figure 1

[0067] S210: The computing node performs depth calculation on the first image to obtain a first relative depth image.

[0068] In an embodiment of the present application, when S210 is implemented, the computing node can perform depth calculation on the first image by using a depth estimation model to obtain the first relative depth image. The depth estimation model is a machine vision model, for example, a deep learning model, which can be used to estimate the distance information from the surface of an object in a scene to a camera from a single frame of image. The distance information is represented in the form of a depth map, in which the value of each point represents the depth value of the corresponding pixel. Exemplarily, the depth estimation model can be a neural network model, for example, a convolutional neural network (CNN) or a feedforward neural network, and the specific implementation of the model is not limited in the present application.

[0069] In a specific implementation, exemplarily, the depth estimation model described above can be a monocular depth estimation model, including a monocular relative depth estimation model or a monocular absolute depth estimation model. The monocular relative depth estimation model can be used to obtain relative depth estimation information corresponding to an image, and the monocular absolute depth estimation model can be used to obtain absolute depth estimation information corresponding to an image. In one example, when S210 is implemented, the computing node can perform depth calculation on the first image by using a monocular relative depth estimation model to obtain the first relative depth image. The monocular depth models described below are all monocular relative depth estimation models, and will not be distinguished and described in detail.

[0070] The monocular depth estimation model can be pre-installed on the computing node, and the monocular depth estimation model can be obtained by training according to the business data set and the public data set of the cameras of various vehicle models or various models. The source of the training data is various. For example, the training data can be obtained from different vehicle manufacturers or different individual users after authorization. The monocular depth estimation model obtained by training is not limited by the camera internal and external parameters, and the relative depth image or the absolute depth image obtained based on the monocular depth estimation model is not sensitive to the camera internal and external parameters. The training process of the monocular depth estimation model is not limited in the present application.

[0071] ​The first image can be an image, such as an RGB image, captured by a camera pair of a scene containing at least one target object. As an example, the camera can be deployed on a vehicle, and the first image can include a captured region of the camera. For example, if the camera is deployed at the front of the vehicle, the first image can be a captured image of a region in front of the vehicle. Or for example, if the camera is deployed at the rear of the vehicle, the first image can be a captured image of a region behind the vehicle. If the camera is a 360-degree multi-angle camera on the vehicle, the first image can be a captured image of a region in a 360-degree range around the vehicle.

[0072] If the camera and the computing node are deployed on the same vehicle, before S210 is implemented, the camera can transmit the first image to the computing node in real time, and accordingly, the computing node can receive the first image from the camera. If the camera and the computing node are deployed on different vehicles, or the computing node is deployed on a roadside unit or a cloud server, before S210 is implemented, the camera can transmit the first image to the computing node through wireless communication technology. Accordingly, the computing node can receive the first image from the camera. In another example, the first image can also be stored in a storage medium accessible to the computing node, and before S210 is implemented, the computing node can read the first image from the storage medium. Embodiments of the present application do not make specific limitations on the manner of obtaining the first image.

[0073] S220: The computing node converts the first relative depth image into a second relative depth image according to the scale correction parameter associated with the camera.

[0074] In embodiments of the present application, the scale correction parameter associated with the camera can be used to represent the scale and translation parameters of the relative depth information of the image provided by the camera relative to the absolute depth information.

[0075] Both the relative depth image and the absolute depth image can be converted into corresponding point cloud data. The point cloud data corresponding to the relative depth image obtained by depth estimation based on the original captured image is scale-free, and cannot be directly applied to the absolute depth point cloud processing algorithm with fixed scale. The point cloud data corresponding to the absolute depth image obtained by depth estimation based on the original captured image is scale-dependent, but sensitive to the internal and external parameters of the camera. By using the scale correction parameter associated with the camera, the relative depth image obtained by the monocular depth estimation model can be corrected in scale and translation, and further, the scale-free relative depth point cloud can be converted into a relative depth point cloud with a scale similar to that of the absolute depth point cloud, which is called a measurable relative depth point cloud. Although there is an error between the measurable relative depth point cloud and the absolute depth point cloud, it is still applicable to various algorithms for processing the absolute depth point cloud. When obstacle detection is performed based on the measurable relative depth point cloud, the error of the scale correction parameter is not sensitive, which helps to ensure the accuracy of the obstacle detection result.

[0076] Before implementing S220, the computing node can obtain supervision data corresponding to the second image. As an example but not limitation, the supervision data corresponding to the second image can include absolute depth information corresponding to the second image. When implementing S220, the computing node can take the absolute depth information corresponding to the second image captured by the camera as supervision data (or true value data) to obtain the scale correction parameter associated with the camera. In a specific implementation, the second image can include an image, such as an RGB image, taken by the camera for a scene containing at least one target object. The number of second images can be a small data set, such as tens to hundreds of frames of images. The supervision data can be a small business data set with true value of absolute depth information, such as tens to hundreds of frames of absolute depth information corresponding to the second image. The supervision data can label the position information of the obstacle in the at least one target object. The computing node can perform depth calculation on the second image to obtain a third relative depth image, and determine the scale correction parameter associated with the camera according to the third relative depth image and the supervision data corresponding to the second image. The scale correction parameter can also be referred to as a scale balancing parameter between the monocular depth estimation model and the camera.

[0077] As an example but not limitation, the computing node can obtain the scale balancing parameter between the monocular depth estimation model and the camera by using the least square method or the like. As shown in Figure 3 M0 represents the supervision data corresponding to the second image, and the third relative depth image can be obtained after the second image is input into the monocular depth estimation model, denoted as Mr. The mapping relationship between M0 and Mr can be obtained by the least square method, which satisfies the following expression (1):

[0078] M0 = a * Mr + b; (1)

[0079] Wherein, the scale correction parameter associated with the camera includes: a, b. a and b represent the scale parameter and the translation parameter between the third relative depth image and the supervision data corresponding to the second image, respectively.

[0080] The computing node can initialize the monocular depth estimation model by using the scale correction parameter associated with the camera. Subsequently, after the monocular depth estimation model is normally running, the scale correction parameter can be used for the model output to obtain the measurable point cloud data of the relative depth image. As shown in Figure 4 After performing depth calculation on the first image by using the monocular depth estimation model, the first relative depth image can be obtained, and after processing the first relative depth image by using the scale correction parameter associated with the camera, the second relative depth image can be obtained. The point cloud data corresponding to the second relative depth image is the measurable relative depth point cloud data.

[0081] It should be understood that the monocular depth estimation model of the embodiments of the present application is a general model suitable for the computing device or equipment associated with various types or models of cameras. The training data is more abundant, and better point cloud effects can be obtained. When the monocular depth estimation model is applied to other computing devices or equipment, the monocular depth estimation model can be initialized in the same way, and details are not repeated here.

[0082] S230: The computing node determines the position information of the obstacle in the at least one target object according to the second relative depth image.

[0083] In the embodiments of the present application, when S230 is implemented, the second relative depth image can be converted into corresponding point cloud data, i.e., the relative depth point cloud data can be measured. Since the measurable relative depth point cloud data is close to the absolute depth point cloud data in the order of magnitude scale, the same processing method as the absolute depth point cloud data can be used for obstacle detection.

[0084] For example, the computing node can extract the obstacle contact point and the vertical contact point of the floating obstacle in the measurable relative depth point cloud data, and then find the corresponding pixels of the obstacle contact point and the vertical contact point of the floating obstacle in the first image. As shown in Figure 5 The corresponding pixels of the obstacle contact point in the first image are represented by a dashed line. Further, the computing node can calculate the position of the obstacle contact point in the ego vehicle coordinate system by using the contact point assumption method, i.e., obtain the position information of the obstacle.

[0085] When S230 is implemented, the computing node extracts the obstacle contact point and the vertical contact point of the floating obstacle in the point cloud data corresponding to the second relative depth image and realizes the positioning detection of the obstacle, which can specifically include the following examples:

[0086] Example 1: The computing node extracts the obstacle in the point cloud data corresponding to the second relative depth image to determine the contact point of the obstacle in the at least one target object; in the first image, determines the pixel point corresponding to the contact point of the obstacle; according to the position point of the pixel point after mapping to the target imaging plane, determines the position information of the obstacle.

[0087] Let B' represent the obstacle contact point or the vertical contact point of the floating obstacle in the point cloud data corresponding to the second relative depth image, and the pixel point corresponding to the contact point can be found in the first image, and converted into a point A(X A ,Y A ,Z A ) on the normalized imaging plane. As shown in Figure 6 , taking the camera optical axis as the origin, represented as the camera optical center O(X O ,Y O ,ZO ), the actual grounding point corresponding to A in the ego-vehicle coordinate system is represented as B(X B ,Y B ,0). The following expression (2) is obtained by using the triangle similarity relationship, and the coordinates (X B ,Y B ) of the grounding point B in the ego-vehicle coordinate system are further obtained.

[0088]

[0089] Based on the coordinates (X B ,Y B ) of the grounding point B in the ego-vehicle coordinate system, the computing node can know whether there is an obstacle in the front or rear of the vehicle.

[0090] Example 2: The computing node performs obstacle extraction in the point cloud data corresponding to the second relative depth image to determine the equivalent vertex of the obstacle in the at least one target object in the point cloud data; in the first image, a pixel point corresponding to the equivalent vertex is determined; and according to a position point of the pixel point after being mapped to a target imaging plane, position information of the obstacle is determined.

[0091] C’ represents the equivalent vertex of the obstacle in the point cloud data corresponding to the second relative depth image, and Zc represents the height of the obstacle. The physical meaning of the equivalent vertex is consistent with the position of the grounding point. In the first image, a pixel point corresponding to the equivalent vertex can be found, and the pixel point is converted into a point D(X D ,Y D ,Z D ) on the normalized imaging plane. As shown in Figure 7 , taking the camera optical axis as the origin, represented as the camera optical center O(X O ,Y O ,Z O ), the equivalent vertex of the obstacle corresponding to D in the ego-vehicle coordinate system is C(X C ,Y C ,Z C ). The following expression (3) is obtained by using the triangle similarity relationship, and the height Z C of the obstacle in the ego-vehicle coordinate system is further obtained.

[0092]

[0093] Based on the height Z C of the obstacle in the ego-vehicle coordinate system, the computing node can know whether the obstacle in the front or rear of the vehicle will affect driving.

[0094] Example 3: The computing node can determine the ground point and the equivalent vertex of the obstacle in the point cloud data corresponding to the second relative depth image at the same time, so as to determine whether the obstacle at the head or tail will affect driving based on the coordinates (X B ,Y B ) of the ground point B of the obstacle in the ego vehicle coordinate system and the height Z C of the obstacle in the ego vehicle coordinate system.

[0095] Example 4: The computing node extracts the obstacle in the point cloud data corresponding to the second relative depth image to determine the height change of at least one target object; and determines the position information of the obstacle according to the height change of the at least one target object.

[0096] In this example 4, the height change of the at least one target object can refer to the introduction of analyzing the height of the obstacle in the ego vehicle coordinate system in example 2, and will not be repeated here.

[0097] In an optional implementation, in examples 1-4 above, before extracting the obstacle in the point cloud data corresponding to the second relative depth image, the computing node can also rasterize the point cloud data and analyze the height change trend of the point cloud of each grid and the change trend of the point cloud of adjacent grids to extract the obstacle. Specifically, the computing node can rasterize the point cloud data corresponding to the second relative depth image according to a first resolution (for example, 10). The resolution used for rasterization in the embodiments of the present application is not specifically limited.

[0098] Example 5: The computing node converts the second relative depth image into an absolute depth image; and extracts the obstacle in the point cloud data corresponding to the absolute depth image to determine the position information of the obstacle. For example, the ground point, the equivalent vertex, the height change and other information of the obstacle are extracted in the point cloud data corresponding to the absolute depth image, and then the corresponding pixel points in the first image are found, so as to obtain the position information of the obstacle.

[0099] In this example 5, by performing obstacle detection on the absolute depth image converted from the relative depth image, the association with the camera internal and external parameters can be eliminated, and the accuracy of obstacle detection can be improved.

[0100] Therefore, by using the above method, the monocular depth estimation module and the scale correction parameter associated with the camera can obtain measurable relative depth point cloud data, and the degree of influence of the detection result on the camera internal and external parameters can be reduced, and the accuracy and universality of the detection of the objects in the image can be improved.

[0101] For ease of understanding, the following example 1 of the obstacle detection method is taken as an example to introduce a complete process of the object detection method of the embodiments of the application.

[0102] As Figure 8 shown, the method can include the following steps:

[0103] S801: The computing node obtains a first image and inputs the first image as input information into a monocular depth estimation model, performs depth calculation on the first image through the monocular depth estimation model, and obtains a first relative depth image, denoted as M1.

[0104] S802: The computing node converts the first relative depth image into a second relative depth image, denoted as M2, through the monocular depth estimation model according to the scale correction parameters associated with the relative depth image. Wherein, M1 and M2 satisfy the following expression (4):

[0105] M2 = a * M1 + b; (4)

[0106] Wherein, a and b are scale correction parameters associated with the relative depth image.

[0107] S803: The computing node converts the second relative depth image into corresponding point cloud data through the monocular depth estimation model.

[0108] S804: The computing node rasterizes the point cloud data corresponding to the second relative depth image according to a first resolution (for example, 10), and analyzes the height variation trend of each grid point cloud and the variation trend of adjacent grid point clouds, extracts the obstacle contact point or the suspended obstacle contact point, and denotes it as B'.

[0109] S805: The computing node can perform contact point ranging and positioning based on the first image to obtain obstacle position information.

[0110] For example, the computing node can find the pixel point corresponding to the contact point in the first image, and convert it into a point A (X A ,Y A ,Z A ) on the normalized imaging plane. Taking the camera optical axis as the origin, denoted as the camera optical center O (X O ,Y O ,Z O ), the actual contact point corresponding to A in the ego vehicle coordinate system is denoted as B (X B ,Y B ,0). Using the triangle similarity relationship and the above expression (2), the coordinates (X B ,Y B ) of the contact point B in the ego vehicle coordinate system are further obtained.

[0111] Optionally, between S804 and S805, the computation node can also determine the height of the obstacle in the point cloud data, obtaining the equivalent vertex of the obstacle. The pixel corresponding to this equivalent vertex can be found in the first image and converted into a point D(X) on the normalized imaging plane. D ,Y D Z D With the camera's optical axis as the origin, it is denoted as the camera's optical center O(X). O ,Y O Z O In the vehicle coordinate system, the equivalent vertex of the obstacle corresponding to D is C(X). C ,Y C Z C Using the similarity relationship of triangles and the above expression (3), the height Z of the obstacle in the vehicle coordinate system can be obtained. C The calculation node can be combined with the coordinates (X, Y, F) of the grounding point B in the vehicle coordinate system. B ,Y B The height Z of the obstacle in the vehicle's coordinate system. C Analyze whether obstacles at the front or rear of the vehicle will affect its movement.

[0112] Therefore, by using the above method, measurable relative depth point cloud data can be obtained by utilizing the monocular depth estimation module and the scale correction parameters associated with the camera. Obstacle detection based on relative depth point cloud data can reduce the degree to which the detection results are affected by the camera's intrinsic and extrinsic parameters, and can improve the accuracy and versatility of detecting objects in images.

[0113] This application also provides a communication device for executing the method executed by the computing node in the above method embodiments. The relevant features can be found in the above method embodiments, and will not be repeated here.

[0114] like Figure 9 As shown, the communication device 900 may include: a depth estimation unit 901, used to perform depth calculation on a first image to obtain a first relative depth image, wherein the first image is an image captured by a camera of a scene containing at least one target object; a correction unit 902, used to convert the first relative depth image into a second relative depth image according to a scale correction parameter associated with the camera; and a detection unit 903, used to determine the position information of obstacles in at least one target object based on the second relative depth image. For specific implementation details, please refer to the method steps implemented by the computing node in the above method embodiment; they will not be repeated here.

[0115] It should be understood that the division of units in the above apparatus is only a logical function division, and all or part of them can be integrated into a physical entity or physically separated when actually implemented. In addition, the units in the apparatus can be implemented in the form of processor calling software; for example, the apparatus includes a processor, the processor is connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the units of the apparatus, wherein the processor is, for example, a general processor such as a central processing unit (CPU) or a microprocessor, and the memory is a memory in the apparatus or a memory outside the apparatus. Alternatively, the units in the apparatus can be implemented in the form of hardware circuit, and the functions of part or all of the units can be realized by the design of the hardware circuit, which can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of part or all of the units are realized by the design of the logical relationship of elements in the circuit; for example, in another implementation, the hardware circuit is realized by a programmable logic device (PLD), and taking a field programmable gate array (FPGA) as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of part or all of the units. All units of the above apparatus can be implemented in the form of processor calling software, or all units can be implemented in the form of hardware circuit, or part of the units can be implemented in the form of processor calling software, and the remaining part can be implemented in the form of hardware circuit.

[0116] In the embodiments of the present application, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a CPU, a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), a digital signal processor (DSP), etc. In another implementation, the processor can implement certain functions through a logical relationship of a hardware circuit, which is fixed or reconfigurable. For example, the processor is an ASIC or a PLD implemented hardware circuit, such as an FPGA. In the reconfigurable hardware circuit, the processor loads a configuration document to implement the hardware circuit configuration. It can be understood that the processor loads an instruction to implement the functions of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0117] It can be seen that each unit in the above apparatus can be one or more processors (or processing circuits) configured to implement the above method, such as a CPU, a GPU, an NPU, a TPU, a DPU, a microprocessor, a DSP, an ASIC, an FPGA, or a combination of at least two of these processor forms.

[0118] In addition, each unit in the above apparatus can be integrated together or can be independently implemented. In one implementation, the units are integrated together to implement a system-on-a-chip (SOC). The SOC can include at least one processor for implementing any of the above methods or implementing the functions of the units of the apparatus. The at least one processor can be of different types, such as a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0119] In a simple embodiment, those skilled in the art can conceive that the communication apparatus in the above embodiments can adopt the form shown in the above. Figure 10

[0120] As shown in the apparatus 1000 shown in Figure 10 , the apparatus 1000 includes at least one processor 1010 and a communication interface 1030. In an optional design, it can also include a memory 1020.

[0121] ​The specific connection medium between the processor 1010 and the memory 1020 is not limited in the embodiments of this application.

[0122] In the apparatus as shown in Figure 10 , the processor 1010 can perform data transmission through the communication interface 1030 when communicating with other devices.

[0123] When the communication apparatus adopts the form as shown in Figure 10 , the processor 1010 in the apparatus can invoke the computer-executed instructions stored in the memory 1020, so that the apparatus 1000 can execute any of the above method embodiments. Figure 10

[0124] The embodiments of this application also relate to a chip system, which comprises a processor, and is configured to invoke a computer program or computer instructions stored in a memory, so that the processor executes the method of any of the above embodiments.

[0125] In a possible implementation, the processor can be coupled with the memory through an interface.

[0126] In a possible implementation, the chip system can also directly comprise the memory, and the memory stores the computer program or computer instructions.

[0127] ​By way of example, the memory can be volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, nonvolatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which acts as external cache. By way of example, and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double-data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM). By way of example, and not limitation, computer-readable media can include transmission-type media, such as those involving transmission of Synchronous Optical Network (SONET) signals over fiber optic cables. Herein, the specification and claims can use "computer-readable storage media" interchangeably with other similar terminology, including "computer-readable media." Examples of computer-readable storage media include, but are not limited to, a floppy disk, a magnetic hard disk drive, or any other magnetic device, a magneto-optical disk, a solid state drive, a flash memory device, or any other storage device.

[0128] Embodiments of the present application also relate to a processor configured to invoke a computer program or computer instructions stored in a memory to cause the processor to perform the method of any of the embodiments described above.

[0129] For example, in the embodiments of the present application, the processor is an integrated circuit chip with a signal processing capability. For example, the processor can be an FPGA, a general-purpose processor, a DSP, an ASIC, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, a system on chip (SoC), a CPU, a network processor (NP), a microcontroller unit (MCU), a PLD, or other integrated circuits, which can perform or execute the methods, steps, and logical block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the methods disclosed in the embodiments of the present application can be directly embodied in hardware code executed by the processor, or a combination of hardware and software modules in the processor. The software module can reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an electrically programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), registers, or a non-removable memory, or a removable memory, or a combination of the memory. The memory is located in the storage device, and the processor reads information in the memory to perform the steps of the above-described methods in combination with hardware.

[0130] It should be understood that the embodiments of the present application can be provided as a method, a system, or a computer program product.

[0131] In a possible implementation, the embodiment of the present application provides a computer readable storage medium, which stores program codes, and when the program codes are executed on the computer, the computer executes the method embodiments.

[0132] In a possible implementation, the embodiment of the present application provides a computer program product, which, when executed on the computer, enables the computer to execute the method embodiments.

[0133] Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, a disk memory, a CD-ROM, an optical memory, etc.) having computer-usable program codes embodied in the medium.

[0134] These computer program instructions can also be stored in a computer readable memory that can direct the computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer readable memory produce the products including the instruction device, which implement the functions specified in the flowcharts Figure 1 one flowchart or multiple flowcharts and / or blocksFigure 1 the function specified in the one or more blocks.

[0135] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide a process for implementing the functions described in the flowcharts Figure 1 the flowcharts or multiple flowcharts and / or blocks Figure 1 the function specified in the one or more blocks.

[0136] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present application without departing from the scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations. In various embodiments of the present application, if there is no special description and logical conflict, the terms and / or descriptions between various embodiments are consistent and can be mutually referred, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationship.

Claims

1. An object detection method characterized by, The method comprises: performing depth calculation on a first image to obtain a first relative depth image, the first image being an image captured by a camera for a scene containing at least one target object; converting the first relative depth image into a second relative depth image according to a scale correction parameter associated with the camera; determining position information of an obstacle in the at least one target object according to the second relative depth image.

2. The method of claim 1, wherein, The method further comprises: performing depth calculation on a second image to obtain a third relative depth image, the second image being an image captured by a camera for a scene containing at least one target object; determining a scale correction parameter associated with the camera according to supervised data corresponding to the third relative depth image and the second image.

3. The method according to claim 1 or 2, characterized in that, The determination of the position information of the obstacle in the at least one target object according to the second relative depth image comprises: obstacle extraction in point cloud data corresponding to the second relative depth image to determine a ground point of the obstacle in the at least one target object; determination of a pixel point corresponding to the ground point of the obstacle in the first image; determination of the position information of the obstacle according to a position point of the pixel point after mapping to a target imaging plane.

4. The method according to claim 1 or 2, characterized in that, The determination of the position information of the obstacle in the at least one target object according to the second relative depth image comprises: obstacle extraction in point cloud data corresponding to the second relative depth image to determine an equivalent vertex of a height of the obstacle in the at least one target object in the point cloud data; determination of a pixel point corresponding to the equivalent vertex in the first image; determination of the position information of the obstacle according to a position point of the pixel point after mapping to a target imaging plane.

5. The method according to claim 1 or 2, characterized in that, The determination of the position information of the obstacle in the at least one target object according to the second relative depth image comprises: obstacle extraction in point cloud data corresponding to the second relative depth image to determine a height variation of the at least one target object; determination of the position information of the obstacle according to the height variation of the at least one target object.

6. The method according to any one of claims 3-5, characterized in that, The obstacle extraction in the point cloud data corresponding to the second relative depth image comprises: rasterization of the point cloud data corresponding to the second relative depth image, and analysis of a point cloud height variation trend of each grid and a neighboring grid point cloud variation trend for obstacle extraction.

7. The method of claim 6, wherein, The rasterization of the point cloud data corresponding to the second relative depth image comprises: rasterization of the point cloud data corresponding to the second relative depth image according to a first resolution.

8. The method of claim 1 or 2, wherein, The determination of the position information of the obstacle in the at least one target object according to the second relative depth image comprises: conversion of the second relative depth image into an absolute depth image; obstacle extraction based on point cloud data corresponding to the absolute depth image to determine the position information of the obstacle.

9. The method according to any one of claims 1-8, characterized in that, The depth calculation on the first image comprises: depth calculation on the first image by a monocular depth estimation model.

10. An object detection device, characterized by, It comprises: a depth estimation unit configured to perform depth calculation on the first image to obtain a first relative depth image, the first image being an image of a scene containing at least one target object taken by a camera; a correction unit configured to convert the first relative depth image into a second relative depth image according to a scale correction parameter associated with the camera; a detection unit configured to determine position information of an obstacle in the at least one target object according to the second relative depth image.

11. A communications device, characterized by A computer program product comprising a computer readable medium storing program code which, when executed on a computer, causes the computer to carry out the method of any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer readable medium stores program code which, when executed on a computer, causes the computer to carry out the method of any one of claims 1-9.

13. A computer program product, characterised in that, The computer program product, when executed on a computer, causes the computer to carry out the method of any one of claims 1-9.

14. A vehicle characterized by comprising: comprising means for carrying out the method of any one of claims 1-9.

Images