Data processing method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of electric vehicles, and discloses a data processing method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed target two-dimensional image; according to the target two-dimensional image, determining a two-dimensional detection frame of an observation object in the target two-dimensional image and a first relative depth of the observation object; determining a plurality of second relative depths of the observation object under different dimensions according to the two-dimensional detection frame; determining the relative depth of the observation object according to the first relative depth and the plurality of second relative depths; and generating three-dimensional motion information corresponding to the observation object according to the relative depth and the two-dimensional detection frame. According to the invention, based on pure vision object detection, the relative depth of the object can be accurately determined by integrating a plurality of depths, the implementation mode is simple, the required calculation amount is small, and excessive calculation power does not need to be consumed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electric vehicles, in particular to a data processing method and device, computer equipment and a storage medium. BACKGROUND

[0002] During driving, a vehicle needs to perceive the surrounding environment to detect pedestrians, other vehicles and other targets on the driving path, so as to ensure driving safety.

[0003] Current object detection methods mostly rely on multi-sensor data such as laser radar, camera and millimeter wave radar, and detect objects by fusing multi-modal sensor data. This data fusion method requires a large amount of calculation, especially when detecting distant objects (for example, the distance between the ego vehicle and other vehicles is more than 100 m), the consumed computing power almost increases exponentially, and the computing power consumption is large.

[0004] Pure visual detection by camera 2D perception data can reduce the demand for computing power. However, due to the limitation of camera detection accuracy and the variety of surrounding environment, the object detection result is not very stable, especially when detecting distant objects, it is difficult to accurately identify other objects around the vehicle. SUMMARY

[0005] Therefore, the present application provides a data processing method, device, computer equipment and storage medium to improve the accuracy of object detection under the condition of using low computing power.

[0006] In a first aspect, the present application provides a data processing method, comprising:

[0007] obtaining a target two-dimensional image to be processed;

[0008] determining a two-dimensional detection box of an observed object in the target two-dimensional image and a first relative depth of the observed object according to the target two-dimensional image;

[0009] determining a plurality of second relative depths of the observed object in different dimensions according to the two-dimensional detection box;

[0010] determining a relative depth of the observed object according to the first relative depth and the plurality of second relative depths;

[0011] generating three-dimensional motion information corresponding to the observed object according to the relative depth and the two-dimensional detection box; the three-dimensional motion information represents the motion state of the observed object in three-dimensional space.

[0012] In some optional embodiments, the determining a plurality of second relative depths of the observed object in different dimensions according to the two-dimensional detection box comprises:

[0013] determining a spatial parameter of the two-dimensional detection frame in different dimensions;

[0014] determining a plurality of second relative depths of the observed object in different dimensions according to the spatial parameter of the two-dimensional detection frame.

[0015] In some optional embodiments, the plurality of second relative depths of the observed object in different dimensions include at least two of an area relative depth of the observed object in an area dimension, a height relative depth of the observed object in a height dimension, and a projection relative depth of the observed object in a projection dimension; and the spatial parameter includes at least two of an area of the two-dimensional detection frame, a height of the two-dimensional detection frame, and a bottom position coordinate of the two-dimensional detection frame.

[0016] The area relative depth of the observed object is determined according to the area of the two-dimensional detection frame; the height relative depth of the observed object is determined according to the height of the two-dimensional detection frame; and the projection relative depth of the observed object is determined by inversely perspective projecting the two-dimensional detection frame using the bottom position coordinate of the two-dimensional detection frame.

[0017] In some optional embodiments, the determining, according to the target two-dimensional image, of the two-dimensional detection frame of the observed object in the target two-dimensional image and the first relative depth of the observed object includes:

[0018] performing object detection on the target two-dimensional image according to a preset object detection model to identify the two-dimensional detection frame of the observed object in the target two-dimensional image;

[0019] inputting the target two-dimensional image into a preset perception network to obtain the first relative depth of the observed object in the target two-dimensional image output by the perception network.

[0020] In some optional embodiments, after the generating of the three-dimensional motion information corresponding to the observed object, the method further includes:

[0021] obtaining an observed object queue corresponding to the observed object; wherein the observed object queue includes three-dimensional motion information of a historical observed object, the historical observed object in the observed object queue and the observed object are associated observed objects, and the three-dimensional motion information of the historical observed object is determined according to a historical two-dimensional image, a collection time of the historical two-dimensional image being earlier than a collection time of the target two-dimensional image;

[0022] fitting the three-dimensional motion information of the observation object, the three-dimensional motion information of the historical observation object in the observation object queue, the acquisition time of the historical two-dimensional image and the acquisition time of the target two-dimensional image, to obtain a fitting result between the three-dimensional motion information and the acquisition time;

[0023] According to the fitting result, the three-dimensional motion information of the observation object is corrected to generate the corrected motion information corresponding to the observation object, and the corrected motion information corresponding to the observation object is taken as the new three-dimensional motion information corresponding to the observation object.

[0024] In some optional embodiments, the observation object is provided with a corresponding identifier; and the observation object queue corresponding to the observation object is obtained by:

[0025] determining whether the identifier of the observation object is the same as the identifier of the historical observation object in the pre-recorded object queue;

[0026] In the case that the identifier of the observation object is the same as the identifier of the historical observation object in the object queue, the object queue to which the historical observation object with the same identifier as the observation object belongs is taken as the observation object queue; the historical observation object in the observation object queue and the observation object correspond to the same object.

[0027] In some optional embodiments, the target two-dimensional image is used to represent the vehicle condition around the target mobile device; and the observation object queue corresponding to the observation object is obtained by:

[0028] determining whether the observation object is a preset type object; wherein the preset type object is an object whose influence on the driving safety of the target mobile device meets a preset condition;

[0029] In the case that the observation object is a preset type object, the observation object and the historical observation object in the object queue are associated by the intersection-over-union ratio for the pre-recorded object queue, to obtain an intersection-over-union ratio association result between the historical observation object in the object queue and the observation object;

[0030] According to the intersection-over-union ratio association result between the historical observation object in all object queues and the observation object, the observation object queue is determined from the object queue.

[0031] In some optional embodiments, the observation object queue corresponding to the observation object is obtained by:

[0032] If the observed object is not an object of the preset type, or the intersection-over-union correlation result indicates that none of the object queues are the observed object queues, then for each object queue, performing nearest neighbor correlation between the observed object and the historical observed objects in the object queue to obtain a nearest neighbor correlation result between the historical observed objects in the object queue and the observed object;

[0033] An observed object queue is determined from the object queue according to the nearest neighbor association results between the historical observed objects and the observed objects in all object queues.

[0034] In some optional embodiments, the method further comprises:

[0035] If no observed object queue exists, creating a new first object queue for the observed object;

[0036] and / or,

[0037] If there is a second object queue, the number of unassociated times of the second object queue is increased by one; the historical observation objects in the second object queue are not associated with any of the observation objects, and the number of unassociated times of the second object queue is the number of times the second object queue has not been associated with an object in a row; when the number of unassociated times of the second object queue exceeds the first preset number, the second object queue is deleted.

[0038] In some optional implementations, the fitting according to the three-dimensional motion information of the observed object, the three-dimensional motion information of historical observed objects in the observed object queue, the acquisition time of the historical two-dimensional images, and the acquisition time of the target two-dimensional image includes:

[0039] When the number of objects in the observation object queue exceeds a first preset threshold, fitting is performed based on the observation object, the three-dimensional motion information of the latest k historical observation objects in the observation object queue, and the corresponding acquisition time; k≥3, the latest k historical observation objects are the last k historical observation objects determined by arranging the historical two-dimensional images corresponding to the historical observation objects in the observation object queue from first to last according to the acquisition time.

[0040] In some optional embodiments, the method further comprises:

[0041] Inputting the target two-dimensional image into a preset perception network to obtain the speed of the observed object in the target two-dimensional image output by the perception network;

[0042] If the speed of the observation object is less than a preset value, and the number of historical observation objects associated with the observation object and having a speed less than a preset value exceeds a second preset number within a preset time period adjacent to the time when the target two-dimensional image is collected, the speed in the corrected motion information is set to zero.

[0043] In some optional embodiments, the target two-dimensional image is used to represent a vehicle situation around the target moving device; and the method further comprises:

[0044] According to the three-dimensional motion information corresponding to the observation object, path planning information of the target moving device is determined, so that the target moving device travels according to the path planning information.

[0045] In some optional embodiments, the determination of the path planning information of the target moving device according to the three-dimensional motion information corresponding to the observation object comprises:

[0046] It is determined whether the observation object is a key observation object.

[0047] In the case where the observation object is a key observation object, the path planning information of the target moving device is determined according to the three-dimensional motion information corresponding to the observation object.

[0048] In some optional embodiments, the determination of whether the observation object is a key observation object comprises:

[0049] At least one of the number of objects in the observation object queue corresponding to the observation object, a fitting index representing a fitting effect, and a collision time predicted according to the three-dimensional motion information corresponding to the observation object is used to determine whether the observation object is a key observation object; the collision time is the time when the target moving device collides with the observation object.

[0050] In the case where at least one of the number of objects in the observation object queue corresponding to the observation object is greater than a preset number, the fitting effect represented by the fitting index is greater than a preset fitting effect, and the collision time is less than a preset collision time, the observation object is determined to be a key observation object.

[0051] In some optional embodiments, the determination of the relative depth of the observation object according to the first relative depth and the plurality of second relative depths comprises:

[0052] The first relative depth and the plurality of second relative depths are fused in a weighted manner to determine the relative depth of the observation object; the first relative depth and the plurality of second relative depths each have a corresponding weight.

[0053] In some optional embodiments, the fusing the first relative depth and the second relative depths in a weighted manner comprises:

[0054] The first relative depth and the second relative depths are weighted according to a first preset weight corresponding to the first relative depth and a second preset weight corresponding to the second relative depths.

[0055] Alternatively, the first weight corresponding to the first relative depth and the second weight corresponding to the second relative depths are determined by using a weighted least square method, and the first relative depth and the second relative depths are weighted according to the first weight corresponding to the first relative depth and the second weight corresponding to the second relative depths.

[0056] In some optional embodiments, the two-dimensional bounding box is a bounding box for locating and identifying an observed object in the target two-dimensional image, and the bounding box comprises one of the following: a rectangular box, a polygonal box, a circular or elliptical box, a rotated rectangular box, and a self-defined shape box.

[0057] In a second aspect, the present application provides a data processing apparatus, comprising:

[0058] An acquisition module configured to acquire a target two-dimensional image to be processed;

[0059] A depth determination module configured to determine, according to the target two-dimensional image, a two-dimensional bounding box of an observed object in the target two-dimensional image and a first relative depth of the observed object, determine, according to the two-dimensional bounding box, a plurality of second relative depths of the observed object in different dimensions, and determine a relative depth of the observed object according to the first relative depth and the plurality of second relative depths.

[0060] A processing module configured to generate three-dimensional motion information corresponding to the observed object according to the relative depth and the two-dimensional bounding box, wherein the three-dimensional motion information represents a motion state of the observed object in a three-dimensional space.

[0061] In a third aspect, the present application provides a computer device, comprising a memory and a processor, which are communicatively connected with each other, and the memory stores computer instructions, and the processor executes the computer instructions to perform the data processing method of the first aspect or any of the corresponding embodiments thereof.

[0062] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer instructions for causing a computer to perform the data processing method of the first aspect or any of the corresponding embodiments thereof.

[0063] In a fifth aspect, the present application provides a computer program product comprising computer instructions for causing a computer to execute the data processing method of the first aspect or any of its corresponding embodiments.

[0064] The present application determines the first relative depth of the surrounding observation object according to the collected target two-dimensional image, and determines the second relative depth of the observation object in several dimensions by identifying the two-dimensional detection frame of the observation object, and the relative depth of the observation object can be accurately determined by comprehensively considering multiple depths, and finally the three-dimensional motion information of the observation object can be obtained, and the path planning can be performed based on the three-dimensional motion information. The motion parameters of the observation object can be determined only by the two-dimensional image collected by the image collection device, the object detection can be realized by pure vision, and data fusion and other processing are not required, the implementation is simple, the required calculation amount is small, the motion parameters of each object in the surrounding environment can be accurately determined without consuming too much calculation power, and the accuracy of subsequent path planning is improved. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the specific embodiments or the related art, the drawings needed to be used in the specific embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0066] Figure 1 is a flowchart of a data processing method according to an embodiment of the present application;

[0067] Figure 2 is a flowchart of another data processing method according to an embodiment of the present application;

[0068] Figure 3 is a flowchart of still another data processing method according to an embodiment of the present application;

[0069] Figure 4 is a schematic diagram of associating observation objects according to an embodiment of the present application;

[0070] Figure 5 is a detailed flowchart of a data processing method according to an embodiment of the present application;

[0071] Figure 6 is a structural block diagram of a data processing device according to an embodiment of the present application;

[0072] Figure 7 is a hardware structure schematic diagram of a computer device according to an embodiment of the present application;

[0073] Figure 8 is a schematic diagram of a two-dimensional detection frame according to an embodiment of the present application. DETAILED DESCRIPTION

[0074] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0075] According to the embodiments of the present application, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0076] In the present embodiment, a data processing method is provided, which can be applied to a mobile device such as a smart car, a robot, etc. that needs to continuously perform object detection, and can also be applied to a server on the network side or a distributed system located on the network side, which can be determined based on actual conditions. Figure 1 is a flowchart of a data processing method according to an embodiment of the present application, as shown in Figure 1 The flowchart includes the following steps.

[0077] Step S101, obtaining a target two-dimensional image to be processed.

[0078] In the present embodiment, the data processing method is used to perform object detection on the surrounding environment of a mobile device (such as a vehicle, etc.) to enable path planning of the mobile device. For ease of description, the mobile device is referred to as a target mobile device. For example, the target mobile device can be an electric vehicle (such as a pure electric vehicle, a range-extended vehicle), a fuel vehicle, a flying vehicle, etc., can also be a robot such as a humanoid robot, a legged robot, etc., or can also be a flying vehicle such as a drone, etc.

[0079] When performing object detection, an image acquisition device such as a camera can be used to acquire images of the surrounding environment in real time, and a two-dimensional image representing the surrounding environment information can be acquired. The two-dimensional image can be an image of the front, rear, left side, or right side of the target mobile device, or can be a fusion image determined by fusing images acquired by multiple cameras. For example, the two-dimensional image can be an image representing the environment in front of the vehicle. For a frame of image acquired by the image acquisition device that needs to be processed, it is referred to as a target two-dimensional image. For example, the two-dimensional image acquired by the image acquisition device at the current time can be used as the target two-dimensional image.

[0080] It can be understood that the image acquisition device can be arranged on the target mobile device, or can be an image acquisition device capable of moving synchronously with the target mobile device, or can be an image acquisition device in a roadside device, and the acquired image can be transmitted to the target mobile device such as a vehicle for subsequent processing. The image acquisition device can continuously acquire images, so that multiple two-dimensional images can be sequentially acquired. For some or all of the two-dimensional images, they can be used as the target two-dimensional image and subsequent processing can be performed.

[0081] For long-distance object detection, the camera can be a long-focus camera, and the effective distance of the camera can be greater than 100 meters.

[0082] In step S102, a two-dimensional detection box of the observed object in the target two-dimensional image and a first relative depth of the observed object are determined according to the target two-dimensional image.

[0083] The target two-dimensional image includes objects observed by the camera, and the objects in the target two-dimensional image are referred to as observed objects.

[0084] For example, the camera is a vehicle-mounted camera, which can acquire images of the environment around the vehicle in real time, generate a corresponding target two-dimensional image, and identify objects such as vehicles, pedestrians, and obstacles (such as broken-down vehicles and falling rocks) in the target two-dimensional image by performing object detection on the target two-dimensional image. These objects can be used as observed objects.

[0085] For the target two-dimensional image, object detection can be performed to identify each observed object in the target two-dimensional image. When performing object detection, a two-dimensional (2D) detection box of each observed object can be determined, which is used to locate and identify the bounding box of the observed object in the target two-dimensional image. The bounding box includes one of the following: a rectangular box, a polygonal box, a circular or elliptical box, a rotated rectangular box, and a self-defined shape box. For example, the two-dimensional detection box is a bounding box surrounding the observed object in a two-dimensional coordinate system (such as an image coordinate system).

[0086] The two-dimensional detection frame can realize positioning, feature extraction, etc. In the embodiment, the two-dimensional detection frame can represent the size, position, etc. of the observation object in the camera view angle.

[0087] Taking a rectangular frame as an example, the two-dimensional detection frame can be defined by four parameters, which can include the horizontal and vertical coordinates of two opposite vertices of the two-dimensional detection frame, such as the horizontal and vertical coordinates of the upper left corner vertex and the horizontal and vertical coordinates of the lower right corner vertex; or the four parameters can also include the horizontal and vertical coordinates of a vertex and the width and height of the two-dimensional detection frame, such as the x-coordinate (i.e. horizontal coordinate) and y-coordinate (i.e. vertical coordinate) of the upper left corner vertex, and the width and height of the two-dimensional detection frame.

[0088] Figure 8 A schematic diagram of a two-dimensional detection frame is shown. As shown in Figure 8 The upper left corner of the target two-dimensional image 801 is the origin (0, 0), and there is an observation object 802, which can be a pedestrian on the road for example, corresponding to a two-dimensional detection frame 803, and the upper left corner vertex coordinate of the two-dimensional detection frame 803 is (x1, y1), the width is W, and the height is H. The two-dimensional detection frame 803 can be represented by the four parameters x1, y1, W, and H.

[0089] In the embodiment, the observation object in the target two-dimensional image corresponds to certain motion parameters. The motion parameters can specifically include position information and speed information of the observation object, and can also include acceleration, etc. Generally, the main purpose of object detection is to determine the relative position between the self (e.g. the ego vehicle) and the observation object in the surrounding environment, so as to avoid collision and perform path planning, etc. Therefore, the position information in the motion parameters specifically includes the relative depth (i.e. relative distance) between the observation object, and the speed information can be the speed of the observation object itself, or the relative speed between the self (e.g. the ego vehicle) and the observation object, which is not limited here.

[0090] Specifically, each observation object corresponds to a certain relative depth, which is the relative depth (i.e. relative distance) between the target mobile device and the observation object. In the embodiment, the relative depth corresponding to the observation object can be directly determined based on the target two-dimensional image. For ease of description, the relative depth directly determined based on the target two-dimensional image is referred to as the first relative depth.

[0091] In step S103, a plurality of second relative depths of the observation object in different dimensions are determined according to the two-dimensional detection frame.

[0092] In this embodiment, for each observation object, the motion parameters of the observation object can be preliminarily extracted based on the two-dimensional detection box of the observation object, and the motion parameters include one or more dimensions of relative depth, i.e., second relative depth. For example, the relative distance between the observation object and the target mobile device (e.g., ego vehicle) can be determined based on the size of the two-dimensional detection box, i.e., the relative depth, and the larger the two-dimensional detection box, the smaller the relative depth, which can be used as a second relative depth.

[0093] It can be understood that the first relative depth and the second relative depth are both used to represent the relative depth between the target mobile device and the observation object, but the determination methods are different. As described above, the first relative depth is directly obtained based on the target two-dimensional image, e.g., by identifying the features of the observation object in the target two-dimensional image, so that the first relative depth of the observation object can be directly determined; and for the second relative depth, the two-dimensional detection box corresponding to the observation object needs to be determined first, and then the second relative depth between the target mobile device and the observation object is determined based on the two-dimensional detection box.

[0094] In step S104, the relative depth of the observation object is determined according to the first relative depth and the second relative depth.

[0095] In this embodiment, as described above, the first relative depth of the observation object can be determined based on the target two-dimensional image, and the second relative depth of the observation object can be determined according to the two-dimensional detection box of the observation object. By integrating these relative depths (including the first relative depth and the second relative depth) determined in different ways, the relative depth of the observation object can be finally determined, and the finally determined relative depth of the observation object is used for subsequent recognition processing.

[0096] The relative depth of the observation object obtained according to the first relative depth and the second relative depth is the finally determined relative depth, and the relative distance between the target mobile device and the observation object is represented by the relative depth. For example, the first relative depth or a certain second relative depth can be directly used as the relative depth of the observation object, or the first relative depth and the second relative depth can be fused to finally determine the relative depth of the observation object.

[0097] In step S105, the three-dimensional motion information corresponding to the observation object is generated according to the relative depth and the two-dimensional detection box, and the three-dimensional motion information represents the motion state of the observation object in the three-dimensional space.

[0098] In this embodiment, the two-dimensional detection frame of the observation object can include two-dimensional size information of the observation object, for example, information such as width and height of the observation object; in combination with the finally determined relative depth, the motion information of the observation object in the three-dimensional space, that is, the three-dimensional motion information, can be obtained; the three-dimensional motion information can represent the motion state of the observation object in the three-dimensional space. The motion state is used to represent the state of the target mobile device relative to a certain reference system when the target mobile device moves, for example, in the world coordinate system or in the self-coordinate system with the center of the target mobile device as the origin, and can specifically represent physical parameters such as position and velocity of the observation object.

[0099] For example, the three-dimensional motion information can include three-dimensional coordinates of the observation object (for example, coordinates of the observation object in the world coordinate system), width, height, and depth in the depth direction of the observation object, and orientation of the observation object. If the velocity of the observation object can also be determined, the generated three-dimensional motion information also includes the velocity of the observation object. For example, the two-dimensional detection frame of the observation object can be back projected to the 3D space, so as to be converted into a corresponding three-dimensional (3D) detection frame, and the three-dimensional motion information of the observation object is represented by the three-dimensional detection frame.

[0100] It can be understood that the three-dimensional motion information can represent the position, size, and the like of the observation object in the three-dimensional space, and after the three-dimensional motion information is determined, the moving path of the target mobile device can be determined according to the detected three-dimensional motion information of the observation object. For example, the position of the observation object can be determined based on the three-dimensional motion information, and the moving path of the target mobile device can be adaptively adjusted to avoid collision between the target mobile device and the observation object.

[0101] The data processing method provided in this embodiment can determine the first relative depth of the surrounding observation object according to the collected target two-dimensional image, and can further determine the second relative depth of the observation object in multiple dimensions by identifying the two-dimensional detection frame of the observation object. The relative depth of the observation object can be accurately determined by comprehensively considering multiple depths, and the three-dimensional motion information of the observation object can be finally obtained, and the path planning can be performed based on the three-dimensional motion information. The motion parameters of the observation object can be determined only by using the two-dimensional image collected by the image collection device, and the object detection can be realized by pure vision, without the need for data fusion and the like. The implementation is simple, the required calculation amount is small, the motion parameters of each object in the surrounding environment can be accurately determined without consuming too much calculation power, and the accuracy of subsequent path planning is improved.

[0102] In this embodiment, a data processing method is provided, which can be applied to a mobile device that needs to continuously detect objects, such as a smart car, a robot, and the like, and can also be applied to a server on the network side or a distributed system on the network side, and can be specifically determined based on actual conditions. Figure 2is a flowchart of a data processing method according to an embodiment of the present application, as shown in the figure, the flow includes the following steps. Figure 2

[0103] In step S201, a target two-dimensional image to be processed is acquired.

[0104] For details, please refer to step S101 of the embodiment shown in the figure, which will not be repeated here. Figure 1

[0105] In step S202, according to the target two-dimensional image, a two-dimensional detection frame of the observed object in the target two-dimensional image is determined, and a first relative depth of the observed object is determined.

[0106] For details, please refer to step S102 of the embodiment shown in the figure, which will not be repeated here. Figure 1

[0107] Optionally, the above step S202 "determining a two-dimensional detection frame of the observed object in the target two-dimensional image and a first relative depth of the observed object according to the target two-dimensional image" can include: performing object detection on the target two-dimensional image according to a preset object detection model to identify a two-dimensional detection frame of the observed object in the target two-dimensional image; inputting the target two-dimensional image into a preset perception network to obtain a first relative depth of the observed object in the target two-dimensional image output by the perception network.

[0108] In this embodiment, after the target two-dimensional image is acquired, object detection can be performed on it based on an object detection model (such as a convolutional neural network), so as to identify each observed object in the target two-dimensional image, and the detection frame of each observed object, i.e. the two-dimensional detection frame, can be determined, and the motion parameters of the observed object, including a plurality of second relative depths, can be extracted based on the two-dimensional detection frame.

[0109] In addition, a network for determining the motion parameters (such as relative depth, speed, etc.) of each object in the image based on the image, i.e. a perception network, can be preset; for example, the perception network can be constructed based on a convolutional neural network. After the target two-dimensional image is acquired, the target two-dimensional image is input into the perception network, so as to determine the speed of each observed object and the relative depth between the observed object and the target mobile device itself (i.e. the target mobile device), for the sake of distinguishing description, the relative depth determined based on the perception network is referred to as the first relative depth.

[0110] In this embodiment, the object detection model and the perception network are constructed in advance, so that the relative depth and other motion parameters of the observed object can be determined based on different ways, and the accuracy of the finally determined relative depth can be improved by integrating a plurality of relative depths.

[0111] In step S203, a plurality of second relative depths of the observed object in different dimensions are determined according to the two-dimensional detection frame.​​​

[0112] For details, please refer to Figure 1 The step S103 of the embodiment is not described here again.

[0113] Optionally, the step S202 of determining the second relative depths of the observation object in different dimensions according to the two-dimensional detection frame comprises: determining the spatial parameters of the two-dimensional detection frame in different dimensions; and determining the second relative depths of the observation object in different dimensions according to the spatial parameters of the two-dimensional detection frame.

[0114] In the embodiment, for each two-dimensional detection frame of the observation object, since it has certain spatial parameters, the spatial parameters of each two-dimensional detection frame can be determined, and then the second relative depth corresponding to the two-dimensional detection frame can be determined based on the spatial parameters.

[0115] Specifically, the two-dimensional detection frame can include the spatial parameters of the observation object in different dimensions, which are used to represent the size or position of the observation object in the two-dimensional space. By using various spatial parameters, the second relative depths of the observation object in different dimensions can be determined. In the embodiment, the second relative depths of the observation object can be simply and conveniently determined based on the two-dimensional detection frame.

[0116] Optionally, the spatial parameters include at least two of the area of the two-dimensional detection frame, the height of the two-dimensional detection frame, and the bottom position coordinates of the two-dimensional detection frame. Correspondingly, the second relative depths of the observation object in different dimensions include at least two of the area relative depth of the observation object in the area dimension, the height relative depth of the observation object in the height dimension, and the projection relative depth of the observation object in the projection dimension.

[0117] The area relative depth of the observation object is determined according to the area of the two-dimensional detection frame; the height relative depth of the observation object is determined according to the height of the two-dimensional detection frame; and the projection relative depth of the observation object is determined by inversely perspective projecting the two-dimensional detection frame using the bottom position coordinates of the two-dimensional detection frame.

[0118] In the embodiment, the spatial parameters of the two-dimensional detection frame are parameters for representing the spatial features of the two-dimensional detection frame, which can be parameters related to the two-dimensional detection frame in a certain coordinate system (for example, a world coordinate system).

[0119] For example, the spatial parameters of the two-dimensional detection frame can represent the area and height of the two-dimensional detection frame, and the second relative depth between the target mobile device and the observation object can be roughly estimated based on the area and height of the two-dimensional detection frame. In addition, the spatial parameters can also include the coordinates corresponding to the two-dimensional detection frame, and another second relative depth between the target mobile device and the observation object can be determined based on the result of inversely perspective projecting the two-dimensional detection frame based on the coordinates.

[0120] Specifically, for any observation object, the area of the corresponding two-dimensional detection frame can be determined, and then the second relative depth corresponding to the observation object can be roughly determined by using the relationship that the farther the distance between the observation object and the two-dimensional detection frame (i.e., the greater the relative depth), the smaller the area of the two-dimensional detection frame. For the convenience of description, the second relative depth determined based on the area of the two-dimensional detection frame is referred to as the area relative depth. Similarly, based on the height of the two-dimensional detection frame, the second relative depth corresponding to the two-dimensional detection frame, i.e., the height relative depth, can also be determined.

[0121] In addition, the bottom position coordinate (generally the coordinate of the middle position of the bottom) of the two-dimensional detection frame can be mapped to a three-dimensional world coordinate system (usually a ground coordinate system) by inverse perspective projection (Inverse Perspective Mapping, IPM) or the like, so as to extract the corresponding depth information and obtain the second relative depth corresponding to the two-dimensional detection frame, i.e., the projection relative depth.

[0122] It can be understood that the type of the observation object, such as a vehicle, a pedestrian, etc., can be determined first, and then the area relative depth, the height relative depth, etc., can be determined based on this, so as to be able to adapt to different types of observation objects.

[0123] In addition, the plurality of second relative depths described above can be selectively used based on actual situations. For example, if the observation object corresponds to a front vehicle, the area of the two-dimensional detection frame of the front vehicle is not suitable for representing the relative depth between the front vehicle and the ego vehicle in the state of being transversely or obliquely placed, and therefore the area relative depth of the front vehicle can not be determined, but the height relative depth and the projection relative depth can be calculated.

[0124] In the embodiment, the second relative depths are respectively determined based on a plurality of different manners, so that the second relative depth of the observation object determined subsequently is more accurate.

[0125] In step S204, the relative depth of the observation object is determined according to the first relative depth and the plurality of second relative depths.

[0126] For details, please refer to Figure 1 The step S104 of the embodiment shown will not be described again here.

[0127] In some optional embodiments, the step S204 of “determining the relative depth of the observation object according to the first relative depth and the plurality of second relative depths” includes a step A1.

[0128] In step A1, the first relative depth and the plurality of second relative depths are fused in a weighted manner to determine the relative depth of the observation object; the first relative depth and the plurality of second relative depths each have a corresponding weight.

[0129] In this embodiment, after determining the motion parameters such as the first relative depth and the second relative depths, the motion parameters are fused in a weighted manner, and the fusion result can more accurately represent the relative depth of the observed object.

[0130] For example, the weight of each first relative depth and second relative depth can be determined based on the confidence degree of obtaining the first relative depth and the second relative depth, and then the weighted processing can be performed based on the weights.

[0131] Optionally, the step A1 of fusing the first relative depth and the second relative depths in a weighted manner can include the step A11 or the step A12.

[0132] The step A11 performs weighted processing on the first relative depth and the second relative depths according to the first preset weight corresponding to the first relative depth and the second preset weight corresponding to the second relative depth.

[0133] The step A11 determines the first weight corresponding to the first relative depth and the second weight corresponding to the second relative depth by using the weighted least square method, and performs weighted processing on the first relative depth and the second relative depths according to the first weight corresponding to the first relative depth and the second weight corresponding to the second relative depth.

[0134] In this embodiment, the weights corresponding to the first relative depth and the second relative depth can be set in advance based on the actual situation. For ease of description, the weight corresponding to the first relative depth is referred to as the first preset weight, and the weight corresponding to the second relative depth is referred to as the second preset weight. After determining the first preset weight and the second preset weight, the first relative depth and the second relative depths can be weighted processed according to the first preset weight and the second preset weight, and the parameter fusion is realized.

[0135] Alternatively, the first weight corresponding to the first relative depth and the second weight corresponding to the second relative depth can also be dynamically determined based on the weighted least square method, and then the weighted processing is performed. For example, the Iteratively Reweighted Least Squares (IRLS) algorithm can be used to gradually approach the optimal solution through the iteration process, and the weighted fusion is realized to ensure the accuracy of the fusion result.

[0136] For example, for a certain observed object, the perception network can determine the first relative depth d1 between itself and the observed object, and determine the area relative depth d2, height relative depth d3 and projection relative depth d4 corresponding to the two-dimensional detection frame according to the size information of the two-dimensional detection frame of the observed object. Then, through the iterative weighted least squares algorithm, d1, d2, d3, and d4 can be fused to finally obtain the relative depth d of the observed object. The relative depth d can be used as a motion parameter of the observed object and recorded in the three-dimensional motion information.

[0137] In this embodiment, relying only on a single sensor, the camera, the object detection model and the perception network, such as a convolutional neural network, can obtain the perception results of the two-dimensional detection frame and the corresponding speed, relative depth and other information. This can greatly reduce the demand for processor computing power, thereby achieving target detection under low-cost and low-computing conditions, and is particularly suitable for long-distance object detection. The second relative depth is determined based on a variety of different methods, making the subsequent final determination of the relative depth of the observed object more accurate; the first relative depth determined by the perception network and the second relative depth determined based on the area, height and IPM projection of the two-dimensional detection frame are fused in a weighted manner to obtain the final relative depth of the observed object, which can effectively improve the depth estimation of the observed object.

[0138] Step S205 : generating three-dimensional motion information corresponding to the observed object according to the relative depth and the two-dimensional detection frame; the three-dimensional motion information represents the motion state of the observed object in the three-dimensional space.

[0139] For details, please see Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0140] Step S206, obtain the observation object queue corresponding to the observation object; wherein the observation object queue includes the three-dimensional motion information of the historical observation objects, the historical observation objects in the observation object queue and the observation object are associated observation objects, and the three-dimensional motion information of the historical observation objects is determined based on the historical two-dimensional images, and the acquisition time of the historical two-dimensional images is earlier than the acquisition time of the target two-dimensional images.

[0141] In this embodiment, as described above, the image acquisition device such as a camera can continuously acquire multiple frames of two-dimensional images, and each frame of two-dimensional image can detect and observe a corresponding object, and the object observed based on each frame of two-dimensional image is recorded; wherein, compared with the observation object of the target two-dimensional image, the object that has been recorded is the object observed before the target two-dimensional image, for the convenience of description, these previously observed objects are called historical observation objects. And the two-dimensional image corresponding to the historical observation object is called historical two-dimensional image, that is, by detecting the historical observation object in the historical two-dimensional image, each historical observation object can be identified.

[0142] It can be understood that each frame of two-dimensional image corresponds to a corresponding acquisition time, which can be, for example, the timestamp of the two-dimensional image. Since the historical two-dimensional image is a two-dimensional image acquired before the target two-dimensional image, the acquisition time of the historical two-dimensional image is earlier than the acquisition time of the target two-dimensional image.

[0143] Among them, the historical observation objects determined based on different frames of historical two-dimensional images may correspond to the same object, these historical observation objects are associated with each other, and can form a corresponding queue (Queue), in this embodiment, the queue used to record the associated objects is called object queue, and the object queue includes the associated historical observation objects. In addition, for the recently appearing object, only one historical observation object may be detected (i.e. only one frame of historical two-dimensional image detects the object), and there is no other object associated with it, so the object queue at this time only contains one historical observation object. It can be understood that since there may be multiple objects in the surrounding environment, multiple object queues can be constructed.

[0144] After identifying the observation object, the observation object and the pre-recorded object queue can be matched to determine whether they match. Specifically, based on judging whether the observation object and the object queue correspond to the same object, it is judged whether they match, and if they correspond to the same object, they match.

[0145] Specifically, if there is an object queue that matches the observation object in one or more pre-recorded object queues, the object queue corresponds to the observation object, for the convenience of description, the object queue corresponding to the observation object is called observation object queue.

[0146] For example, the observation object corresponds to the front vehicle A, if the front vehicle A is also detected in the previous frame and the corresponding object queue is formed, the observation object matches the object queue corresponding to the front vehicle A.

[0147] And the observation object can be added to the observation object queue to be associated with the observation object queue, and the observation object queue is expanded. Each object queue records the three-dimensional motion information of the corresponding object at different times, that is, the object queue records the three-dimensional motion information of the historical observation object; when an object is added to the object queue, the three-dimensional motion information of the object is recorded synchronously. Correspondingly, the observation object queue also records the three-dimensional motion information of the observation object, so as to facilitate subsequent processing based on the three-dimensional motion information of the observation object.

[0148] It can be understood that the observation object becomes a new historical observation object afterwards. Accordingly, when recording the historical observation object before, for example, at a certain historical time before, the object determined at the historical time is also an "observation object", that is, the corresponding three-dimensional motion information can also be associated and recorded based on the method.

[0149] In step S207, the fitting result between the three-dimensional motion information and the collection time is obtained by fitting according to the three-dimensional motion information of the observation object, the three-dimensional motion information of the historical observation object in the observation object queue, the collection time of the historical two-dimensional image and the collection time of the target two-dimensional image.

[0150] In this embodiment, the observation object and the historical observation object in the observation object queue correspond to the same object, that is, the three-dimensional motion information of the observation object and the three-dimensional motion information of the historical observation object in the observation object queue all represent the three-dimensional motion information of the same object at different times, so that the fitting based on these three-dimensional motion information can obtain a fitting result capable of representing the change trend of the three-dimensional motion information, and the fitting result is specifically the change of the three-dimensional motion information with the collection time. For example, the least square method can be used to realize the fitting.

[0151] In step S208, the three-dimensional motion information of the observation object is corrected according to the fitting result to generate the corrected motion information corresponding to the observation object, and the corrected motion information corresponding to the observation object is taken as the new three-dimensional motion information corresponding to the observation object.

[0152] In this embodiment, the three-dimensional motion information of the observation object determined in the above step S205 is an observation value collected, and the new three-dimensional motion information of the observation object can be calculated according to the fitting result, which is an estimated value obtained based on the fitting result. The observation value is easily affected by observation errors (such as camera collection errors, etc.), and there may be a large deviation, especially for distant objects; and the estimated value is obtained by fitting the observation values of multiple objects such as the observation object and the historical observation object, which can reduce the influence of observation errors as much as possible, so that the finally calculated estimated value can more accurately represent the motion state of the observation object.

[0153] Therefore, the three-dimensional motion information of the observation object is corrected according to the fitting result, for example, the three-dimensional motion information of the observation object is corrected to the estimated value, so that the corrected three-dimensional motion information can more accurately represent the motion state of the observation object. For the convenience of description, the corrected three-dimensional motion information is referred to as corrected motion parameter.

[0154] For example, since each object in the observation object queue corresponds to a frame of two-dimensional image, the three-dimensional motion information corresponds to different time points (collection time). According to the three-dimensional motion information of multiple time points, the fitting function between the three-dimensional motion information and the time can be obtained, and then based on the fitting function, the estimated value of the three-dimensional motion information of the observation object at the collection time corresponding to the target two-dimensional image, that is, the corrected motion information, can be determined, so as to complete the correction.

[0155] Moreover, after determining the corrected motion information corresponding to the observation object at this time, the corrected motion information can be used as the new three-dimensional motion information corresponding to the observation object, that is, the three-dimensional motion information of the observation object is corrected by using the corrected motion information to obtain the new three-dimensional motion information corresponding to the observation object, and subsequent path planning is based on the new three-dimensional motion information.

[0156] The data processing method provided in this embodiment can realize the updating of the object queue by matching the observation object in the target two-dimensional image with the object queue, and can obtain more accurate corrected motion information based on the fitting result by using the three-dimensional motion information of multiple objects in the object queue, and correct the three-dimensional motion information of the observation object based on the corrected motion information, so as to accurately represent the motion state of the observation object. This method can determine the three-dimensional motion information of the object only by using the images collected by the camera, and can realize pure visual object detection without the need for data fusion and other processing, and the implementation is simple. Moreover, the required computing power for fitting calculation is less, and the three-dimensional motion information of the object can be accurately determined without consuming too much computing power. In addition, the required computing power of this method is irrelevant to the distance of the object, and when detecting a remote object, no additional computing power is required, which can greatly reduce the demand for processor computing power.

[0157] In this embodiment, another data processing method is provided, which can be applied to mobile devices that need to continuously detect objects, such as smart cars, robots, etc., and can also be applied to servers on the network side or distributed systems on the network side, which can be determined based on actual conditions. Figure 3 The flowchart of the data processing method according to the embodiments of the present application is shown in FIG. 1, which includes the following steps. Figure 3 As shown in FIG. 1, the flowchart of the data processing method according to the embodiments of the present application includes the following steps.

[0158] In step S301, a target two-dimensional image to be processed is acquired.

[0159] For details, please refer to Figure 1 In step S101 of the embodiment shown, no further elaboration is given here.

[0160] In step S302, according to the target two-dimensional image, a two-dimensional detection frame of an observed object in the target two-dimensional image is determined, and a first relative depth of the observed object is determined.

[0161] For details, please refer to Figure 1 In step S102 of the embodiment shown, or Figure 2 In step S202 of the embodiment shown, no further elaboration is given here.

[0162] In step S303, according to the two-dimensional detection frame, a plurality of second relative depths of the observed object in different dimensions are determined.

[0163] For details, please refer to Figure 1 In step S103 of the embodiment shown, or Figure 2 In step S203 of the embodiment shown, no further elaboration is given here.

[0164] In step S304, according to the first relative depth and the plurality of second relative depths, a relative depth of the observed object is determined.

[0165] For details, please refer to Figure 1 In step S104 of the embodiment shown, or Figure 2 In step S204 of the embodiment shown, no further elaboration is given here.

[0166] In step S305, according to the relative depth and the two-dimensional detection frame, three-dimensional motion information corresponding to the observed object is generated; the three-dimensional motion information represents a motion state of the observed object in a three-dimensional space.

[0167] For details, please refer to Figure 1 In step S105 of the embodiment shown, no further elaboration is given here.

[0168] In step S306, an observed object queue corresponding to the observed object is acquired; wherein the observed object queue includes three-dimensional motion information of a historical observed object, the historical observed object in the observed object queue and the observed object are associated observed objects, and the three-dimensional motion information of the historical observed object is determined according to a historical two-dimensional image, and the acquisition time of the historical two-dimensional image is earlier than the acquisition time of the target two-dimensional image.

[0169] For details, please refer to Figure 2 In step S206 of the embodiment shown, the observed object is provided with a corresponding identifier; the step S306 of acquiring the observed object queue corresponding to the observed object includes steps S3061 to S3062.

[0170] Step S3061, judging whether the identity of the observation object is same as the identity of the historical observation object in the object queue recorded in advance.

[0171] Step S3062, in the case that the identity of the observation object is same as the identity of the historical observation object in the object queue, taking the object queue to which the historical observation object with the same identity as the observation object belongs as the observation object queue; the historical observation object in the observation object queue and the observation object correspond to the same object.

[0172] In the embodiment, the observation object is provided with a corresponding identity (for example, ID), and different objects correspond to different identities. Wherein, the object detection is performed on the two-dimensional image, and in addition to the objects in the two-dimensional image can be recognized, the identity (ID) of each object can also be generated; and for the objects corresponding to the same object, the identity determined by the object detection is unchanged in normal cases. Therefore, the association of the objects can be performed based on the identity of the objects.

[0173] Specifically, the object queue includes a plurality of associated historical observation objects, and the historical observation objects correspond to the same object and have the same identity, and the identity can also be taken as the identity of the object queue. After the identity of the observation object is obtained, the identity of the observation object and the identity of the historical observation object in the object queue can be compared, and if they are the same, it can be explained that the observation object and the object queue correspond to the same object, and therefore they are matched, and the object queue is the observation object queue corresponding to the observation object. It can be understood that the historical observation object in the observation object queue and the observation object correspond to the same object.

[0174] If the identity of the observation object is not the same as the identity of any object queue, it can be determined that the observation object fails to match, or further matching can be performed based on other manners.

[0175] In some optional embodiments, the target two-dimensional image is used to represent the vehicle situation around the target mobile device; and the step S306 of obtaining the observation object queue corresponding to the observation object can include the following steps B1 to B3.

[0176] Step B1, judging whether the observation object is a preset type object; wherein, the preset type object is an object whose influence on the driving safety of the target mobile device meets a preset condition.

[0177] Step B2, in the case that the observation object is the preset type object, performing the intersection and union association on the observation object and the historical observation object in the object queue for the object queue recorded in advance, to obtain the intersection and union association result between the historical observation object in the object queue and the observation object.

[0178] Step B3, determining the observation object queue of the observation object from the object queue according to the historical observation objects in the object queue and the IoU association result between the observation objects.

[0179] In this embodiment, the observation object queue corresponding to the observation object can be determined by calculating the IoU. In step S3061, if the identifier of the observation object is different from the identifier of any object queue, steps B1 to B3 can be performed.

[0180] If the identifier of the observation object is different from the identifier of any object queue, i.e., there is no object queue with the same identifier as the observation object, it is further determined whether the observation object is a preset type object. The preset type object is an object whose impact on the driving safety of the target mobile device meets a preset condition. For example, an object whose impact on the driving safety of the target mobile device exceeds a preset degree is regarded as a preset type object. For example, the preset type object is a most critical dynamic object (MCDO). In the automatic driving scenario, there is a most critical dynamic object that affects the driving of the vehicle. Since the first object in the vehicle lane has the highest degree of danger, the first object in the vehicle lane can generally be regarded as the MCDO.

[0181] If the observation object is a preset type object, e.g., a most critical dynamic object, to improve the association success rate of the MCDO, the observation object corresponding to the MCDO is associated by IoU in addition to the identifier association, i.e., the IoU between the observation object and the historical observation objects in the object queue is determined, and the IoU association is performed based on the IoU. If the IoU of the two exceeds a preset threshold, it is determined that the two correspond to the same object, i.e., the two are associated.

[0182] To ensure the accuracy of the association, the IoU association is preferentially performed with the latest historical observation object (i.e., the last historical observation object) in the object queue.

[0183] Optionally, the step S306 of obtaining the observation object queue corresponding to the observation object can further include steps B4 to B5.

[0184] Step B4, if the observation object is not a preset type object, or the IoU association result indicates that each object queue is not the observation object queue, the nearest neighbor association between the observation object and the historical observation objects in the object queue is performed for each object queue, and the nearest neighbor association result between the historical observation objects in the object queue and the observation object is obtained.

[0185] Step B5, determining the observation object queue from the object queue according to the historical observation objects in the object queue and the nearest neighbor association result between the observation object and the observation objects.

[0186] In this embodiment, if the observation object is not a preset type object, for example, is not a most critical driving object, or the observation object is a preset type object but the IOU association fails (i.e., the IOU association result indicates that each object queue is not the observation object queue), further nearest neighbor algorithm is used for association, i.e., the nearest neighbor association is performed on the observation object and the historical observation objects in the object queue, and the corresponding nearest neighbor association result is obtained. For example, the nearest neighbor association can be performed by using the greedy algorithm, if the nearest neighbor association result indicates that the observation object and the historical observation object in a certain object queue are nearest neighbors, it is considered that the two are associated, and the object queue can be taken as the observation object queue.

[0187] Similarly, to ensure the accuracy of the association, the nearest neighbor association is also preferentially performed with the latest historical observation object (i.e., the last historical observation object) in the object queue.

[0188] In this embodiment, by using multiple association methods such as the identification association, the IOU association, and the nearest neighbor association, the association success rate can be effectively ensured, and the problem of association omission can be avoided; and for the MCDO and other important preset type objects, the IOU association is additionally performed, which can improve the association success rate of the MCDO, ensure the detection accuracy and recall rate of the MCDO, and thus improve the safety of the automatic driving.

[0189] Optionally, the method further includes the following steps C1 and / or step C2.

[0190] Step C1, creating a new first object queue for the observation object in the absence of the observation object queue.

[0191] Step C2, if the second object queue exists, increasing the unassociated number of the second object queue by one; the historical observation object in the second object queue is not associated with any observation object, and the unassociated number of the second object queue is the number of times that the second object queue is continuously unassociated with objects; and in the case that the unassociated number of the second object queue exceeds a first preset number, the second object queue is deleted.

[0192] In this embodiment, for the observation object, if the observation object queue does not exist, i.e., the observation object association fails, for example, after the identification association, the IOU association, and the nearest neighbor association, no object queue that can match the observation object is found, i.e., no observation object queue corresponding to the observation object exists, at this time, the observation object can be taken as a new appearing object, and therefore a new object queue can be created for the observation object, for the convenience of distinguishing and describing, the newly created object queue is referred to as the first object queue.

[0193] And, for a certain object queue that has been recorded, its unassociated times can be determined in real time, the unassociated times refer to the number of times that the object queue is continuously unassociated with objects, which is also the number of frames of the two-dimensional image. If the unassociated times exceed the first preset number of times, it means that the object corresponding to the object queue has not been detected for a long time, and the object may not be within the field of view, so the object queue can be deleted to avoid affecting the association of subsequent other observation objects.

[0194] Specifically, when associating each observation object in the target two-dimensional image with the object queue, there may be an object queue that cannot be associated with any observation object. For ease of description, this object queue is referred to as a second object queue. It can be understood that the unassociated times of the second object queue are the number of times that the second object queue is continuously unassociated with objects.

[0195] When associating the target two-dimensional image, since the second object queue cannot be associated with any observation object, the unassociated times of the second object queue can be incremented by one; if the unassociated times of the second object queue exceed the first preset number of times, it means that the object corresponding to the second object queue has not been detected for a long time, so the second object queue can be deleted.

[0196] Figure 4 An example of associating observed objects is shown. As shown in Figure 4 At the current time, three object queues have been recorded, namely object queue A, object queue B, and object queue C. Among them, object queue A includes five historical observation objects, namely a1, a2, a3, a4, and a5; object queue B includes five historical observation objects, namely b1, b2, b3, b4, and b5; object queue C includes seven historical observation objects, namely c1, c2, c3, c4, c5, c6, and c7.

[0197] And, if the target two-dimensional image collected at the current time includes two observation objects, namely observation object and observation object.

[0198] If it is determined after matching processing that the observation object matches object queue A (for example, their IDs are the same), but the observation object fails to match. In this case, as shown in Figure 4 The observation object is associated with object queue A, so object queue A adds the observation object. And, a new object queue is created for the observation object, namely object queue D, which only contains the observation object at this time.

[0199] For object queue B and object queue C, neither of them successfully matches to an object in the current frame, so the number of unassociated times of both of them is increased by 1; if the number of unassociated times of object queue B does not exceed the first preset number, the object queue B is kept (for example, due to reasons such as occlusion or light, the object is missed, and if the object matching the object queue B is detected again in the next frame, the number of unassociated times of the object queue B can be set to 0). On the contrary, if the number of unassociated times of object queue C exceeds the first preset number (for example, the first preset number is 7), the object queue C is deleted. As shown in FIG. 6, at this time, only object queue A, object queue B and object queue D are recorded. Figure 4

[0200] In step S307, fitting is performed according to the three-dimensional motion information of the observed object, the three-dimensional motion information of the historical observed object in the observed object queue, the collection time of the historical two-dimensional image and the collection time of the target two-dimensional image, to obtain a fitting result between the three-dimensional motion information and the collection time.

[0201] For details, refer to step S207 of the embodiment shown in FIG. 6, which will not be described here again. Figure 2

[0202] Optionally, the step S307 of fitting according to the three-dimensional motion information of the observed object, the three-dimensional motion information of the latest k historical observed objects in the observed object queue and the corresponding collection time can include: fitting according to the three-dimensional motion information of the observed object, the three-dimensional motion information of the latest k historical observed objects in the observed object queue and the corresponding collection time when the number of objects in the observed object queue exceeds the first preset threshold; k≥3, and the latest k historical observed objects are the last k historical observed objects determined according to the collection time of the historical two-dimensional image corresponding to the historical observed objects in the observed object queue from the first to the last.

[0203] In this embodiment, for the observed object queue, the number of objects in the observed object queue can be determined; wherein if the corresponding object is associated in a frame, the number of objects in the observed object queue is also increased by 1; it can be understood that the number is the number of all historical observed objects in the observed object queue, plus the observed object, that is, plus 1. If the number exceeds the preset threshold, it means that there are enough objects in the observed object queue, and the length (size) of the observed object queue meets the threshold condition, at which time fitting can be performed.

[0204] ​​In the fitting, the latest objects in the object queue are selected, i.e., the observation object and the latest k historical observation objects, and the fitting is performed based on the three-dimensional motion information of the objects. In this embodiment, the historical observation objects can be arranged in the order of the collection time from early to late (i.e., from old to new) based on the collection time of the historical two-dimensional images, and thus the last k historical observation objects determined are the latest k historical observation objects.

[0205] In addition, if the fitting is performed based on the motion parameters of only two or three objects, the fitting result is difficult to accurately correct the three-dimensional motion information of the current frame, and thus k≥3, i.e., the fitting is performed based on the three-dimensional motion information of at least four objects. In addition, a larger value can be selected, such as k=10, 15, etc., which can be determined based on the actual situation.

[0206] The preset threshold can also be determined based on the actual situation, for example, the preset threshold is 15, 20, etc. In addition, the preset threshold can be greater than k to ensure that there is enough data for fitting. In addition, k can be a fixed value or dynamically changed, which is not limited in this embodiment.

[0207] In this embodiment, the fitting is performed based on the observation object and the latest k historical observation objects in the object queue, which can ensure the fitting effect.

[0208] In step S308, the three-dimensional motion information of the observation object is corrected based on the fitting result, the corrected motion information corresponding to the observation object is generated, and the corrected motion information corresponding to the observation object is used as the new three-dimensional motion information corresponding to the observation object.

[0209] For details, please refer to Figure 2 The step S208 of the embodiment shown is not described herein.

[0210] Optionally, the method can further include: inputting the target two-dimensional image into a preset perception network to obtain the speed of the observation object in the target two-dimensional image output by the perception network; and for each observation object, if the speed of the observation object is less than a preset value, and the number of historical observation objects associated with the observation object and having a speed less than the preset value in a preset time period adjacent to the collection time of the target two-dimensional image exceeds a second preset number of times, the speed in the corrected motion information is set to zero.

[0211] In this embodiment, the perception network set in advance can not only calculate the first relative depth, but also determine the speed of each object in the image, for example, the perception network can be a velocity network (Velocity Net). Therefore, the target two-dimensional image is input into the perception network, and the speed of the observation object in the target two-dimensional image can be determined based on the output of the perception network.

[0212] If the speed of the observation object is less than the preset value, it indicates that the current speed of the observation object is too small, and the observation object can be static. For example, if the perception network determines that the speed of the observation object is less than the preset value, the perception network can output a static flag of the observation object, and the static flag indicates that the observation object is in a static state.

[0213] In general, the motion parameters of the observation object can be corrected directly based on the fitting result. However, for a static observation object, the perception network generally outputs a static flag indicating that the observation object is static. In addition, if the number of historical observation objects associated with the observation object and having a speed less than the preset value exceeds a second preset number of times in a preset time period adjacent to the acquisition time of the target two-dimensional image, it indicates that the perception network always considers the observation object to be static in the preset time period, and the perception network will always output the static flag, for example, outputting a static flag every frame.

[0214] If the number of times of outputting the static flag by the perception network exceeds the second preset number of times for the observation object in the preset time period, it can be determined that the observation object is static, and therefore the speed obtained by fitting can be discarded, and the speed in the corrected motion parameters can be set to zero to accurately generate the corrected motion information indicating that the object is static. The second preset number of times can be 3, 5, 10, etc., or can be 1 (i.e., detecting the static flag and setting the speed to zero), which can be determined based on actual conditions.

[0215] In some optional embodiments, the target two-dimensional image is used to represent the traffic situation around the target mobile device; and the method further includes step D1.

[0216] Step D1, determining path planning information of the target mobile device according to the three-dimensional motion information corresponding to the observation object, so that the target mobile device travels according to the path planning information.

[0217] In this embodiment, after the three-dimensional motion information corresponding to the observation object is determined, the path planning information corresponding to the target mobile device can be obtained based on the three-dimensional motion information, and the path planning can be performed based on the path planning information, and then the control of the target mobile device is realized. The path planning information is used to control the travel path of the target mobile device; for example, the path planning information includes coordinates of each position point in the travel path of the target mobile device.

[0218] The three-dimensional motion information in step D1 can be the three-dimensional motion information corresponding to the observation object determined in step S305, or can be new three-dimensional motion information corresponding to the observation object determined in step S308, which is not limited in this embodiment.

[0219] Optionally, the step D1 of determining the path planning information of the target mobile device according to the three-dimensional motion information of the observation object comprises steps D11 and D12.

[0220] The step D11 determines whether the observation object is a key observation object.

[0221] The step D12, in the case that the observation object is a key observation object, determines the path planning information of the target mobile device according to the three-dimensional motion information of the observation object.

[0222] In this embodiment, there are generally multiple observation objects in the target two-dimensional image, each observation object corresponds to an object in the surrounding environment, and the path planning can be performed based on the three-dimensional motion information of all observation objects. However, some observation objects have little influence on the path planning, such as other vehicles far away from the target mobile device, and therefore, when generating the path planning information, it can be determined whether the observation object is a key observation object.

[0223] The key observation object is an object whose influence on the path planning exceeds a preset value. If the observation object is a key observation object, the path planning information of the target mobile device is determined according to the three-dimensional motion information of the observation object. By determining whether the observation object is a key observation object, unnecessary three-dimensional motion information can be avoided for path planning, the processing flow is simplified, and the processing amount is reduced.

[0224] Optionally, the step D11 of determining whether the observation object is a key observation object can comprise: determining whether the observation object is a key observation object according to at least one of the number of objects in the observation object queue, a fitting index representing a fitting effect, and a collision time predicted according to the three-dimensional motion information of the observation object; and the collision time is the time when the target mobile device collides with the observation object.

[0225] The observation object is determined to be a key observation object in the case that at least one of the following conditions is met: the number of objects in the observation object queue is greater than a preset number, the fitting effect represented by the fitting index is greater than a preset fitting effect, and the collision time is less than a preset collision time.

[0226] In this embodiment, the number of objects in the observation object queue can be determined. If the number of objects in the observation object queue is greater, it means that the detection result of the object at this time is relatively accurate, and the determined three-dimensional motion information is also relatively accurate, which can ensure the effect of subsequent path planning based on the three-dimensional motion information.

[0227] Similarly, based on the fitting of the three-dimensional motion information of multiple objects, a fitting index capable of representing the fitting effect can be obtained, which can be, for example, a mean squared error (MSE). The better the fitting effect represented by the fitting index (for example, the smaller the mean squared error), the more accurate the fitting result, and the more likely it is to be based on the three-dimensional motion information for path planning.

[0228] In addition, according to the relative depth and speed information of the observed objects, the predicted time to collision (TTC) of the target mobile device and each observed object can be calculated. The shorter the collision time, the more likely there is a collision risk, so the more likely it is to be based on the three-dimensional motion information for path planning.

[0229] Among them, the number of objects in the observed object queue, the fitting index and the collision time can be set with corresponding thresholds, that is, the preset number, the preset fitting effect and the preset collision time, and by comparing the three, it can be determined whether the observed object is a key observed object. For example, if the number of objects in the observed object queue is greater than the preset number, the fitting effect represented by the fitting index is greater than the preset fitting effect, and the collision time is less than the preset collision time, it is determined that the observed object is a key observed object, and path planning needs to be based on the three-dimensional motion information of the observed object.

[0230] In this embodiment, after determining the three-dimensional motion information corresponding to each observed object, the controller (such as a vehicle controller) implementing the data processing method can continue to perform path planning; or each software module in the controller can cooperate, and after determining the three-dimensional motion information, the three-dimensional motion information can be published to other processing modules downstream, and path planning can be performed by other processing modules.

[0231] For example, for each identified observed object, a detection box representing the object trajectory can be set, which is referred to as a track detection box (Track) in the following. Since different object queues correspond to different objects, a corresponding track detection box can be created for each object queue.

[0232] After determining the corrected motion parameters of the observed objects, the motion parameters of the track detection box can be set as the corrected motion parameters, so that the motion state of the observed object can be more accurately represented based on the track detection box, and subsequent adaptive control can be performed based on the track detection box. Among them, for the track detection box with corrected motion parameters, it can be published externally to enable subsequent control based on the published track detection box.

[0233] For example, for a vehicle controller, where various software modules can be divided, one of the software modules determines the track bounding box according to the above method, and publishes the track bounding box containing the corrected motion parameters to the downstream other software modules, so that the downstream software modules can perform path planning control based on it.

[0234] In this embodiment, the corrected motion parameters can be used only to set the motion parameters of the track bounding box, and the three-dimensional motion information of each observed object recorded in the object queue is actually collected observation value, not the estimated value obtained by fitting, to ensure that the real observation value is used each time fitting.

[0235] In this embodiment, if the observed object is a key observed object, the track bounding box of the observed object needs to be published to the downstream module for path planning; if the observed object is a key observed object, the track bounding box can not be created, or only the motion parameters of the track bounding box are updated, but the publishing action is not performed. Based on this, unnecessary track bounding boxes can be avoided, the processing flow is simplified, and the processing amount of the downstream is also reduced.

[0236] Figure 5 A flowchart of the data processing method is shown. As shown in Figure 5 For a target two-dimensional image, the velocity v and relative depth d of the observed object can be determined based on an object detection model (such as a convolutional neural network) and a perception network (Velocity Net), respectively.

[0237] Based on the two-dimensional detection box (i.e. 2D detection box) and the relative depth d of the observed object identified by the object detection model, a three-dimensional detection box (i.e. 3D detection box) is obtained by back projection, which can represent the three-dimensional motion parameters of the observed object, and a corresponding track bounding box (i.e. Track) is created based on the 3D detection box.

[0238] In addition, an observation cache pool can be constructed, and each object queue can be recorded based on the cache pool. For example, a cache pool with a maximum cache number of K frames can be constructed, that is, each object queue can record at most K observed objects.

[0239] For each observed object, identification association, IOU association (only for MCDO), and nearest neighbor association can be performed, which can be referred to the above description. As long as one kind of association is successful, the state of the Track can be updated, that is, the motion parameters of the Track are updated using the corrected velocity and relative depth. If the nearest neighbor association also fails, the Track does not need to be updated, but a new object queue, that is, the first object queue described above, is created.

[0240] For each Track after state update, it can be determined whether it meets the preset publishing condition, wherein, whether the observation object is a key observation object is judged to determine whether the corresponding Track meets the publishing condition; if the observation object is a key observation object, the Track corresponding to the observation object meets the publishing condition, and the Track can be published to the downstream, otherwise, the above process is repeated based on the two-dimensional image of the next frame, and the Track is updated again until it can be published, or the process is ended.

[0241] The data processing method provided in the embodiment can effectively guarantee the association success rate, avoid the problem of association omission, and improve the association success rate of the MCDO, so as to guarantee the detection accuracy and recall rate of the MCDO and improve the safety of automatic driving. The observation results of multiple frames are used for fitting, which can correct the motion state estimation of the Track, can improve the estimation result, and can stably estimate the motion state of the object when the object detection is missed due to light and the like, and can achieve high-precision state estimation for close-range targets, long-distance targets, and stationary targets. In addition, the method can also be combined with other object detection methods, and has a wide application range.

[0242] In the embodiment, a data processing apparatus is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and is contemplated.

[0243] The embodiment provides a data processing apparatus, as shown in the following Figure 6 The apparatus comprises:

[0244] The acquisition module 601 is configured to acquire a target two-dimensional image to be processed.

[0245] The depth determination module 602 is configured to determine a two-dimensional detection frame of an observation object in the target two-dimensional image and a first relative depth of the observation object according to the target two-dimensional image, determine a plurality of second relative depths of the observation object in different dimensions according to the two-dimensional detection frame, and determine a relative depth of the observation object according to the first relative depth and the plurality of second relative depths.

[0246] The processing module 603 is configured to generate three-dimensional motion information corresponding to the observation object according to the relative depth and the two-dimensional detection frame, wherein the three-dimensional motion information represents a motion state of the observation object in a three-dimensional space.

[0247] In an optional implementation, the depth determination module 602 determines, according to the two-dimensional bounding box, a plurality of second relative depths of the observed object in different dimensions, including:

[0248] determining spatial parameters of the two-dimensional bounding box in different dimensions;

[0249] determining, according to the spatial parameters of the two-dimensional bounding box, a plurality of second relative depths of the observed object in different dimensions.

[0250] In some optional implementations, the plurality of second relative depths of the observed object in different dimensions include at least two of an area relative depth of the observed object in an area dimension, a height relative depth of the observed object in a height dimension, and a projection relative depth of the observed object in a projection dimension; and the spatial parameters include at least two of an area of the two-dimensional bounding box, a height of the two-dimensional bounding box, and a bottom position coordinate of the two-dimensional bounding box.

[0251] The area relative depth of the observed object is determined according to the area of the two-dimensional bounding box; the height relative depth of the observed object is determined according to the height of the two-dimensional bounding box; and the projection relative depth of the observed object is determined by inversely perspective projecting the two-dimensional bounding box using the bottom position coordinate of the two-dimensional bounding box.

[0252] In some optional implementations, the depth determination module 602 determines, according to the target two-dimensional image, a two-dimensional bounding box of an observed object in the target two-dimensional image and a first relative depth of the observed object, including:

[0253] performing object detection on the target two-dimensional image according to a preset object detection model to identify a two-dimensional bounding box of an observed object in the target two-dimensional image;

[0254] inputting the target two-dimensional image into a preset perception network to obtain a first relative depth of the observed object in the target two-dimensional image output by the perception network.

[0255] In some optional implementations, after the three-dimensional motion information corresponding to the observed object is generated, the processing module 603 is further configured to:

[0256] obtain an observed object queue corresponding to the observed object; the observed object queue includes three-dimensional motion information of a historical observed object, the historical observed object in the observed object queue is an associated observed object of the observed object, and the three-dimensional motion information of the historical observed object is determined according to a historical two-dimensional image, a collection time of the historical two-dimensional image being earlier than a collection time of the target two-dimensional image;

[0257] fitting the three-dimensional motion information of the observation object, the three-dimensional motion information of the historical observation object in the observation object queue, the acquisition time of the historical two-dimensional image, and the acquisition time of the target two-dimensional image, to obtain a fitting result between the three-dimensional motion information and the acquisition time;

[0258] According to the fitting result, the three-dimensional motion information of the observation object is corrected to generate the corrected motion information corresponding to the observation object, and the corrected motion information corresponding to the observation object is taken as the new three-dimensional motion information corresponding to the observation object.

[0259] In some optional embodiments, the observation object is provided with a corresponding identifier; the processing module 603 acquires the observation object queue corresponding to the observation object, including:

[0260] determining whether the identifier of the observation object is the same as the identifier of the historical observation object in the pre-recorded object queue;

[0261] In the case where the identifier of the observation object is the same as the identifier of the historical observation object in the object queue, the object queue to which the historical observation object with the same identifier as the observation object belongs is taken as the observation object queue; the historical observation object in the observation object queue and the observation object correspond to the same object.

[0262] In some optional embodiments, the target two-dimensional image is used to represent the vehicle condition around the target mobile device; the processing module 603 acquires the observation object queue corresponding to the observation object, including:

[0263] determining whether the observation object is a preset type object; wherein the preset type object is an object whose influence on the driving safety of the target mobile device meets a preset condition;

[0264] In the case where the observation object is a preset type object, the intersection and union ratio correlation between the observation object and the historical observation object in the pre-recorded object queue is obtained by performing intersection and union ratio correlation on the observation object and the historical observation object in the object queue;

[0265] According to the intersection and union ratio correlation between the historical observation object in all object queues and the observation object, the observation object queue is determined from the object queue.

[0266] In some optional embodiments, the processing module 603 acquires the observation object queue corresponding to the observation object further includes:

[0267] If the observation object is not a preset type object, or the Jaccard index result indicates that none of the object queues is an observation object queue, the processing module 603 is further configured to perform nearest neighbor association between the observation object and historical observation objects in each object queue to obtain a nearest neighbor association result between the historical observation objects in the object queue and the observation object.

[0268] The processing module 603 is further configured to determine an observation object queue from the object queues according to the nearest neighbor association result between the historical observation objects in all the object queues and the observation object.

[0269] In some optional embodiments, the processing module 603 is further configured to:

[0270] In the absence of an observation object queue, a new first object queue is created for the observation object.

[0271] And / or,

[0272] If there is a second object queue, the number of times of non-association of the second object queue is increased by one; the historical observation objects in the second object queue are not associated with any of the observation objects, and the number of times of non-association of the second object queue is the number of times of continuous non-association of the object by the second object queue; and the second object queue is deleted if the number of times of non-association of the second object queue exceeds a first preset number of times.

[0273] In some optional embodiments, the processing module 603 is configured to perform fitting according to the three-dimensional motion information of the observation object, the three-dimensional motion information of the historical observation objects in the observation object queue, the acquisition time of the historical two-dimensional image, and the acquisition time of the target two-dimensional image, including:

[0274] If the number of objects in the observation object queue exceeds a first preset threshold, fitting is performed according to the three-dimensional motion information of the observation object, the three-dimensional motion information of the latest k historical observation objects in the observation object queue, and the corresponding acquisition times; k≥3, and the latest k historical observation objects are the last k historical observation objects determined according to the acquisition times of the historical two-dimensional images corresponding to the historical observation objects in the observation object queue from early to late.

[0275] In some optional embodiments, the processing module 603 is further configured to:

[0276] The processing module 603 is further configured to input the target two-dimensional image into a preset perception network to obtain the speed of the observation object in the target two-dimensional image output by the perception network.

[0277] If the speed of the observation object is less than a preset value, and the number of historical observation objects associated with the observation object and having a speed less than a preset value exceeds a second preset number of times within a preset time period adjacent to the time at which the target two-dimensional image is captured, the speed in the corrected motion information is set to zero.

[0278] In some optional embodiments, the target two-dimensional image is used to represent a traffic situation around the target mobile device; and the processing module 603 is further configured to:

[0279] According to the three-dimensional motion information corresponding to the observation object, determine path planning information of the target mobile device, so that the target mobile device travels according to the path planning information.

[0280] In some optional embodiments, the processing module 603 determines the path planning information of the target mobile device according to the three-dimensional motion information corresponding to the observation object, including:

[0281] Determine whether the observation object is a key observation object;

[0282] If the observation object is a key observation object, determine the path planning information of the target mobile device according to the three-dimensional motion information corresponding to the observation object.

[0283] In some optional embodiments, the processing module 603 determines whether the observation object is a key observation object, including:

[0284] Determine whether the observation object is a key observation object according to at least one of the number of objects in an observation object queue corresponding to the observation object, a fitting index representing a fitting effect, and a collision time predicted according to the three-dimensional motion information corresponding to the observation object; the collision time is the time at which the target mobile device collides with the observation object.

[0285] If at least one of the number of objects in the observation object queue is greater than a preset number, the fitting effect represented by the fitting index is greater than a preset fitting effect, and the collision time is less than a preset collision time, the observation object is determined to be a key observation object.

[0286] In some optional embodiments, the depth determination module 602 determines the relative depth of the observation object according to the first relative depth and the plurality of second relative depths, including:

[0287] Determine the relative depth of the observation object by fusing the first relative depth and the plurality of second relative depths in a weighted manner; the first relative depth and the plurality of second relative depths each have a corresponding weight.

[0288] In some optional embodiments, the depth determination module 602 fuses the first relative depth and the second relative depths in a weighted manner, including:

[0289] According to a first preset weight corresponding to the first relative depth and a second preset weight corresponding to the second relative depth, the first relative depth and the second relative depths are weighted.

[0290] Alternatively, a weighted least square method is used to determine a first weight corresponding to the first relative depth and a second weight corresponding to the second relative depth, and according to the first weight corresponding to the first relative depth and the second weight corresponding to the second relative depth, the first relative depth and the second relative depths are weighted.

[0291] In some optional embodiments, the two-dimensional detection frame is a bounding box for positioning and identifying an observed object in the target two-dimensional image, and the bounding box includes one of the following: a rectangular box, a polygonal box, a circular or elliptical box, a rotated rectangular box, and a self-defined shape box.

[0292] Further function descriptions of the above-mentioned modules and units are the same as those of the corresponding embodiments, and will not be repeated here.

[0293] The data processing apparatus in the embodiment is presented in the form of a functional unit. The unit herein refers to an ASIC (Application Specific Integrated Circuit) circuit, including a processor and a memory for executing one or more software or fixed programs, and / or other devices that can provide the above functions.

[0294] The embodiment of the present application also provides a computer device having the above Figure 6 data processing apparatus.

[0295] Please refer to Figure 7 , Figure 7 is a structural schematic diagram of a computer device provided by the optional embodiment of the present application, as Figure 7As shown, the computer device includes one or more processors 10, memory 20, and interfaces 30 for external devices such as a keyboard and a mouse and peripheral devices such as disk devices or other storage devices. One or more busses 10 can be used to implement the interface between the various internal and external components and can be implemented using any one or more of a variety of bus technologies including a System bus, PCI, SCSI, AGP, Super- I / O bus, etc. Furthermore, various buses can be used in front side buses, back side buses, and other bus configurations based on any bus or messaging technology known to those skilled in the art. Figure 7 The processor 10 is used in the embodiments described herein as an example.

[0296] The processor 10 can be a central processing unit, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.

[0297] The memory 20 stores instructions that can be executed by the at least one processor 10, so that the at least one processor 10 can perform the method shown in the above embodiments.

[0298] The memory 20 can include a program region and a data region. The program region can store an operating system and an application program required by at least one function. The data region can store data created according to the use of the computer device. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory disposed remotely with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0299] The memory 20 can include a volatile memory such as a random access memory, and can further include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk. The memory 20 can further include a combination of the above-mentioned memories.

[0300] The computer device further includes a communication interface 30 for communication with other devices or communication networks.

[0301] The embodiments of the present application further provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or be implemented as computer code originally stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded through a network and stored in a local storage medium, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0302] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be invoked or provided. Those skilled in the art should understand that the form of computer program instructions in a computer readable medium includes but is not limited to source files, executable files, installation package files, etc. Correspondingly, the way of executing computer program instructions by a computer includes but is not limited to: the computer directly executes the instructions, or the computer executes the corresponding compiled program after compiling the instructions, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0303] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes should be covered within the scope of protection of the present application.

Claims

1. A data processing method, characterized by, The method comprises: acquiring a target two-dimensional image to be processed; determining a two-dimensional detection frame of an observed object in the target two-dimensional image and a first relative depth of the observed object according to the target two-dimensional image; determining a plurality of second relative depths of the observed object in different dimensions according to the two-dimensional detection frame; determining a relative depth of the observed object according to the first relative depth and the plurality of second relative depths; generating three-dimensional motion information corresponding to the observed object according to the relative depth and the two-dimensional detection frame; the three-dimensional motion information represents a motion state of the observed object in a three-dimensional space.

2. The method of claim 1, wherein, The determining of the plurality of second relative depths of the observed object in different dimensions according to the two-dimensional detection frame comprises: determining spatial parameters of the two-dimensional detection frame in different dimensions; determining the plurality of second relative depths of the observed object in different dimensions according to the spatial parameters of the two-dimensional detection frame.

3. The method of claim 2, wherein, The plurality of second relative depths of the observed object in different dimensions comprises at least two of an area relative depth of the observed object in an area dimension, a height relative depth of the observed object in a height dimension, and a projection relative depth of the observed object in a projection dimension; and the spatial parameters comprise at least two of an area of the two-dimensional detection frame, a height of the two-dimensional detection frame, and a bottom position coordinate of the two-dimensional detection frame. The area relative depth of the observed object is determined according to the area of the two-dimensional detection frame; the height relative depth of the observed object is determined according to the height of the two-dimensional detection frame; and the projection relative depth of the observed object is determined by inversely perspective projecting the two-dimensional detection frame using the bottom position coordinate of the two-dimensional detection frame.

4. The method of claim 1, wherein, The determining of the two-dimensional detection frame of the observed object in the target two-dimensional image and the first relative depth of the observed object according to the target two-dimensional image comprises: performing object detection on the target two-dimensional image according to a preset object detection model to identify the two-dimensional detection frame of the observed object in the target two-dimensional image; inputting the target two-dimensional image into a preset perception network to obtain the first relative depth of the observed object in the target two-dimensional image output by the perception network.

5. The method of claim 1, wherein, After the generation of the three-dimensional motion information corresponding to the observed object, the method further comprises: acquiring an observed object queue corresponding to the observed object; wherein the observed object queue comprises three-dimensional motion information of a historical observed object, the historical observed object in the observed object queue and the observed object are associated observed objects, and the three-dimensional motion information of the historical observed object is determined according to a historical two-dimensional image, and the acquisition time of the historical two-dimensional image is earlier than the acquisition time of the target two-dimensional image; performing fitting according to the three-dimensional motion information of the observed object, the three-dimensional motion information of the historical observed object in the observed object queue, the acquisition time of the historical two-dimensional image, and the acquisition time of the target two-dimensional image to obtain a fitting result between the three-dimensional motion information and the acquisition time; and According to the fitting result, the three-dimensional motion information of the observation object is corrected, the corrected motion information corresponding to the observation object is generated, and the corrected motion information corresponding to the observation object is taken as new three-dimensional motion information corresponding to the observation object.

6. The method of claim 5, wherein, The observation object is provided with a corresponding identifier; the observation object queue corresponding to the observation object is obtained by: Judging whether the identifier of the observation object is the same as the identifier of the historical observation object in the object queue recorded in advance; In the case that the identifier of the observation object is the same as the identifier of the historical observation object in the object queue, the object queue to which the historical observation object with the same identifier as the observation object belongs is taken as the observation object queue; the historical observation object in the observation object queue and the observation object correspond to the same object.

7. The method according to claim 5 or 6, characterized in that, The target two-dimensional image is used to represent the vehicle condition around the target mobile device; the observation object queue corresponding to the observation object includes: Judging whether the observation object is a preset type object; wherein the preset type object is an object whose influence on the driving safety of the target mobile device meets a preset condition; In the case that the observation object is a preset type object, the intersection and union association between the observation object and the historical observation object in the object queue is performed on the object queue recorded in advance, to obtain the intersection and union association result between the historical observation object in the object queue and the observation object; According to the intersection and union association result between the historical observation object in all object queues and the observation object, the observation object queue is determined from the object queue.

8. The method of claim 7, wherein, The observation object queue corresponding to the observation object further includes: If the observation object is not a preset type object, or the intersection and union association result indicates that each object queue is not an observation object queue, the nearest neighbor association between the observation object and the historical observation object in each object queue is performed, to obtain the nearest neighbor association result between the historical observation object in the object queue and the observation object; According to the nearest neighbor association result between the historical observation object in all object queues and the observation object, the observation object queue is determined from the object queue.

9. The method of claim 5, wherein, Further comprising: In the case that there is no observation object queue, a new first object queue is created for the observation object; And / or, If there is a second object queue, the unassociated number of the second object queue is increased by one; the historical observation object in the second object queue is not associated with any observation object, and the unassociated number of the second object queue is the number of objects that the second object queue has been continuously unassociated with; in the case that the unassociated number of the second object queue exceeds a first preset number, the second object queue is deleted.

10. The method of claim 5, wherein, The fitting according to the three-dimensional motion information of the observation object, the three-dimensional motion information of the historical observation object in the observation object queue, the collection time of the historical two-dimensional image, and the collection time of the target two-dimensional image includes: In a case where the number of objects in the observation object queue exceeds a first preset threshold, fitting is performed according to the three-dimensional motion information of the observation object and the latest k historical observation objects in the observation object queue and corresponding collection times; k≥3, the latest k historical observation objects are the last k historical observation objects determined according to the collection times of the historical two-dimensional images corresponding to the historical observation objects in the observation object queue from early to late.

11. The method of claim 5, wherein, The method further comprises: inputting the target two-dimensional image into a preset perception network to obtain a speed of an observation object in the target two-dimensional image output by the perception network; for each observation object, if the speed of the observation object is less than a preset value, and the number of historical observation objects associated with the observation object and having a speed less than a preset value within a preset time period adjacent to the collection time of the target two-dimensional image exceeds a second preset number of times, the speed in the corrected motion information is set to zero.

12. The method of claim 1 or 5, wherein, The target two-dimensional image is used to represent the vehicle situation around the target mobile device; the method further comprises: determining path planning information of the target mobile device according to the three-dimensional motion information corresponding to the observation object, so that the target mobile device travels according to the path planning information.

13. The method of claim 12, wherein, The determination of the path planning information of the target mobile device according to the three-dimensional motion information corresponding to the observation object comprises: determining whether the observation object is a key observation object; in a case where the observation object is a key observation object, determining the path planning information of the target mobile device according to the three-dimensional motion information corresponding to the observation object.

14. The method of claim 13, wherein, The determination of whether the observation object is a key observation object comprises: determining whether the observation object is a key observation object according to at least one of the number of objects in the observation object queue corresponding to the observation object, a fitting index representing fitting effect, and a collision time predicted according to the three-dimensional motion information corresponding to the observation object; the collision time is the time when the target mobile device collides with the observation object; wherein, in a case where at least one of the number of objects in the observation object queue is greater than a preset number, the fitting effect represented by the fitting index is greater than a preset fitting effect, and the collision time is less than a preset collision time, the observation object is determined to be a key observation object.

15. The method of claim 1, wherein, The determination of the relative depth of the observation object according to the first relative depth and the plurality of second relative depths comprises: fusing the first relative depth and the plurality of second relative depths in a weighted manner to determine the relative depth of the observation object; the first relative depth and the plurality of second relative depths each have a corresponding weight.

16. The method of claim 15, wherein, The fusion of the first relative depth and the plurality of second relative depths in a weighted manner comprises: weighting the first relative depth and the plurality of second relative depths according to a first preset weight corresponding to the first relative depth and a second preset weight corresponding to the second relative depth. Alternatively, the first weight corresponding to the first relative depth and the second weight corresponding to the second relative depth are determined by using a weighted least square method, and the first relative depth and the second relative depths are weighted according to the first weight corresponding to the first relative depth and the second weight corresponding to the second relative depth.

17. The method of claim 1, wherein, The two-dimensional detection frame is a bounding box for positioning and identifying an observed object in the target two-dimensional image, and the bounding box includes one of a rectangular box, a polygonal box, a circular or elliptical box, a rotated rectangular box, and a self-defined shape box.

18. A data processing apparatus, characterized by The device comprises: an acquisition module configured to acquire a target two-dimensional image to be processed; a depth determination module configured to determine, according to the target two-dimensional image, a two-dimensional detection frame of an observed object in the target two-dimensional image and a first relative depth of the observed object, determine, according to the two-dimensional detection frame, a plurality of second relative depths of the observed object in different dimensions, and determine a relative depth of the observed object according to the first relative depth and the plurality of second relative depths; a processing module configured to generate three-dimensional motion information corresponding to the observed object according to the relative depth and the two-dimensional detection frame, wherein the three-dimensional motion information represents a motion state of the observed object in a three-dimensional space.

19. A computer device, comprising: comprise: a memory and a processor, which are communicatively connected, and the memory stores computer instructions, and the processor executes the computer instructions to perform the data processing method in any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make the computer execute the data processing method in any one of claims 1 to 17.

Images