A target detection method, apparatus, storage medium, and electronic device

Abstract

This specification discloses a target detection method, apparatus, storage medium, and electronic device. The method includes: acquiring image feature data sent by a second agent in a driving environment at historical times; inputting the image feature data into a pre-trained trajectory prediction model to determine the trajectory data of a target object; wherein the trajectory data includes the target object's position and direction information in the driving environment at historical and current times; determining first image feature data corresponding to the target object at the current time based on the trajectory data; and performing target detection based on the first image feature data and image feature data corresponding to an image containing the target object collected by the first agent at the current time. This solution effectively avoids the adverse effects of transmission delay during data transmission between different agents on target detection, ensuring the accuracy of target detection.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a target detection method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the rapid development of vehicle-to-everything (V2X) technology, cooperative perception systems have been widely applied in the field of intelligent transportation. Cooperative perception systems improve the safety and efficiency of transportation systems by sharing and fusing sensor data (such as LiDAR point clouds and camera images) from multiple intelligent agents (such as vehicles and roadside units) in real time to construct a global environmental perception model and perform target recognition.

[0003] However, in the process of collaborative perception, the existence of network connection inevitably introduces time delays, such as network communication delays, data processing delays, and clock synchronization errors. These time delays cause the information currently received by the intelligent agent in the driving environment to actually be the lagging information collected by other intelligent agents earlier. The information of different intelligent agents cannot be aligned in time and space, which seriously affects the accuracy of target recognition results.

[0004] Therefore, how to avoid the impact of time delay in the collaborative perception process on target detection and ensure the accuracy of target detection is an urgent problem to be solved. Summary of the Invention

[0005] This specification provides a target detection method, apparatus, storage medium, and electronic device to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] A target detection method, applied to a first intelligent agent in a driving environment, the method comprising:

[0008] Acquire historical image feature data sent by a second intelligent agent in the driving environment; wherein the image feature data contains a time series of image feature data of the target object;

[0009] The image feature data is input into a pre-trained trajectory prediction model to determine the trajectory data of the target object; wherein, the trajectory data contains the position and orientation information of the target object in the driving environment;

[0010] Based on the trajectory data, determine the first image feature data corresponding to the target object at the current moment;

[0011] Target detection is performed based on the first image feature data and the second image feature data corresponding to the target object in the image collected by the first agent at the current moment.

[0012] Optionally, based on the trajectory data, determining the first image feature data corresponding to the target object at the current moment specifically includes:

[0013] Based on the trajectory data, determine the feature points on the motion trajectory of the target object;

[0014] The feature point data at each feature point is input into a preset trajectory perception model to determine the first image feature data through the trajectory perception model.

[0015] Optionally, based on the trajectory data, the feature points on the motion trajectory of the target object are determined, specifically including:

[0016] The trajectory data is input into a pre-trained offset generation model, which determines the feature points at each attention position based on the trajectory distribution and temporal sequence of the target object.

[0017] Optionally, the feature point data at each feature point is input into a preset trajectory perception model to determine the first image feature data through the trajectory perception model, specifically including:

[0018] Based on the correlation between the image feature data of the target object at the current moment and the data of each feature point, the weight between the image feature data of the target object at the current moment and the data of each feature point is determined by the trajectory perception model.

[0019] Based on the weights between the image feature data of the target object at the current moment and the data of each feature point, the first image feature data is obtained by weighted summation of the data of each feature point.

[0020] Optionally, training the trajectory prediction model specifically includes:

[0021] Obtain a historical image sequence containing the target object;

[0022] For each historical image in the historical image sequence, determine the spatiotemporal data of the target object under that historical image, and determine the tag trajectory data based on the spatiotemporal data of the target object under each historical image;

[0023] The feature data corresponding to the historical image sequence is input into the trajectory prediction model to be trained, so as to determine the predicted trajectory data corresponding to the target object through the trajectory prediction model;

[0024] Based on the deviation between the predicted trajectory data and the labeled trajectory data, the loss value of the trajectory prediction model is determined, and the trajectory prediction model is trained based on the loss value.

[0025] Optionally, training the offset generation model specifically includes:

[0026] Acquire the historical trajectory data of the target object and the label feature point data corresponding to each actual feature point in the historical trajectory data;

[0027] The historical trajectory data is input into the offset generation model to be trained, so as to generate the predicted feature point data corresponding to the historical trajectory data through the offset generation model.

[0028] Based on the labeled feature point data and the predicted feature point data, the loss value of the offset generation model is determined, and the offset generation model is trained based on the loss value.

[0029] Optionally, the loss value of the offset generation model is determined based on the label feature point data and the predicted feature point data, specifically including:

[0030] Determine the spatial distance between each feature point in the predicted feature point data and each feature point in the label feature point data, and determine the matching cost data between different feature points based on the spatial distance;

[0031] Based on the matching cost data, determine the matching rate between each feature point in the predicted feature point data and each feature point in the label feature point data;

[0032] The feature point matching data is determined with the goal of maximizing the matching rate between each feature point in the predicted feature point data and each feature point in the label feature point data;

[0033] The loss value is determined based on the matching cost data and the feature point matching data.

[0034] This specification provides a target detection device, including:

[0035] The acquisition module is used to acquire historical image feature data sent by the second intelligent agent in the driving environment; wherein the image feature data contains a time series of image feature data of the target object;

[0036] The input module is used to input the image feature data into a pre-trained trajectory prediction model to determine the trajectory data of the target object; wherein the trajectory data includes the position and orientation information of the target object in the driving environment;

[0037] The determining module is used to determine the first image feature data corresponding to the target object at the current time based on the trajectory data;

[0038] The detection module is used to perform target detection based on the first image feature data and the second image feature data corresponding to the target object in the image collected by the first agent at the current moment.

[0039] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described target detection method.

[0040] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target detection method described above.

[0041] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0042] The target detection method provided in this specification acquires image feature data sent by a second agent in the driving environment at historical moments; inputs the image feature data into a pre-trained trajectory prediction model to determine the trajectory data of the target object; wherein, the trajectory data contains the target object's position and direction information in the driving environment at historical and current moments; based on the trajectory data, determines the first image feature data corresponding to the target object at the current moment; and performs target detection based on the first image feature data and the image feature data corresponding to the image containing the target object collected by the first agent at the current moment. This scheme effectively avoids the adverse effects of transmission delay during data transmission between different agents on target detection, ensuring the accuracy of target detection.

[0043] As can be seen from the above method, after the first agent receives the historical image feature data sent by the second agent, it first uses a pre-trained trajectory prediction model to predict the spatiotemporal information containing the current and historical moments. Based on the predicted trajectory data, it determines the image feature data of the target object at the current moment. This avoids the lag between the image feature data sent by the second agent at the previous moment and the image feature data sent by the first agent at the current moment, thus accurately aligning multiple image feature data in time and space, effectively improving the accuracy of target detection. Attached Figure Description

[0044] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0045] Figure 1 This is a schematic diagram illustrating the image feature alignment for target detection provided in this specification;

[0046] Figure 2 This is a schematic diagram of a target detection method provided in this specification;

[0047] Figure 3 This is a schematic diagram of a visualized trajectory data provided in this specification;

[0048] Figure 4 This is a schematic diagram illustrating one method for determining image feature data provided in this specification;

[0049] Figure 5 This is a schematic diagram of a target detection process provided in this specification;

[0050] Figure 6 This is a schematic diagram of a target detection device provided in this specification;

[0051] Figure 7 This specification provides a corresponding Figure 2 A schematic diagram of an electronic device. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0053] In complex multi-agent cooperative systems, the dynamic characteristics of network connections inevitably introduce an error accumulation effect over time; this systematic delay is called latency. Its existence not only weakens the accuracy of real-time interaction but also poses a fundamental challenge to the effectiveness of collaborative decision-making. Specifically, latency can be attributed to the following three core dimensions:

[0054] Communication delay: In wireless transmission, signals must overcome the dual constraints of medium propagation delay and channel congestion. When multiple nodes communicate concurrently in a dense spectrum environment, the MAC layer contention backoff mechanism may cause queuing delays of hundreds of milliseconds. This, combined with routing jumps and traffic shaping strategies in the transmission path, ultimately results in end-to-end communication delay fluctuations ranging from 50 to 400 ms.

[0055] Computational processing latency: Sensing data undergoes dual processing steps before and after transmission. The front-end device needs to perform feature extraction (e.g., YOLOv5 on Jetson AGX requires approximately 50ms per frame), which involves layer-by-layer operations of the convolutional neural network and data compression. The receiving end needs to perform spatial coordinate system transformation and calibration (typically, the transformation between UTM and WGS84 coordinate systems requires 20ms of matrix operations). This computational load increases non-linearly with the data accuracy requirements and scene complexity.

[0056] Limitations in clock synchronization accuracy: Although GPS timing accuracy can reach the nanosecond level, in practical applications, ionospheric interference, urban canyon effects, and receiver crystal oscillator drift can cause clock deviations between nodes to accumulate to the order of 10ms. In high-speed mobile scenarios (such as vehicle-to-everything (V2X) at a relative speed of 120km / h), a 10ms time difference corresponds to a 33cm position error. This timing misalignment can severely disrupt the mathematical consistency of sensor fusion.

[0057] like Figure 1 As shown, due to the lag effect of the multidimensional time delay, it is impossible to align the same features of the target in time when performing target detection, which seriously affects the target detection results.

[0058] Figure 1 This is a schematic diagram illustrating the alignment of image features for target detection as provided in this specification.

[0059] Among them, spatial misalignment means that asynchronous observations cause features to be misaligned in spatial location, resulting in target localization deviation, while semantic misalignment means that differences in feature representations of the same target across agents lead to false detections and false negatives.

[0060] Based on this, this specification provides a target detection method that eliminates cross-agent feature space and semantic misalignment caused by communication delays, improves the motion compensation accuracy of multi-frame point cloud features in dynamic scenes, and achieves end-to-end low-latency robust collaborative perception.

[0061] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0062] Figure 2 This is a schematic diagram of a target detection method provided in this specification, including the following steps:

[0063] S201: Obtain historical image feature data sent by the second intelligent agent in the driving environment; wherein the image feature data contains a time series of image feature data of the target object.

[0064] In intelligent driving environments, intelligent agents typically integrate multiple sensors (such as radar, lidar, and cameras) to perceive the surrounding environment in real time, understand traffic rules, predict the behavior of other road users, and make driving decisions accordingly. These intelligent agents can include driving equipment (such as vehicles and aircraft) as well as roadside equipment (such as traffic lights, roadside units, and parking systems).

[0065] During target detection, each agent can acquire images of the target object through its own sensors, thereby obtaining image data. This image data can include images acquired by different sensors, such as point cloud images, camera images, infrared images, and depth images.

[0066] The target object can be an obstacle in the driving environment or other intelligent agents, and there can be one or more target objects.

[0067] These agents can encode the raw image data they collect to obtain image feature data, and then send the image feature data to the first agent that needs to perform target detection.

[0068] For example, an agent can use a deep learning model (such as PointPillars) to encode raw data into low-dimensional features (BEV feature maps) and transmit the features to the first agent via C-V2X or DSRC protocols.

[0069] The first intelligent agent can acquire image feature data sent by other intelligent agents (the second intelligent agent) in the driving environment at historical moments. This image feature data may include a time series of image feature data containing the target object sent by the second intelligent agent at several previous moments.

[0070] It should be noted that, due to the data transmission delay between the agents, the image feature data received by the first agent at any given moment is actually collected and sent by the second agent before that moment. Therefore, the image feature data obtained by the first agent is the data sent by the second agent at a historical moment.

[0071] S202: Input the image feature data into a pre-trained trajectory prediction model to determine the trajectory data of the target object; wherein, the trajectory data contains the position information and direction information of the target object in the driving environment.

[0072] After receiving image feature data, the first intelligent agent can input it into a pre-trained trajectory prediction model, thereby determining the trajectory data of the target object through the trajectory prediction model.

[0073] The trajectory data mentioned above includes not only the position and orientation information of the target object at several previous moments, but also the position and orientation information of the target object at the current moment predicted by the trajectory prediction model based on historical trajectory data. For ease of understanding, this specification provides a visual diagram of the trajectory data, such as... Figure 3 As shown,

[0074] The trajectory data mentioned above can be the trajectory field of the target object, with the position information and direction information corresponding to the position field (trajectory heat) and direction field (tangent direction of the trajectory), respectively.

[0075] Before using the trajectory prediction model described above, it can be trained on a server. Only after training is complete can it be deployed and used for trajectory prediction. The training process for the trajectory prediction model can include the following steps:

[0076] S301: Obtain a historical image sequence containing the target object;

[0077] S301: For each historical image in the historical image sequence, determine the spatiotemporal data of the target object under that historical image, and determine the tag trajectory data based on the spatiotemporal data of the target object under each historical image.

[0078] For each frame of historical image, the server can determine the bounding box of the target object in the historical image, and based on the annotation information of the bounding box (such as time, location, etc.), determine the spatiotemporal data of the target object in the historical image.

[0079] The server can extract bounding box annotation information for each object within a maximum time window. This time window starts from the first appearance of the target object and continues until the current moment. By connecting the center points of these bounding boxes, the motion trajectory of the target object, i.e., the label trajectory data, can be constructed.

[0080] In general, the generation process of labeled trajectory data can be divided into two steps: interpolating the trajectory and projecting the interpolation result onto the feature space. This yields the labeled trajectory data of the target object. It should be noted that when trajectories overlap, the trajectory with the more recent timestamp can overwrite the older trajectory.

[0081] S302: Input the feature data corresponding to the historical image sequence into the trajectory prediction model to be trained, so as to determine the predicted trajectory data corresponding to the target object through the trajectory prediction model;

[0082] S303: Determine the loss value of the trajectory prediction model based on the deviation between the predicted trajectory data and the label trajectory data, and train the trajectory prediction model based on the loss value.

[0083] In this specification, the loss value of the trajectory prediction model can be composed of two parts, namely the position field loss L. pos and directional field loss L ori , where L pos It can be determined based on the deviation between the location information corresponding to the predicted trajectory data and the label trajectory data, as expressed as:

[0084]

[0085] The location information of the predicted trajectory data is x. xyc The location information of the label trajectory data is

[0086] L ori The direction can be determined based on the deviation between the predicted trajectory data and the corresponding label trajectory data, as shown below:

[0087]

[0088] Among them, the direction information corresponding to the predicted trajectory data is y, and the direction information corresponding to the label trajectory data is... M is a mask matrix composed of 0s and 1s, with the same dimension as the trajectory field. A value of 1 indicates that there is a target trajectory at that location.

[0089] The server can minimize L ori and L pos To optimize the objective, the trajectory prediction model is trained until the training objective is met (such as reaching a preset number of iterations or converging within a preset range).

[0090] S203: Based on the trajectory data, determine the first image feature data corresponding to the target object at the current moment.

[0091] After determining the trajectory data of the target object, the first intelligent agent can determine the feature points on the target object's motion trajectory based on the trajectory data.

[0092] Specifically, the first agent can input trajectory data into a pre-trained offset generation model, which can then determine the feature points at each attention location based on the trajectory distribution of the target object and the temporal sequence.

[0093] After determining each feature point, the first intelligent agent can input the feature point data at each feature point into a preset trajectory perception model, so as to determine the first image feature data through the trajectory perception model.

[0094] Specifically, the first intelligent agent can use a trajectory perception model to determine the weight between the image feature data of the target object at the current moment and each feature point data based on the correlation between the image feature data of the target object at the current moment and each feature point data. Then, based on the weight between the image feature data of the target object at the current moment and each feature point data, the first intelligent agent performs a weighted summation of each feature point data to obtain the first image feature data.

[0095] For ease of understanding, this specification provides a schematic diagram illustrating a method for determining image feature data, such as... Figure 4 As shown.

[0096] In this process, the feature representation of the feature point data at each feature point consists of k and v. Based on q and k, the correlation between the image feature data of the target object at the current time and the data of each feature point can be determined, thereby obtaining the weight w. Then, the weight w and the v corresponding to each feature point data are weighted and summed to obtain the first image feature data.

[0097] Furthermore, before using the aforementioned offset generation model, it can be trained on a server. The training process may include the following steps:

[0098] S401: Obtain the historical trajectory data of the target object and the label feature point data corresponding to each actual feature point in the historical trajectory data.

[0099] The server can obtain historical trajectory data of the target object and label feature point data corresponding to each actual feature point in the historical trajectory data. These feature points can be feature points on the target object's trajectory as desired by the user. When determining label feature points, these label feature points need to meet the following conditions:

[0100] The feature point data at the feature point contains the target object;

[0101] The feature points are located on the trajectory of the target object;

[0102] The timestamp corresponding to a feature point at any given time must be earlier than the next feature point (i.e., the feature point is located in the historical position of the target object).

[0103] S402: Input the historical trajectory data into the offset generation model to be trained, so as to generate the predicted feature point data corresponding to the historical trajectory data through the offset generation model;

[0104] S403: Determine the loss value of the offset generation model based on the label feature point data and the predicted feature point data, and train the offset generation model based on the loss value.

[0105] The server can determine the loss value of the offset generation model based on the user's expected label feature point data and the predicted feature point data output by the offset generation model.

[0106] Specifically, the server can determine the spatial distance (e.g., L1 distance) between each feature point in the predicted feature point data and each feature point in the label feature point data, and based on this spatial distance, determine the matching cost data between different feature points. The cost matrix corresponding to this matching cost data can be represented as:

[0107] C jk =∑|p j -p k |1

[0108] Among them, C jk p represents the cost matrix for matching cost data. j p represents the position vector of the predicted feature point data j. k This represents the position vector of the label feature point data k. By traversing all predicted feature points and label feature points, the matching cost data can be obtained.

[0109] The server can then determine the matching rate P between each feature point in the predicted feature point data and each feature point in the label feature point data based on the matching cost data.

[0110] Furthermore, the server can determine the feature point matching data with the goal of maximizing the matching rate between each feature point in the predicted feature point data and each feature point in the label feature point data.

[0111] In practical applications, the server can use the sinkhorn algorithm to calculate the matching rate. The goal is to find a matrix such that the sum of each row of P is 1, and the sum of each column is also 1, satisfying the following:

[0112]

[0113] The server can then use the matching cost data C jk Matching data P with the feature points * The loss value is determined, and this loss value can be expressed as:

[0114]

[0115] S204: Target detection is performed based on the first image feature data and the second image feature data corresponding to the target object in the image collected by the first agent at the current moment.

[0116] Specifically, the first intelligent agent can use its own sensors to collect image data of the target object at the current moment, and determine the second image feature data corresponding to the target object based on the image data. Then, the first image feature data and the second image feature data of the target object are fused to obtain fused data, and then the target object is detected based on the fused data.

[0117] In this diagram, the first image feature data represents the feature data corresponding to the target object image collected by the second agent at the current moment, and the second image feature data represents the feature data corresponding to the target object image collected by the first agent at the current moment. Both are contemporaneous (i.e., both are at the current moment). For ease of understanding, this specification provides a schematic diagram of the target detection process, as shown below. Figure 5 As shown.

[0118] Here, Agent i represents the second intelligent agent in the driving environment, and Ego represents the first intelligent agent. The input historical image feature data (t-1) is processed by the trajectory prediction model, the offset prediction model, and the trajectory perception model to obtain the image feature data at the current time (time t). By fusing the first image feature data corresponding to the second intelligent agent at the current time and the second image feature data corresponding to the first intelligent agent at the current time, target detection can be performed based on the fused feature data.

[0119] It should be noted that for each agent, it can also process the image data acquired at the current time t based on the locally deployed trajectory prediction model, offset prediction model, and trajectory perception model to obtain image feature data at the future time t+1, and then send it to the first agent. Due to the data transmission delay, after the first agent receives the first image feature data sent by the second agent, the second image feature data corresponding to the image acquired by the first agent itself is exactly aligned with the first image feature data in time, that is, both are image feature data at time t+1.

[0120] In vehicle-to-vehicle (V2V) scenarios, the trajectory prediction model can be input to 4 frames of LiDAR data (0.4m rasterization) with a temporal embedding dimension of 64. The model architecture can adopt a 5-layer UNet with an output channel number of [32, 64, 128, 256, 512].

[0121] The number of attention heads in the trajectory awareness model can be set to 4, and the number of feature points can be set to 18.

[0122] In the Vehicle-to-Infrastructure (V2I) scenario, the Roadside Unit (RSU) can use 4 frames of point cloud data, while the vehicle side uses 2 frames. Timing compensation employs GPS / IMU-based ego-motion compensation. The deployment platform is NVIDIA Jetson AGX Xavier (inference latency ≤100ms).

[0123] As can be seen from the above methods, this scheme guides the alignment of high-dimensional features through low-dimensional trajectory prediction, avoiding the instability of directly predicting high-dimensional features. It distributes dynamic attention points along the trajectory, supports nonlinear motion compensation, and eliminates feature ambiguity based on a trajectory consistency-based attention mechanism. This effectively improves the accuracy of target detection.

[0124] In actual testing, this solution improved AP50 / 70 to 74.28% / 44.14% on the V2V4Real dataset (compared to +25.53% / 13.61% for single agent), effectively improving detection accuracy;

[0125] At a latency of 400ms, AP50 decreased by only 4.87% (compared to a 13.48% decrease in ERMVP), improving the robustness of object detection;

[0126] End-to-end inference speed reaches 25 FPS (NVIDIA RTX 3090), effectively improving computing efficiency.

[0127] The above describes one or more target detection methods implemented in this specification. Based on the same approach, this specification also provides corresponding target detection devices, such as... Figure 6 As shown.

[0128] Figure 6 A schematic diagram of a target detection device provided in this specification includes:

[0129] The acquisition module 601 is used to acquire historical image feature data sent by the second intelligent agent in the driving environment; wherein the image feature data includes a time series of image feature data of the target object;

[0130] The input module 602 is used to input the image feature data into a pre-trained trajectory prediction model to determine the trajectory data of the target object; wherein, the trajectory data includes the position information and orientation information of the target object in the driving environment;

[0131] The determining module 603 is used to determine the first image feature data corresponding to the target object at the current moment based on the trajectory data;

[0132] The detection module 604 is used to perform target detection based on the first image feature data and the second image feature data corresponding to the target object in the image collected by the first agent at the current moment.

[0133] Optionally, the determining module 603 is specifically used to determine each feature point on the motion trajectory of the target object based on the trajectory data; input the feature point data at each feature point into a preset trajectory perception model, so as to determine the first image feature data through the trajectory perception model.

[0134] Optionally, the determining module 603 is specifically used to input the trajectory data into a pre-trained offset generation model, so as to determine the feature points at each attention position through the offset generation model according to the trajectory distribution and temporal sequence of the target object.

[0135] Optionally, the determining module 603 is specifically used to: determine the weight between the image feature data of the target object at the current moment and each feature point data based on the correlation between the image feature data of the target object at the current moment and each feature point data through the trajectory perception model; and perform weighted summation on each feature point data according to the weight between the image feature data of the target object at the current moment and each feature point data to obtain the first image feature data.

[0136] Optionally, the device further includes:

[0137] The training module 605 is used to acquire a historical image sequence containing a target object; for each historical image in the historical image sequence, determine the spatiotemporal data of the target object under that historical image, and determine the label trajectory data based on the spatiotemporal data of the target object under each historical image; input the feature data corresponding to the historical image sequence into the trajectory prediction model to be trained, so as to determine the predicted trajectory data corresponding to the target object through the trajectory prediction model; determine the loss value of the trajectory prediction model based on the deviation between the predicted trajectory data and the label trajectory data, and train the trajectory prediction model based on the loss value.

[0138] Optionally, the training module 605 is specifically used to: acquire historical trajectory data of the target object and label feature point data corresponding to each actual feature point in the historical trajectory data; input the historical trajectory data into the offset generation model to be trained, so as to generate predicted feature point data corresponding to the historical trajectory data through the offset generation model; determine the loss value of the offset generation model according to the label feature point data and the predicted feature point data, and train the offset generation model according to the loss value.

[0139] Optionally, the training module 605 is specifically configured to: determine the spatial distance between each feature point in the predicted feature point data and each feature point in the label feature point data; and based on the spatial distance, determine matching cost data between different feature points; determine the matching rate between each feature point in the predicted feature point data and each feature point in the label feature point data according to the matching cost data; and determine feature point matching data with the goal of maximizing the matching rate between each feature point in the predicted feature point data and each feature point in the label feature point data.

[0140] The loss value is determined based on the matching cost data and the feature point matching data.

[0141] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 2 This provides a target detection method.

[0142] This instruction manual also provides Figure 7 One of the corresponding Figure 2 A schematic diagram of the structure of an electronic device. (e.g.) Figure 7 At the hardware level, the electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 3 The target detection method described herein. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0143] Improvements in a technology can be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology can now be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement in methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used when writing program development code. The original code before compilation must also be written in a specific programming language, which is called a Hardware Description Language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog.

[0144] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0145] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0146] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0147] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] In a typical configuration, the processing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0152] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0153] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by processing devices. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0154] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0155] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0156] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0157] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0158] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A target detection method, characterized in that, The method is applied to a first intelligent agent in a driving environment, and the method includes: Acquire historical image feature data sent by a second intelligent agent in the driving environment; wherein the image feature data contains a time series of image feature data of the target object; The image feature data is input into a pre-trained trajectory prediction model to determine the trajectory data of the target object; wherein, the trajectory data contains the position and orientation information of the target object in the driving environment; Based on the trajectory data, determine the first image feature data corresponding to the target object at the current moment; Target detection is performed based on the first image feature data and the second image feature data corresponding to the target object in the image collected by the first agent at the current moment.

2. The method as described in claim 1, characterized in that, Based on the trajectory data, the first image feature data corresponding to the target object at the current moment is determined, specifically including: Based on the trajectory data, determine the feature points on the motion trajectory of the target object; The feature point data at each feature point is input into a preset trajectory perception model to determine the first image feature data through the trajectory perception model.

3. The method as described in claim 2, characterized in that, Based on the trajectory data, the feature points on the motion trajectory of the target object are determined, specifically including: The trajectory data is input into a pre-trained offset generation model, which determines the feature points at each attention position based on the trajectory distribution and temporal sequence of the target object.

4. The method as described in claim 2, characterized in that, The feature point data at each feature point is input into a preset trajectory perception model to determine the first image feature data through the trajectory perception model, specifically including: Based on the correlation between the image feature data of the target object at the current moment and the data of each feature point, the weight between the image feature data of the target object at the current moment and the data of each feature point is determined by the trajectory perception model. Based on the weights between the image feature data of the target object at the current moment and the data of each feature point, the first image feature data is obtained by weighted summation of the data of each feature point.

5. The method as described in claim 1, characterized in that, Training the trajectory prediction model specifically includes: Obtain a historical image sequence containing the target object; For each historical image in the historical image sequence, determine the spatiotemporal data of the target object under that historical image, and determine the tag trajectory data based on the spatiotemporal data of the target object under each historical image; The feature data corresponding to the historical image sequence is input into the trajectory prediction model to be trained, so as to determine the predicted trajectory data corresponding to the target object through the trajectory prediction model; Based on the deviation between the predicted trajectory data and the labeled trajectory data, the loss value of the trajectory prediction model is determined, and the trajectory prediction model is trained based on the loss value.

6. The method as described in claim 3, characterized in that, Training the offset generation model specifically includes: Acquire the historical trajectory data of the target object and the label feature point data corresponding to each actual feature point in the historical trajectory data; The historical trajectory data is input into the offset generation model to be trained, so as to generate the predicted feature point data corresponding to the historical trajectory data through the offset generation model. Based on the labeled feature point data and the predicted feature point data, the loss value of the offset generation model is determined, and the offset generation model is trained based on the loss value.

7. The method as described in claim 6, characterized in that, Based on the labeled feature point data and the predicted feature point data, the loss value of the offset generation model is determined, specifically including: Determine the spatial distance between each feature point in the predicted feature point data and each feature point in the label feature point data, and determine the matching cost data between different feature points based on the spatial distance; Based on the matching cost data, determine the matching rate between each feature point in the predicted feature point data and each feature point in the label feature point data; The feature point matching data is determined with the goal of maximizing the matching rate between each feature point in the predicted feature point data and each feature point in the label feature point data; The loss value is determined based on the matching cost data and the feature point matching data.

8. A target detection device, characterized in that, include: The acquisition module is used to acquire historical image feature data sent by a second intelligent agent in the driving environment; wherein the image feature data contains a time series of image feature data of the target object; The input module is used to input the image feature data into a pre-trained trajectory prediction model to determine the trajectory data of the target object; wherein the trajectory data includes the position and orientation information of the target object in the driving environment; The determining module is used to determine the first image feature data corresponding to the target object at the current time based on the trajectory data; The detection module is used to perform target detection based on the first image feature data and the second image feature data corresponding to the target object in the image collected by the first agent at the current moment.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 7.

Images