Target detection method and related equipment

Abstract

The invention provides a target detection method and related equipment, and the method comprises the steps: constructing a first three-dimensional image based on image data collected by a sensor in a vehicle at a t moment, and predicting a target three-dimensional image at a t + 1 moment according to the first three-dimensional image and the motion sequence data of the vehicle; constructing a second three-dimensional image according to the environment image collected by the sensor in the vehicle at the t + 1 moment; according to a comparison result between the target three-dimensional image and the second three-dimensional image, a subspace body is determined, and the subspace body comprises points with different comparison results in the second three-dimensional image; and generating text description information of an unknown object corresponding to the subspace body, wherein the text description information is used for target detection. According to the invention, the driving safety of the vehicle is improved.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a target detection method and related equipment. Background Technology

[0002] With the development of vehicle technology, intelligent functions such as driver assistance and fully autonomous driving have been gradually applied to vehicles. The core of driver assistance and fully autonomous driving functions is environmental perception capability, which aims to accurately identify, locate, and predict the state of traffic participants and objects in the vehicle's surrounding environment.

[0003] In the exemplary technology, a deep learning-based object detection model is used to detect objects. However, this model is trained on predefined, limited-class training data. Real-world traffic environments are highly open, dynamic, and uncertain. Unknown objects frequently appear in traffic environments that are not covered by the training data or are outside its distribution, such as atypical engineering vehicles, animals, scattered cargo, unusual obstacles, and humans behaving abnormally. The object detection model cannot correctly classify these unknown objects not included in the training data, preventing vehicles from making correct decisions based on correctly classified objects, thus reducing driving safety. Summary of the Invention

[0004] Based on the aforementioned technological status, this application provides a target detection method and related equipment to address the problem of low vehicle driving safety.

[0005] To achieve the above-mentioned technical objectives, this application proposes the following technical solution: Firstly, this application provides a target detection method, including: A first three-dimensional image is constructed based on image data collected by sensors in the vehicle at time t. Based on the first three-dimensional image and the vehicle's action sequence data, a target three-dimensional image at time t+1 is predicted. The action sequence data includes each action to be performed by the vehicle between time t and time t+1. A second three-dimensional image is constructed based on the image data collected by the sensors in the vehicle at time t+1. Based on the comparison results between the target 3D image and the second 3D image, a subspace volume is determined, wherein the subspace volume includes points in the second 3D image whose comparison results are different; Generate textual description information for the unknown objects corresponding to the subspace volume, and the textual description information is used for target detection.

[0006] In some implementations, predicting the target 3D image at time t+1 based on the first 3D image and the vehicle's motion sequence data includes: The first three-dimensional image is segmented into multiple first spatial volumes, and based on the action sequence data and each first spatial volume, the target spatial volume corresponding to each first spatial volume at time t+1 is predicted. The step of determining the subspace volume based on the comparison result between the target 3D image and the second 3D image includes: The second three-dimensional image is segmented into multiple second spatial volumes; Based on the comparison result between the target space volume and the corresponding second space volume, a subspace volume is determined in the second space volume, and the spatial coordinates of the target space volume are the same as the spatial coordinates of the second space volume corresponding to the target space volume.

[0007] In some embodiments, determining the subspace volume based on the comparison result between the target 3D image and the second 3D image includes: The first point in the second three-dimensional image is compared with the second point corresponding to the first point in the target three-dimensional image to obtain the divergence value of the first point. The divergence value is used to indicate whether the first point is the same as the corresponding second point. Based on the divergence value, each first point in the second three-dimensional image is clustered to obtain the subspace volume.

[0008] In some implementations, clustering each first point in the second three-dimensional image based on the divergence value includes: The continuous score of the first point is determined based on the divergence value of the first point and the divergence values ​​of the points adjacent to the first point in the second three-dimensional image. Each adjacent first point whose consecutive scores are greater than a preset threshold is clustered into the subspace volume.

[0009] In some implementations, generating textual description information of the unknown object corresponding to the subspace volume includes: The anomaly score of the subspace volume is determined based on the divergence value of each first point in the subspace volume; The subspace volume is transformed into a two-dimensional image, and the image data corresponding to the two-dimensional image at time t+1 and the anomaly score are input into the text generation model to obtain the text description information output by the text generation model.

[0010] In some implementations, predicting the target 3D image at time t+1 based on the first 3D image and the vehicle's motion sequence data includes: Historical time-series data is constructed based on the first three-dimensional image and multiple historical three-dimensional images. The historical three-dimensional images are three-dimensional images constructed from image data collected at the historical time, and the historical time is earlier than time t. The historical time series data and the action sequence data are input into the prediction model to obtain the target three-dimensional image at time t+1 output by the prediction model.

[0011] In some implementations, the target detection system includes a first model for predicting the 3D image at time t+1. After generating textual description information of the unknown object corresponding to the subspace volume, the system further includes: Based on the image data of the subspace at time t+1, the text description information, and the scene metadata of the subspace, a training sample is constructed and stored in a preset storage area. The first model is trained based on the training samples stored in the preset storage area.

[0012] In some implementations, the target detection system sets up a second model, and the training of the first model based on training samples stored in the preset storage area includes: The second model is trained based on the training samples stored in the preset storage area, and the first model is guided to learn based on the trained second model.

[0013] In some implementations, training the second model based on the training samples stored in the preset storage area includes: Update the loss functions of the second model and the first model based on the first loss function of the second model during the pre-training phase; The second model, which updates the loss function, is trained based on the training samples stored in the preset storage area.

[0014] Secondly, this application provides a target detection system, comprising: The first construction module is used to construct a first three-dimensional image based on image data collected by sensors in the vehicle at time t, and to predict a target three-dimensional image at time t+1 based on the first three-dimensional image and the vehicle's action sequence data. The action sequence data includes each action to be performed by the vehicle between time t and time t+1. The second construction module is used to construct a second three-dimensional image based on the image data collected by the sensor in the vehicle at time t+1. The determining module is used to determine a subspace volume based on the comparison result between the target three-dimensional image and the second three-dimensional image, wherein the subspace volume includes points in the second three-dimensional image whose comparison results are different; The generation module is used to generate text description information of unknown objects corresponding to the subspace volume, and the text description information is used for target detection.

[0015] Thirdly, this application provides a target detection system, including a memory and a processor, wherein, The memory is connected to the processor and is used to store programs; The processor is used to implement the target detection method as described in the first aspect or any implementation thereof by running a program in the memory.

[0016] Fourthly, this application provides a vehicle, characterized in that the vehicle includes a target detection system, the target detection system implementing the target detection method as described in the first aspect or any implementation thereof.

[0017] Fifthly, this application provides a computer program product, which, when executed by a processor, implements the target detection method as described in the first aspect or any implementation thereof.

[0018] In a sixth aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the target detection method as described in the first aspect or any implementation thereof.

[0019] This application provides a target detection method and related equipment. It constructs a first 3D image at time t and a second 3D image at time t+1. Based on the first 3D image, it predicts the target 3D image at time t+1. By comparing the target 3D image with the second 3D image, it obtains a subspace volume composed of points in the two 3D images that differ in the comparison results. Then, it generates textual description information for the unknown object corresponding to the subspace volume, enabling target detection based on the textual description information. In this application, the target 3D image at the current time is predicted from the previous 3D image and compared with the currently perceived 3D image. Significant differences (subspace volumes) caused by physical laws or dynamic behavior are identified as unknown objects. Textual descriptions of these unknown objects are then generated, allowing vehicles to accurately detect targets based on the textual descriptions and make correct decisions based on the detection results, thus improving vehicle driving safety. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 A flowchart of a target detection method provided in this application embodiment Figure 1 .

[0022] Figure 2 A flowchart of a target detection method provided in this application embodiment Figure 2 .

[0023] Figure 3 A flowchart of a target detection method provided in this application embodiment Figure 3 .

[0024] Figure 4 A flowchart of a target detection method provided in this application embodiment Figure 4 .

[0025] Figure 5 A flowchart of a target detection method provided in this application embodiment Figure 5 .

[0026] Figure 6 A schematic diagram of the target detection system is provided for the embodiments of this application.

[0027] Figure 7 This is a schematic diagram of the functional modules of a target detection system provided in an embodiment of this application.

[0028] Figure 8 This is a structural schematic diagram of a vehicle provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] It should be noted that the user information (including but not limited to electrical equipment information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0031] With the development of vehicle technology, intelligent functions such as driver assistance and fully autonomous driving have been gradually applied to vehicles. The core of driver assistance and fully autonomous driving functions is environmental perception capability, which aims to accurately identify, locate, and predict the state of traffic participants and objects in the vehicle's surrounding environment.

[0032] In the exemplary technology, a deep learning-based object detection model is used to detect objects. However, this model is trained on predefined, limited-class training data. Real-world traffic environments are highly open, dynamic, and uncertain. Unknown objects frequently appear in traffic environments that are not covered by the training data or are outside its distribution, such as atypical engineering vehicles, animals, scattered cargo, unusual obstacles, and humans behaving abnormally. The object detection model cannot correctly classify these unknown objects not included in the training data, preventing vehicles from making correct decisions based on correctly classified objects, thus reducing driving safety.

[0033] In view of this, embodiments of this application aim to provide a target detection method and related equipment, which constructs a first three-dimensional image at time t and a second three-dimensional image at time t+1, predicts a target spatial volume at time t+1 based on a first spatial volume segmented from the first three-dimensional image, compares the target spatial volume with the second spatial volume segmented from the second three-dimensional image to obtain a sub-space volume composed of points where the target spatial volume and the second spatial volume differ, and then generates textual description information of the object corresponding to the sub-space volume for target detection based on the textual description information. In this application, the spatial volume predicted at the previous time is compared with the actually perceived spatial volume at the current time to identify significant differences (sub-space volumes) caused by physical laws or behavioral dynamics as unknown objects, thereby generating textual descriptions of unknown objects. This allows vehicles to accurately detect targets based on textual descriptions, make correct decisions based on the detection results, and improve vehicle driving safety.

[0034] In addition, the exemplary technology also has the following problems: The lack of spatiotemporal context information and the limitations of dynamic perception: Existing open set detection methods are mostly limited to static feature analysis based on single-frame images, lacking effective modeling of dynamic temporal information. This makes it difficult for the system to effectively distinguish between "unknown objects with novel appearances" and "known objects with abnormal behavior", which can easily lead to misjudgment and missed detection of key targets in complex dynamic interactive scenarios. For unknown objects, the object detection model outputs the label "Unknown," creating a semantic vacuum. This label lacks semantic guidance for downstream decision-making modules in the vehicle, preventing the vehicle from assessing risks and developing refined avoidance strategies, thus severely limiting the intelligence and safety of autonomous driving in vehicles. Perception model knowledge solidification and decay: Traditional models have a fixed knowledge base after vehicle deployment, and cannot learn and evolve from new experiences in the loop. This makes the system unable to adapt to new things in the open world, and the knowledge decays relatively. It makes repeated mistakes when facing similar new objects and lacks adaptive ability.

[0035] To address the aforementioned technical problems, this application proposes a target detection method, as detailed in the following embodiments.

[0036] Figure 1 A flowchart of a target detection method provided in this application embodiment Figure 1 .like Figure 1 As shown, the target detection method provided in this embodiment includes: Step S101: Construct a first three-dimensional image based on the image data collected by the sensors in the vehicle at time t, and predict the target three-dimensional image at time t+1 based on the first three-dimensional image and the vehicle's action sequence data. The action sequence data includes each action to be performed by the vehicle between time t and time t+1.

[0037] In this embodiment, the executing entity is a target detection system. This system can be any system with target detection capabilities, or it can be a system containing a terminal device with target detection functionality. For ease of description, the term "system" will be used to refer to the target detection system below. The system is located within the vehicle and serves as a component for the vehicle to identify traffic participants and objects within the traffic environment.

[0038] During operation, the vehicle collects traffic environment data using its own sensors, which can be heterogeneous sensors such as multiple cameras and LiDAR. The system acquires image data collected by the sensors at time t and constructs a three-dimensional image at time t using the image data; this three-dimensional image is defined as the first three-dimensional image.

[0039] For example, the system includes a multi-modal bird's eye view (BEV) representation and discretization component, which converts image data collected by multiple cameras and point cloud data collected by LiDAR (Light Laser Detection and Ranging) into a unified multi-modal BEV feature map.

[0040] Specifically, the system first extracts 2D features from 2D images captured by multiple cameras using an image backbone network, while the 3D point cloud acquired by LiDAR is processed by the system's 3D editor to generate an initial LiDAR BEV feature map. The system includes an attention-based view transformation module. This module uses known intrinsic and extrinsic parameters of the cameras (among the multiple cameras) to back-project query points in the BEV space onto the feature maps of all 2D images for sampling. The sampled features are then aggregated to generate a purely visual BEV feature map. The system then concatenates the purely visual BEV feature map with the LiDAR BEV feature map along the channel dimension and performs deep fusion through the system's data convolutional layer to obtain a unified multimodal BEV feature map. The multimodal BEV feature map is represented as follows: ,in The fused BEV feature map Represents the set of real numbers. , , These are its height, width, and number of channels, respectively.

[0041] Understandingly, a multimodal BEV feature map is as follows: data from multiple sensors on the vehicle, such as cameras and radar, is processed through complex AI calculations, translated, and projected onto a two-dimensional map viewed from directly above. Each point on this map contains rich information about the road surface, vehicles, pedestrians, and other objects at that location. A multimodal BEV feature map can be viewed as a two-dimensional map or a top-down view in three-dimensional space; therefore, it can be considered a three-dimensional image, defined as the first three-dimensional image.

[0042] The system includes a prediction model trained on vehicle motion patterns. This model can generate highly relevant dynamic priors by incorporating the vehicle's future intentions. The first 3D image and the vehicle's action sequence data are used as input to the pre-defined model. Based on the scene's motion patterns and dynamic temporal sequence, the prediction model predicts the 3D image of the first 3D image at time t+1 as the target 3D image. The action sequence data includes all planned actions to be performed by the vehicle between time t and t+1, such as lane changing or braking actions planned during this time period.

[0043] Step S102: Construct a second three-dimensional image based on the image data collected by the sensor in the vehicle at time t+1.

[0044] After the target 3D image is predicted, time t+1 arrives. At this time, the vehicle's sensors collect image data from time t+1, and a second 3D image is constructed using this image data. The construction process of the second 3D image is the same as that of the first 3D image, as explained above, and will not be repeated here.

[0045] Step S103: Based on the comparison results between the target 3D image and the second 3D image, determine the subspace volume, which includes points in the second 3D image where the comparison results are different.

[0046] After obtaining the second 3D image, the target 3D image is compared with the second 3D image. For example, the second and target 3D images can be considered as a 3D spatial volume composed of countless smaller spatial volumes, defined as points. Both the second and target 3D images have the same number of points. The system compares a first point in the second 3D image with its corresponding second point in the target 3D image. The spatial coordinates of the first point and its corresponding second point are the same. The comparison focuses on the image parameters of the points, such as grayscale values ​​and pixel values. If the image parameters of the first point and its corresponding second point are different, then the first point and its corresponding second point are different. In this way, each first point in the second 3D image contains a comparison result. The system uses first points with different comparison results as target points and aggregates adjacent target points to obtain a sub-space volume. It can be understood that the first point in the sub-space volume is different from its corresponding second point in the target 3D image; that is, the sub-space volume is composed of points in the second 3D image with different comparison results. There can be multiple sub-space volumes.

[0047] Step S104: Generate text description information of the objects corresponding to the subspace volume. The text description information is used for target detection.

[0048] The entity corresponding to the subspace volume is actually an unknown object identified by the system, and the system needs to generate text description information corresponding to the unknown object.

[0049] For example, the system is equipped with a pre-trained VLM (Visual Language Model) with text generation capabilities. The VLM consists of three main components: 1. A frozen visual encoder is used to extract features from input images or other visual modal data and generate high-dimensional semantic embeddings. This component is based on a visual Transformer structure with a global receptive field to ensure the ability to model complex scenes and fine-grained targets. 2. A frozen language decoder is used to generate natural language descriptions based on visual embeddings. This component uses an autoregressive generation mechanism to model the conditional distribution of text sequences, thereby achieving continuous semantic expression. 3. Lightweight connection module, used to establish a mapping relationship between visual feature space and language representation space, to achieve cross-modal feature alignment and joint semantic reasoning. This module achieves feature distribution matching through a trainable projection network, so that visual input can be effectively interpreted in the semantic space of the language model.

[0050] In the target 3D image, a subspace volume can be represented by a bounding box. The VLM backprojects the subspace volume represented by the bounding box back into the original sensor space, that is, backprojects the 3D subspace volume into a 2D image. Based on this 2D image, the image data of the unknown object is accurately cropped from the image data at time t+1. The cropped image data is multimodal data, such as image patches or subsets of point clouds. It is understandable that the system uses open cueing engineering to determine the subspace volume, rather than traditional classification queries.

[0051] After obtaining multimodal data of the unknown object, the multimodal data is input into the VLM (Vehicle Modeling Library). The VLM generates a natural language description of the unknown object through autoregression. The natural language description is defined as textual description information, such as: "A large black tire stationary in the lane poses a high collision risk." Based on the textual description information, the vehicle can make intelligent, safe, and interpretable refined decisions. These decisions can be represented by control commands; that is, the textual description information is used by the vehicle to generate control commands.

[0052] Furthermore, after obtaining multimodal data of the unknown object, the system can generate a context-aware text prompt, such as: "Please describe this unknown object in detail and assess its potential risks." The system inputs the multimodal data and text prompt into the VLM, enabling the VLM to generate structured text description information based on the text prompt and multimodal data. The structured text description information includes object description, attributes, state, and risk assessment.

[0053] In this embodiment, the system can identify unknown objects based on predicted and actual 3D images, and then output a text description of the unknown objects. This enables the vehicle to accurately provide text descriptions of unknown objects in complex dynamic interactive scenarios, avoiding misjudgments and missed detections. Simultaneously, by providing text descriptions of unknown objects, the vehicle can assess risks and customize refined avoidance maneuvers, improving the intelligence and safety of autonomous driving.

[0054] In this embodiment, a first 3D image at time t and a second 3D image at time t+1 are constructed. The target 3D image at time t+1 is predicted based on the first 3D image. By comparing the target 3D image with the second 3D image, a subspace volume is obtained consisting of points in the two 3D images that differ in the comparison results. Then, textual description information of the unknown object corresponding to the subspace volume is generated for target detection based on the textual description information. In this embodiment, the target 3D image at the current time is predicted from the 3D image at the previous time and compared with the currently perceived 3D image. Significant differences (subspace volumes) caused by physical laws or dynamic behavior are identified as unknown objects, and textual descriptions are generated for these unknown objects. This allows the vehicle to accurately detect targets based on the textual descriptions, making correct decisions based on the detection results and improving vehicle driving safety.

[0055] In one embodiment, the second 3D image is relatively large, and comparing each point in the second 3D image with each point in the target 3D image would be inefficient. To address this, the system segments the first 3D image into multiple first spatial volumes, and based on the action sequence data and each first spatial volume, predicts the target spatial volume corresponding to each first spatial volume at time t+1.

[0056] For example, the system is equipped with a vector quantization variational autoencoder, which uses a codebook... (in For codebook, For the first Each code vector This represents the total number of code vectors in the codebook. (where the dimension is the code vector) Discretize into a world token graph, the discretized token graph is represented as: ,in This is a discretized world token graph. Represents the set of integers. , (where the height and width are respectively), its quantization process can be expressed as: ,in, For vector quantization variational autoencoders, This represents the L2 norm (Euclidean distance). This refers to finding the encoder output that matches the code vector. Code vector index with minimum distance between them .

[0057] It is understandable that the multimodal BEV feature map is the real three-dimensional world perceived by the system, while the discretized token map is a part of the real three-dimensional world. Assuming that the real three-dimensional world is a cube, the discretized token map is a small cube in the cube. The small cubes together make up the cube. That is, the discretized token is actually a spatial volume. In order to distinguish it from other spatial volumes that appear later, the token map discretized from the multimodal BEV feature map at time t is defined as the first spatial volume.

[0058] The process of predicting the target spatial object corresponding to the first spatial object at time t+1 based on the first spatial object and the action timing data is the same as the process of predicting the target three-dimensional image described above. Please refer to the above description for details, and it will not be repeated here.

[0059] Since the first three-dimensional image is segmented into multiple first spatial volumes, in order to adapt to the comparison of the target spatial volumes corresponding to the first spatial volumes, the second three-dimensional image needs to be segmented into multiple second spatial volumes. The segmentation process of the second spatial volumes is the same as that of the first spatial volumes. Please refer to the above description for details, and it will not be repeated here.

[0060] The target 3D image is composed of multiple target spatial volumes corresponding to multiple first spatial volumes. The system can compare multiple second spatial volumes simultaneously, thus improving the comparison efficiency between the second 3D image and the target 3D image. The spatial coordinates of the second spatial volume are the same as the spatial coordinates of the target spatial volume corresponding to the second spatial volume. The comparison is performed by comparing the first point of the second spatial volume with the second point corresponding to the first point in the target spatial volume. The comparison process between the first point and the second point is detailed above and will not be repeated here.

[0061] In this embodiment, the first three-dimensional image is segmented into multiple first spatial volumes, and the target spatial volume of the first spatial volume at time t+1 is predicted. Then, the second spatial volume segmented by the second three-dimensional image constructed at time t+1 is compared with the corresponding target spatial volume. That is, the comparison efficiency is improved by comparing multiple small spatial volumes.

[0062] Figure 2 A flowchart of a target detection method provided in this application embodiment Figure 2 ,based on Figure 1 In the embodiment shown, step S103 includes: Step S201: Compare the first point in the second three-dimensional image with the second point corresponding to the first point in the target three-dimensional image to obtain the divergence value of the first point. The divergence value is used to indicate whether the first point is the same as the corresponding second point.

[0063] In this embodiment, each point in the second three-dimensional image is defined as a first point. The first point has a corresponding second point in the target three-dimensional image. The spatial coordinates of the first point are the same as the spatial coordinates of the second point corresponding to the first point. The system compares each first point with the corresponding second point to obtain the divergence value corresponding to the first point. The divergence value is used to indicate whether the first point is the same as the corresponding second point.

[0064] Each first point (in The divergence value (in spatial coordinates) is expressed as:

[0065] in, This indicates the second point. The formula indicates that if the prediction (second point) differs from the actual result (first point), the divergence value of the first point is 1; otherwise, it is 0.

[0066] Step S202: Based on the divergence value, cluster each first point in the second three-dimensional image to obtain a subspace volume.

[0067] After obtaining the divergence value of each first point, clustering is performed on each first point in the second 3D graph based on the divergence value to obtain the subspace volume. Specifically, adjacent first points with a divergence value of 1 are clustered into a subspace volume. The clustering method can use existing clustering algorithms, that is, after clustering each first point with a preset divergence value into a cluster, a minimum bounding box is generated for the cluster, and all first points within the bounding box constitute the subspace volume. The preset value is, for example, 1.

[0068] Furthermore, the system can determine the anomaly score of the subspace volume by the divergence values ​​of each first point within the subspace. For example, the anomaly score can be obtained by summing the divergence values ​​of each first point within the subspace. The system transforms the subspace volume into a two-dimensional image and inputs the image data corresponding to the two-dimensional image at time t+1, along with the anomaly score, into the text generation model to obtain the text description information output by the text generation model. The text generation model can be the aforementioned large-scale visual language model. The anomaly score is used by the text generation model to assess the risk of unknown objects; the higher the anomaly score, the higher the risk assessment of the unknown object in the text description information. In addition, the process of transforming the subspace volume into a two-dimensional image and the acquisition of the image data corresponding to the two-dimensional image at time t+1 are detailed above and will not be repeated here. This method makes the risk assessment of unknown objects in the generated text description information more accurate.

[0069] It should be noted that if the second spatial body is compared with the corresponding target spatial body, the first point of the second spatial body is compared with the second point of the target spatial body corresponding to the first point.

[0070] In this embodiment, by comparing the first point in the second three-dimensional image with the second point corresponding to the first point in the target three-dimensional image, a divergence value is obtained. Based on the divergence value, each first point in the second three-dimensional image is quickly clustered to obtain a subspace volume.

[0071] Figure 3 A flowchart of a target detection method provided in this application embodiment Figure 3 ,based on Figure 2 In the embodiment shown, step S202 includes: Step S301: Determine the continuous score of the first point based on the divergence value of the first point and the divergence values ​​of the points adjacent to the first point in the second three-dimensional image.

[0072] In this embodiment, the negative log-likelihood of the 3D image or the spatial volume segmented from the 3D image under the predicted distribution can be used to obtain a smoother continuous fraction. :

[0073] in, For the first point The continuous score precisely quantifies the degree of surprise of a real-world observed object. A second spatial volume that characterizes a second three-dimensional image or a segmentation of a second three-dimensional image.

[0074] Specifically, the system determines the continuous score of the first point based on the divergence value of the first point and the divergence values ​​of the points adjacent to the first point in the second 3D image. For example, the continuous score of the first point can be obtained by accumulating the divergence value of the first point with the divergence values ​​of the points adjacent to the first point in the second 3D image.

[0075] Step S302: Cluster adjacent first points with consecutive scores greater than a preset threshold into subspace volumes.

[0076] Using the above method, the continuous score of each first point in the second 3D image can be determined. The system clusters adjacent first points with continuous scores greater than a preset threshold into a subspace volume. The clustering method can use existing clustering algorithms. After clustering adjacent first points with continuous scores greater than the preset threshold into a cluster, a minimum bounding box is generated for each cluster. All first points within the bounding box constitute a subspace volume.

[0077] Furthermore, the system can filter out the first points caused by sensor noise or minor prediction errors in the model. For example, in a region of the second 3D image with 50 points, 45 points have a divergence value of 1, while only 5 points have a divergence value of 0. The points with a divergence value of 0 need to be filtered out. Another example is using the average of the continuous scores of all points within a region of the second 3D image as a dynamic threshold. If there are points within that region with continuous scores significantly higher or lower than the dynamic threshold, these points are considered noise points and are filtered out. After filtering, the remaining first points in the second 3D image are clustered.

[0078] In this embodiment, the degree of surprise of the actually observed object is characterized by the continuous score of the first point, thereby accurately determining the subspace volume representing the unknown object based on the continuous score.

[0079] Figure 4 A flowchart of a target detection method provided in this application embodiment Figure 4 .based on Figures 1 to 3 In any of the embodiments shown, step S102 includes: Step S401: Construct historical time-series data based on the first three-dimensional image and multiple historical three-dimensional images. The historical three-dimensional images are three-dimensional images constructed from image data collected at historical moments, and the historical moments are earlier than time t.

[0080] In this embodiment, the prediction model set in the system can be a diffusion model, which can predict the target 3D image at time t+1 based on the historical time sequence of the 3D image.

[0081] For example, the system constructs historical time-series data based on the first three-dimensional image at time t and multiple historical three-dimensional images. The historical three-dimensional images are three-dimensional images constructed from image data collected by the sensor at historical times. The historical times are earlier than time t, and each historical time is tn, tn-1, ..., t-2, t-1.

[0082] Step S402: Input the historical time series data and action sequence data into the prediction model to obtain the target 3D image at time t+1 output by the prediction model.

[0083] The system inputs historical time series data and action sequence data into the prediction to obtain the target 3D image at time t+1 output by the prediction model.

[0084] For example, the diffusion model is responsible for basing its data on historical time series data. Predicting the next moment .in, for The first three-dimensional image at that moment, For the current time step, The length of the historical sequence used. The next moment to be predicted The target 3D image. It should be noted that if the first 3D image is segmented into multiple first spatial volumes, then... for The first spatial volume of time, The next moment to be predicted The target space volume.

[0085] The diffusion model can be a Transformer (a deep learning model with self-attention mechanism) discrete diffusion model, the core of which is learning an inverse denoising process: a Transformer denoising network that uses historical sequences... Autonomous vehicle planning actions (action sequence data) Current denoising time step (in Using the sampling step in the diffusion process as a strong condition, model the probability transformation kernel. In this expression, For learnable parameters Defined probability distribution, In the noise reduction step Noise-added image at the time, This is the denoised image predicted by the model.

[0086] In this embodiment, the diffusion model uses historical time-series data and vehicle motion planning as conditions to generate a prediction for the next moment through a complete inverse sampling process. It then iterates in an autoregressive manner to achieve long-term time-series forecasting. The predicted... It not only captures complex spatiotemporal dynamics and interactions, but also serves as a dynamic prior benchmark for defining physical reality, which is used for subsequent anomaly detection in the system.

[0087] Figure 5 A flowchart of a target detection method provided in this application embodiment Figure 5 .based on Figures 1 to 4 In any of the embodiments shown, after step S104, the method further includes: Step S501: Based on the image data, text description information and scene metadata of the subspace volume at time t+1, construct training samples and store the training samples in a preset storage area.

[0088] In this embodiment, a first model is set in the system to predict the 3D image at time t+1; that is, the first model is the prediction model described above. The system can fine-tune the prediction model during vehicle downtime.

[0089] For example, the system is equipped with a continuous learning and memory module, which is the core of the closed loop for realizing the evolution of system knowledge. It can transform instantaneous unknown encounters into internalized permanent knowledge, enabling the system to evolve from a static perception system to a lifelong learning system.

[0090] The continuous learning and memory module includes an experience memory pool component. This component is implemented on the onboard computing unit as a space-constrained first-in-first-out queue. Whenever the text generation model successfully generates a semantic description for a physical anomaly, the system sends a data tuple containing the complete context. Encapsulate and store in this memory pool:

[0091] in: The original multimodal data (such as cropped image patches and point clouds) for the abnormal region (subspace volume); Textual description information for the subspace; This includes the scene metadata of the event, such as time, location, vehicle status, and weather. It can be understood that the system constructs training samples based on the image data, text description information, and scene metadata of the subspace at time t+1, and stores the training samples in a preset storage area, such as an experience memory pool.

[0092] Step S502: Train the first model based on the training samples stored in the preset storage area.

[0093] The continuous learning and memory module also includes an online experience playback and model update mechanism. This mechanism utilizes the vehicle's computing idle periods (such as when charging or parking overnight) to sample a small batch of new experiences from the experience memory pool. (in (For batch size), and incrementally fine-tune the first model.

[0094] The core of this fine-tuning process is: First, utilizing... As a weakly supervised signal (e.g., through text-image contrast learning) to guide the model to learn new concepts, that is, to train the first model with training samples stored in a pre-defined storage area.

[0095] Furthermore, to overcome the catastrophic forgetting problem, a knowledge distillation loss is introduced. This involves retaining a frozen second model and adding a regularization term to force the prediction distribution of the continuously updated first model on known scenario data to remain consistent with the teacher model. The second model serves as the teacher model in the system, while the first model acts as the student model for the second model. In other words, the system trains the second model based on training samples stored in a pre-defined storage area and then guides the first model's learning based on the trained second model.

[0096] In this embodiment, training samples are constructed using image data, text description information, and scene metadata from the subspace. These training samples guide the model to learn text-images, enabling the model to learn new knowledge, improve the prediction accuracy of 3D images, continuously learn new knowledge, adapt to new things in the open world, avoid misjudging similar new things, and improve the system's adaptability.

[0097] In one embodiment, the system updates the loss functions of both the second and first models based on the first loss function of the second model during the pre-training phase; then, it trains the second model with the updated loss function using training samples stored in a preset storage area. The updated loss function is used to balance the second model's learning of new knowledge with the retention of old knowledge.

[0098] Specifically, the training and fine-tuning process of the system is guided by a unified, multi-part objective function to ensure a balance between the predictive power of the predictive model and the retention of new knowledge.

[0099] First, in the offline pre-training phase of the prediction model, initial training is performed on a large number of offline driving datasets. Its main objective is to minimize the impact of the next time step on the 3D image or spatial volume. Predicted loss This loss is typically expressed in the form of cross-entropy: ; in, To predict losses, represent coordinates on For the prediction model parameters Defined conditional probability distribution, For in position The true next-time token index (divergence value). The historical time series on which the model relies for prediction (where For the current time step, (length of the historical sequence) Summing the motion sequence data planned for the autonomous vehicle This represents the summation of losses for all spatial locations (points in a 3D image). Simultaneously, the vector quantization variational autoencoder is also trained using a reconstruction loss and a codebook learning loss to ensure its representation learning capability.

[0100] During the online fine-tuning phase after deploying the vehicle prediction model, when the continuous learning and memory module is activated, the objective function (the loss function of the second model during the pre-training phase) is transformed into a weighted sum. : ; in, The total loss function during the online fine-tuning phase. and It is a hyperparameter used to balance the plasticity of learning new knowledge with the stability of retaining old knowledge. This represents the learning loss from new experiences. The specific form of this loss can be determined based on the type of weak supervision signal, such as contrastive learning loss, classification loss based on VLM pseudo-labels, or loss that directly guides the prediction model to reduce its prediction error when encountering new anomalies. For knowledge distillation loss, it measures the student model With teacher model In a fixed, representative dataset of old scenes The difference in the predicted probability distribution.

[0101] The Kullback-Leibler (KL) divergence is typically used for calculation. ; in, and These represent the teacher model and the student model in the dataset, respectively. The predicted probability distribution KL divergence is used to measure the difference between two probability distributions.

[0102] In this embodiment, the loss functions of the second model and the first model are updated using the first loss function of the second model in the pre-training stage, and then the second model with the updated loss function is trained to balance the second model's learning of new knowledge and retention of old knowledge.

[0103] Based on the above embodiments, the target detection method of this application will be described in general as follows: The core idea of ​​the object detection method provided in this application is to construct an intelligent perception system capable of proactively predicting, understanding, and adapting to open driving environments. This system, through a four-module collaborative architecture, achieves a complete closed loop, from detecting physical anomalies to assigning them natural language semantics, and finally internalizing new knowledge into the model's capabilities. The overall system architecture is as follows: Figure 6 As shown, it clearly demonstrates the flow and processing of data among the four core modules: Generative Spatiotemporal Prior Modeling Module: As the cognitive core of the system, this module receives continuous, multimodal sensor input streams (such as camera images and LiDAR point clouds) and encodes them into a unified bird's-eye view (BEV) latent representation. Based on these historical representations and the vehicle's planning intentions, this module autoregressively generates high-fidelity predictions of the scene's physical state over a future period. These predictions constitute dynamic prior knowledge of the normal world.

[0104] Prior-guided anomaly detection module: This module is the execution unit for open-set detection. It compares the predicted BEV generated by the prediction model with the actual BEV generated from the current sensor data in real time. By calculating the spatiotemporal differences between the two, this module can accurately locate regions in the physical world that do not conform to the model's predictions, i.e., anomalous regions. These regions represent potentially unknown or abnormally behaving objects.

[0105] The open VLM semantic interpretation and generation module is responsible for semantic parsing and information enhancement of detected anomalies. Once the anomaly detector outputs the location of an anomaly region, this module extracts the corresponding raw sensor data, especially image patches, and uses a VLM to generate a natural language description of the objects within that region. This process is called semantic grounding, which endows purely physical anomalies with rich and understandable semantic connotations.

[0106] Continuous Learning and Memory Module: This module is crucial for the system's self-evolution. It maintains an experience memory pool to store data pairs generated by previous modules (visual data of abnormal regions, text descriptions generated by VLM). Through online experience replay and knowledge distillation mechanisms, this newly acquired knowledge is used to continuously and incrementally fine-tune the predictive model, constantly expanding its cognitive boundaries and thus achieving long-term adaptation to the open world.

[0107] The system integrates the originally isolated prediction, detection, understanding and learning tasks into a dynamic and self-consistent cognitive loop through the tight coupling between the four modules.

[0108] Understandably, the system's complete closed-loop process of detection, understanding, memory, and learning enables the entire process from identifying unknown events to semantic understanding and knowledge internalization. Objects initially considered anomalous by the prediction model will be correctly modeled and incorporated into the prediction prior after learning, no longer triggering anomalous responses. This achieves a paradigm shift from open-set detection to open-world adaptation.

[0109] Understandably, the generative spatiotemporal prior modeling module learns the spatiotemporal dynamics of complex driving scenarios and predicts future scenario states in an autoregressive manner, thereby constructing a high-fidelity physical dynamic prior. The prior-guided anomaly detection module uses dynamic priors as a benchmark and calculates the spatiotemporal divergence between model predictions and actual observations to achieve accurate detection and localization of unknown or anomalous targets. Open VLM semantic interpretation and generation module: calls a large-scale vision-language model to perform zero-shot semantic annotation on detected anomalous targets and generate interpretable natural language descriptions rich in context information; Continuous learning and memory module: By constructing an experience memory pool and an online replay mechanism, newly acquired (anomaly-semantic) experience samples are incrementally internalized into model knowledge, realizing the adaptive evolution of the system and the expansion of knowledge boundaries.

[0110] The target detection method provided in this application establishes a complete understanding loop from "physical anomaly detection" to "advanced semantic understanding" and then to "online knowledge evolution," enabling the system to proactively identify, understand, and continuously adapt to unknown entities and abnormal events in the open world, thereby significantly improving the driving safety and reliability of vehicles containing this system.

[0111] Corresponding to the target detection method described above, this application also provides a target detection system. Figure 7 This is a schematic diagram of a target detection system provided in an embodiment of this application. The target detection system 700 provided in this embodiment includes: The first construction module 710 is used to construct a first three-dimensional image based on the image data collected by the sensors in the vehicle at time t, and to predict the target three-dimensional image at time t+1 based on the first three-dimensional image and the vehicle's action sequence data. The action sequence data includes each action to be performed by the vehicle between time t and time t+1. The second construction module 720 is used to construct a second three-dimensional image based on the image data collected by the sensor in the vehicle at time t+1. The determination module 730 is used to determine a subspace volume based on the comparison result between the target 3D image and the second 3D image. The subspace volume includes points in the second 3D image where the comparison result is different. The generation module 740 is used to generate text description information of unknown objects corresponding to the subspace volume. The text description information is used for target detection.

[0112] In some implementations, the target detection system 700 is also used for: The first three-dimensional image is segmented into multiple first spatial volumes, and the target spatial volume corresponding to each first spatial volume at time t+1 is predicted based on the action sequence data and each first spatial volume. Based on the comparison results between the target 3D image and the second 3D image, the subspace volume is determined, including: The second three-dimensional image is segmented into multiple second spatial volumes; Based on the comparison results between the target space volume and the corresponding second space volume, a subspace volume is determined in the second space volume. The spatial coordinates of the target space volume are the same as the spatial coordinates of the second space volume corresponding to the target space volume.

[0113] In some implementations, the target detection system 700 is also used for: The first point in the second 3D image is compared with the second point corresponding to the first point in the target 3D image to obtain the divergence value of the first point. The divergence value is used to indicate whether the first point is the same as the corresponding second point. Based on the divergence values, cluster each first point in the second three-dimensional image to obtain a subspace volume.

[0114] In some implementations, the target detection system 700 is also used for: The continuous score of the first point is determined based on the divergence value of the first point and the divergence values ​​of the points adjacent to the first point in the second three-dimensional image. Each adjacent first point with a continuous score greater than a preset threshold is clustered into a subspace volume.

[0115] In some implementations, the target detection system 700 is also used for: The anomaly score of the subspace volume is determined based on the divergence value of each first point in the subspace volume; The subspace volume is transformed into a two-dimensional image, and the image data and anomaly score corresponding to the two-dimensional image at time t+1 are input into the text generation model to obtain the text description information output by the text generation model.

[0116] In some implementations, the target detection system 700 is also used for: Historical time-series data is constructed based on the first 3D image and multiple historical 3D images. The historical 3D images are 3D images constructed from image data collected at historical moments, and the historical moments are earlier than time t. Historical time series data and action sequence data are input into the prediction model to obtain the target 3D image at time t+1 output by the prediction model.

[0117] In some implementations, the target detection system 700 is also used for: Based on the image data, text description information and scene metadata of the subspace at time t+1, training samples are constructed and stored in a preset storage area. The first model is trained based on the training samples stored in the preset storage area.

[0118] In some implementations, the target detection system 700 is also used for: The second model is trained based on the training samples stored in the preset storage area, and the first model is guided to learn based on the trained second model.

[0119] In some implementations, the target detection system 700 is also used for: Update the loss functions of the second model and the first model based on the first loss function of the second model during the pre-training phase; The second model, which updates the loss function, is trained based on the training samples stored in the preset storage area.

[0120] The target detection system and the target detection method provided in the above embodiments of this application belong to the same concept and can execute the target detection method provided in any of the above embodiments of this application. They possess the corresponding functional modules and beneficial effects for executing the target detection method. Technical details not described in detail in this embodiment can be found in the specific processing content of the target detection method provided in the above embodiments of this application, and will not be repeated here.

[0121] The functions implemented by each module in the target detection system can be implemented by the same or different processors, and this application embodiment does not limit this.

[0122] It should be understood that the modules in the above target detection system can be implemented by a processor calling firmware. For example, the system includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each module of the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal to the device or external to the system. Alternatively, the modules in the system can be implemented as hardware circuits. By designing the hardware circuits, some or all of the module functions can be implemented. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above modules are implemented by designing the logical relationships of the components within the circuit. In another implementation, the hardware circuit can be implemented by a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files to implement the functions of some or all of the above modules. All modules of the above target detection system can be implemented entirely by a processor calling firmware, entirely by hardware circuits, or partially by a processor calling firmware with the remaining parts implemented by hardware circuits.

[0123] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above modules. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0124] As can be seen, each module in the above target detection system can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor types.

[0125] Furthermore, the modules in the above target detection system can be integrated in whole or in part, or they can be implemented independently. In one implementation, these modules are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the modules of the device. The at least one processor can be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0126] This application provides another structural schematic diagram of a vehicle, see [link to schematic diagram]. Figure 8 As shown, the vehicle includes a memory 800 and a processor 810; wherein the memory 800 is connected to the processor 810 and is used to store programs; the processor 810 is used to implement the target detection method disclosed in any of the above embodiments by running the programs stored in the memory 800.

[0127] Specifically, the vehicle may also include: a bus, a communication interface 820, an input device 830, an output device 840, and a target detection system 850. The vehicle may also include a data transceiver module, an image monitoring module, and a signal monitoring module.

[0128] The processor 810, memory 800, communication interface 820, input device 830, output device 840, and target detection system 850 are interconnected via a bus. Among them: A bus can include a pathway for transmitting information between various components in a vehicle.

[0129] The processor 810 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0130] The processor 810 may include a main processor, as well as a baseband chip, modem, etc.

[0131] The memory 800 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 800 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0132] Input device 830 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0133] Output device 840 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0134] The communication interface 820 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0135] The processor 810 executes the program stored in the memory 800 and calls other devices, which can be used to implement the various steps of any of the target detection methods provided in the above embodiments of this application.

[0136] It should be noted that the vehicle can be an in-vehicle terminal, mobile phone, wearable device or server, etc.; or it can be a vehicle that includes an in-vehicle terminal, etc.

[0137] This application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in the memory through the data interface to execute the target detection method described in any of the above embodiments. For the specific processing procedure and its beneficial effects, please refer to the above embodiments of the target detection method.

[0138] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the target detection methods according to various embodiments of this application as described in any of the above embodiments of this specification.

[0139] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the power device, as a standalone firmware package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0140] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor to perform the steps of the target detection method according to various embodiments of this application described in any of the above embodiments of this specification, specifically implementing the steps of the target detection method as described above.

[0141] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0142] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0143] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0144] The units of the apparatus in the various embodiments of this application can be merged, divided, and deleted according to actual needs.

[0145] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0146] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0147] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or as firmware functional modules or sub-modules.

[0148] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer firmware, or a combination of both. To clearly illustrate the interchangeability of hardware and firmware, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or firmware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0149] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, firmware units executed by a processor, or a combination of both. The firmware unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0150] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0151] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target detection method, characterized in that, The target detection method, applied to a target detection system, includes: A first three-dimensional image is constructed based on image data collected by sensors in the vehicle at time t. Based on the first three-dimensional image and the vehicle's action sequence data, a target three-dimensional image at time t+1 is predicted. The action sequence data includes each action to be performed by the vehicle between time t and time t+1. A second three-dimensional image is constructed based on the image data collected by the sensors in the vehicle at time t+1. Based on the comparison results between the target 3D image and the second 3D image, a subspace volume is determined, wherein the subspace volume includes points in the second 3D image whose comparison results are different; Generate textual description information for the unknown objects corresponding to the subspace volume, and the textual description information is used for target detection.

2. The target detection method according to claim 1, wherein predicting the target three-dimensional image at time t+1 based on the first three-dimensional image and the vehicle's motion sequence data comprises: The first three-dimensional image is segmented into multiple first spatial volumes, and based on the action sequence data and each first spatial volume, the target spatial volume corresponding to each first spatial volume at time t+1 is predicted. The step of determining the subspace volume based on the comparison result between the target 3D image and the second 3D image includes: The second three-dimensional image is segmented into multiple second spatial volumes; Based on the comparison result between the target space volume and the corresponding second space volume, a subspace volume is determined in the second space volume, and the spatial coordinates of the target space volume are the same as the spatial coordinates of the second space volume corresponding to the target space volume.

3. The target detection method according to claim 1, characterized in that, The step of determining the subspace volume based on the comparison result between the target 3D image and the second 3D image includes: The first point in the second three-dimensional image is compared with the second point corresponding to the first point in the target three-dimensional image to obtain the divergence value of the first point. The divergence value is used to indicate whether the first point is the same as the corresponding second point. Based on the divergence value, each first point in the second three-dimensional image is clustered to obtain the subspace volume.

4. The target detection method according to claim 3, characterized in that, The step of clustering each first point in the second three-dimensional image based on the divergence value includes: The continuous score of the first point is determined based on the divergence value of the first point and the divergence values ​​of the points adjacent to the first point in the second three-dimensional image. Each adjacent first point whose consecutive scores are greater than a preset threshold is clustered into the subspace volume.

5. The target detection method according to claim 3, characterized in that, The generation of text description information for the unknown object corresponding to the subspace volume includes: The anomaly score of the subspace volume is determined based on the divergence value of each first point in the subspace volume; The subspace volume is transformed into a two-dimensional image, and the image data corresponding to the two-dimensional image at time t+1 and the anomaly score are input into the text generation model to obtain the text description information output by the text generation model.

6. The target detection method according to claim 1, characterized in that, The step of predicting the target 3D image at time t+1 based on the first 3D image and the vehicle's motion sequence data includes: Historical time-series data is constructed based on the first three-dimensional image and multiple historical three-dimensional images. The historical three-dimensional images are three-dimensional images constructed from image data collected at the historical time, and the historical time is earlier than time t. The historical time series data and the action sequence data are input into the prediction model to obtain the target three-dimensional image at time t+1 output by the prediction model.

7. The target detection method according to claim 1, characterized in that, The target detection system includes a first model for predicting the 3D image at time t+1. After generating textual description information of the unknown object corresponding to the subspace volume, it further includes: Based on the image data of the subspace at time t+1, the text description information, and the scene metadata of the subspace, a training sample is constructed and stored in a preset storage area. The first model is trained based on the training samples stored in the preset storage area.

8. The target detection method according to claim 7, characterized in that, The target detection system sets up a second model, and the first model is trained based on the training samples stored in the preset storage area, including: The second model is trained based on the training samples stored in the preset storage area, and the first model is guided to learn based on the trained second model.

9. The target detection method according to claim 8, characterized in that, The step of training the second model based on the training samples stored in the preset storage area includes: Update the loss functions of the second model and the first model based on the first loss function of the second model during the pre-training phase; The second model, which updates the loss function, is trained based on the training samples stored in the preset storage area.

10. A target detection system, characterized in that, Including memory and processor, among which, The memory is connected to the processor and is used to store programs; The processor is used to implement the target detection method as described in any one of claims 1-9 by running the program in the memory.

11. A vehicle, characterized in that, The vehicle includes a target detection system, which implements the target detection method as described in any one of claims 1-9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the target detection method as described in any one of claims 1-9.

13. A computer program product, wherein when executed by a processor, the computer program implements the target detection method as described in any one of claims 1-9.

Images