Method, device and storage medium for roadside perception

Abstract

The application discloses a road end perception method, device and storage medium, the method comprises the following steps: collecting a road image to obtain a road image dataset; determining a target dataset according to the perception result of the road image dataset by a first perception model and the detection result of the road image dataset by a road end change detection model; the target dataset represents a set of road images with missing perception results; sending the target dataset to a cloud device; receiving a second perception model sent by the cloud device and updating the first perception model to the second perception model; the second perception model is obtained by fine-tuning training of the first perception model by the cloud device using the target dataset; and performing target detection on the real-time collected road image by using the second perception model, the application continuously improves the road end perception ability in the vehicle-road cooperation scene through the iterative update of the perception model.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to a method, device and storage medium for roadside perception. Background Technology

[0002] Perception systems are a crucial component of autonomous driving. They acquire information about the surrounding environment, outputting the results to prediction and planning decision-making modules. The vehicle then executes driving actions through these modules, achieving autonomous driving. The environmental information acquired by the perception system includes both static and dynamic information. Static information includes lane markings and road boundaries, while dynamic information includes the type, size, position, and speed of vehicles and pedestrians. Compared to vehicle-side perception, roadside perception devices are typically located in fixed positions, resulting in a relatively static perception background. Therefore, improvements to roadside perception capabilities generally focus on enhancing the perception of dynamic information, i.e., the ability to detect moving objects.

[0003] With the development of artificial intelligence technology, existing technologies generally use pre-trained deep learning models as perception models for roadside sensing. However, the training data for roadside sensing models is difficult to collect, and the amount of data on anomalies, newly emerging situations, and potential driving hazards is small, accounting for a low proportion of the training data. Therefore, the generalization ability of existing roadside sensing models is not high, and in practical applications, it is often difficult to accurately identify such scenarios.

[0004] Therefore, improving roadside perception capabilities is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The application provides a method, device, and storage medium for roadside perception, which can continuously improve roadside perception capabilities in vehicle-road cooperative scenarios through iterative updates of the perception model.

[0006] On one hand, embodiments of this application provide a method for roadside sensing, applied to roadside equipment, the method comprising:

[0007] Collect road images to obtain a road image dataset;

[0008] Based on the perception results of the first perception model on the road image dataset and the detection results of the road end change detection model on the road image dataset, a target dataset is determined; the target dataset is used to represent the set of road images for which the perception results are missing.

[0009] Send the target dataset to the cloud device;

[0010] The system receives a second perception model sent by the cloud device and updates the first perception model with the second perception model; the cloud device stores the first perception model; the second perception model is obtained by the cloud device through fine-tuning and training the first perception model using the target dataset;

[0011] The second perception model is used to perform target detection on real-time acquired road images.

[0012] In one exemplary embodiment, the road image dataset includes road images at multiple times. The step of determining the target dataset based on the perception results of the first perception model and the detection results of the road end change detection model on the road image dataset includes:

[0013] The first perception model is used to perform object detection on the first image to obtain a first perception result; the first image is a road image at time Tn of the road image dataset; the first perception result includes the object type and the region information of the corresponding area of ​​the object in the first image; n is a positive integer.

[0014] The first perception model is used to perform target detection on the second image to obtain a second perception result; the second image is the road image at time Tn+1 of the road image dataset; the second perception result includes the object type and the region information of the corresponding area of ​​the object in the second image.

[0015] The first image and the second image are processed using the road end change detection model to obtain the region information of the changed area in the second image relative to the first image.

[0016] The target data is obtained by comparing the area information of the changed area with the area information of the corresponding area of ​​the object in the second perception result.

[0017] When the target data meets the preset conditions, the set of target data is determined as the target dataset.

[0018] In one exemplary embodiment, comparing the region information of the changed region with the region information of the corresponding region of the object in the second perception result to obtain the target data includes:

[0019] Based on the area information of the changed area and the area information of the area corresponding to the object in the second perception result, the overlap between the area corresponding to the object in the second perception result and the changed area is determined;

[0020] When the overlap is less than a preset threshold, the second image is determined as the target image, and the changed region is determined as the target region;

[0021] The target image and the target region are defined as target data.

[0022] In one exemplary embodiment, determining the set of target data as the target dataset when the target data meets preset conditions includes:

[0023] When the target data in the target data set accumulates to a preset quantity, the target data set is determined as the target dataset.

[0024] In one exemplary embodiment, the method further includes:

[0025] Send the version information of the first perception model to the cloud device;

[0026] Receive the version information of the second perception model sent by the cloud device;

[0027] Based on the version information of the second perception model, deploy the second perception model sent by the cloud device.

[0028] In one exemplary embodiment, the method further includes:

[0029] The target dataset is compressed to obtain the compressed target dataset;

[0030] The compressed dataset is de-identified to obtain the de-identified target dataset;

[0031] The de-identified dataset is then encrypted to obtain the encrypted target dataset.

[0032] The encrypted target dataset is sent to the cloud device.

[0033] On the other hand, embodiments of this application provide a roadside sensing method applied to a cloud device, wherein the cloud device stores a first sensing model, and the method includes:

[0034] The system receives a target dataset sent by a roadside device. The roadside device is equipped with a first perception model and a roadside change detection model. The roadside device is used to collect road images to obtain a road image dataset. Based on the perception results of the first perception model on the road image dataset and the detection results of the roadside change detection model on the road image dataset, the system determines a target dataset. The target dataset represents the set of road images in the road image dataset whose perception results from the first perception model are missing.

[0035] The first perception model is fine-tuned and trained using the target dataset to obtain the second perception model, and the stored first perception model is updated to the second perception model.

[0036] The second sensing model is sent to the roadside equipment.

[0037] In one exemplary embodiment, the step of fine-tuning the first perception model using the target dataset to obtain the second perception model includes:

[0038] The target dataset is labeled to obtain labeled data;

[0039] Using the labeled data, the first perception model is fine-tuned and trained to obtain the second perception model.

[0040] In one exemplary embodiment, the method further includes:

[0041] Receive the version information of the first sensing model sent by the roadside device;

[0042] After fine-tuning and training the first perception model using the target dataset to obtain the second perception model, the version information of the second perception model is generated based on the version information of the first perception model.

[0043] The version information of the second sensing model is sent to the roadside equipment.

[0044] On the other hand, embodiments of this application also provide a roadside device, which includes:

[0045] The acquisition module is used to acquire road images and obtain a road image dataset;

[0046] The processing module is used to determine a target dataset based on the perception results of the first perception model on the road image dataset and the detection results of the road end change detection model on the road image dataset; the target dataset is used to represent the set of road images where the perception results are missing.

[0047] The sending module is used to send the target dataset to the cloud device;

[0048] The receiving module is configured to receive the second perception model sent by the cloud device and update the first perception model to the second perception model; the second perception model is obtained by the cloud device through fine-tuning and training the first perception model using the target dataset;

[0049] The perception module is used to perform target detection on real-time acquired road images using the second perception model.

[0050] On the other hand, embodiments of this application also provide a cloud device, which stores a first perception model, and the cloud device includes:

[0051] A receiving module is used to receive a target dataset sent by a roadside device. The roadside device is equipped with a first perception model and a roadside change detection model. The roadside device is used to acquire road images to obtain a road image dataset. Based on the perception results of the first perception model on the road image dataset and the detection results of the roadside change detection model on the road image dataset, a target dataset is determined. The target dataset represents the set of road images in the road image dataset whose perception results from the first perception model are missing.

[0052] The processing module is used to fine-tune the first perception model using the target dataset to obtain the second perception model, and update the stored first perception model to the second perception model.

[0053] The sending module is used to send the second sensing model to the roadside equipment.

[0054] On the other hand, this application also provides a sensing system, which includes roadside equipment and cloud equipment;

[0055] The roadside device is used to acquire road images to obtain a road image dataset; and to determine a target dataset based on the perception results of the first perception model on the road image dataset and the detection results of the roadside change detection model on the road image dataset; the target dataset represents the set of road images in the road image dataset for which the perception results of the first perception model are missing; and to send the target dataset to a cloud device; and to receive a second perception model sent by the cloud device and update the first perception model to the second perception model; and to perform target detection on the real-time acquired road images using the second perception model;

[0056] The cloud device is configured to receive a target dataset sent by the roadside device; fine-tune and train the first perception model using the target dataset to obtain a second perception model, update the stored first perception model to the second perception model; and send the second perception model to the roadside device.

[0057] On the other hand, this application also provides an electronic device including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the roadside sensing method as described above.

[0058] On the other hand, this application also provides a computer storage medium storing at least one instruction or at least one program, which is loaded and executed by a processor to implement the roadside sensing method as described above.

[0059] On the other hand, this application also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for implementing the roadside sensing described above.

[0060] The roadside sensing method, device, and storage medium provided in this application have the following technical advantages:

[0061] In this embodiment, the vehicle-mounted device acquires road images and then compares the perception results of the perception model deployed on it with the detection results of the change detection model to filter out road images whose perception results are missing. This set of road images is then sent to the cloud device to further enrich the training data of the perception model. The cloud device uses the dataset sent by the vehicle-mounted device as training samples to fine-tune the perception model and then sends a new perception model to the vehicle-mounted device. This iterative update process continuously improves the road-mounted perception capability in the vehicle-road cooperative scenario. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 This is an application environment diagram of a roadside sensing method provided in an embodiment of this application.

[0064] Figure 2 This is a flowchart illustrating a roadside sensing method provided in an embodiment of this application.

[0065] Figure 3 This is a schematic block diagram illustrating the determination of target data provided in an embodiment of this application.

[0066] Figure 4 This is a flowchart illustrating the process of determining the target dataset provided in an embodiment of this application.

[0067] Figure 5This is a schematic diagram illustrating the principle of determining the overlap between the corresponding region and the changing region of the object in the second perception result, as provided in the embodiments of this application.

[0068] Figure 6 This is a schematic diagram of the process by which a cloud device uses a target dataset to fine-tune and train a first perception model, as provided in an embodiment of this application.

[0069] Figure 7 This is a schematic diagram of the interactive process for generating version information provided in an embodiment of this application.

[0070] Figure 8 This application is a schematic diagram of a sensing system framework provided in an embodiment.

[0071] Figure 9 This is a schematic diagram of the structure of the roadside equipment provided in the embodiments of this application.

[0072] Figure 10 This is a schematic diagram of the structure of the cloud device provided in the embodiments of this application.

[0073] Figure 11 This is a hardware structure block diagram of a server for a roadside sensing method provided in an embodiment of this application. Detailed Implementation

[0074] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0075] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0076] Figure 1 This is an application environment diagram of a roadside sensing method provided in an embodiment of this application.

[0077] like Figure 1As shown, vehicle-road cooperation is considered an effective means of assisting autonomous driving systems. It utilizes advanced wireless communication and next-generation internet technologies to implement comprehensive, real-time dynamic information interaction between vehicles and infrastructure. Based on the collection and fusion of dynamic traffic information across all times and spaces, it conducts active safety control of vehicles and cooperative road management, fully realizing effective collaboration between people, vehicles, and roads, ensuring traffic safety, improving traffic efficiency, and thus forming a safe, efficient, and environmentally friendly road traffic system. Vehicle-road cooperation relies on roadside communication protocols to distribute auxiliary traffic information and fused perception results to passing vehicles, facilitating autonomous driving decisions on the vehicle side. Specifically, roadside equipment 10 typically carries sensors such as cameras, millimeter-wave radar, and lidar, which can collect road environment information in real time, including static information such as lane lines and road boundaries, as well as dynamic information such as the type, size, and position of vehicles and pedestrians. Roadside equipment 10 also deploys a roadside perception model to process the environmental information collected by the sensors. The roadside perception model is typically a neural network model trained using deep learning technology. When performing roadside perception, the roadside device 10 uploads the collected environmental information to the cloud device 20. The cloud device 20 can analyze the environmental information uploaded by the roadside device 10 to obtain road analysis results, such as the location of objects on the road, traffic flow, pedestrian flow, and information on sudden accidents. The analysis results can then be sent to vehicles to provide a basis for planning driving routes and executing driving actions. The cloud device 20 can also be used to train neural network models to obtain roadside perception models, which are then sent back to the roadside device 10. The cloud device 20 may include a standalone server, a distributed server, or a server cluster consisting of multiple servers. The cloud device 20 may include network communication units, processors, and memory, etc.

[0078] However, existing roadside perception models often have a small proportion of long-tail data in their training samples. Long-tail data refers to data on abnormal, newly emerging, or potentially hazardous driving situations. This type of data is difficult to collect due to its low probability of occurrence. Therefore, existing roadside perception models trained with insufficient long-tail data often have low generalization ability. For example, after a new car model is released, its frequency in the training data may be very low or even zero, causing the roadside perception model to fail to detect it. Similarly, when two vehicles collide, their shapes change. Since most of the training samples consist of vehicles with normal shapes, the roadside perception model often cannot obtain accurate detection results in such abnormal situations. In summary, when performing real-world environmental perception, existing roadside perception models cannot meet the perception requirements if they encounter scenarios not present in the training data or scenarios that occur infrequently. Therefore, this application proposes a roadside perception method, device, and storage medium to continuously improve roadside perception capabilities in vehicle-road cooperative scenarios during the iterative update of the perception model.

[0079] Figure 2 This is a flowchart illustrating a roadside sensing method provided in an embodiment of this application.

[0080] S201: Roadside equipment collects road images to obtain a road image dataset.

[0081] Roadside equipment is typically installed on the roadside, at intersections, etc., where its view is not easily obstructed. Roadside equipment includes multiple sensors; in this embodiment, it includes at least a camera to capture real-time road images. It should be understood that roadside equipment may also include lidar, millimeter-wave radar, weather detectors, etc. In addition to fixed backgrounds such as roads, lane lines, and trees, the road images captured by roadside equipment typically also include dynamic traffic participants such as pedestrians and vehicles. The collection of road images captured by roadside equipment is denoted as the road image dataset.

[0082] In this embodiment, a first perception model and a roadside change detection model are deployed on the roadside equipment. The first perception model is a trained deep learning model capable of performing perception tasks such as target detection and region segmentation on the acquired road images.

[0083] In one embodiment, the first perception model can be trained using preset perception training samples. The preset perception training samples can be image data or video data; this application does not limit the specific form of the preset perception training samples. It should be noted that the preset perception training samples can be datasets commonly used in the field of autonomous driving, such as the Coco dataset, the Cityscapes dataset, etc.; the preset perception training samples can also be multiple road images pre-collected by roadside equipment and label data obtained after annotating multiple road images. Using the above perception training samples, the perception model is trained to obtain the first perception model. The perception model can at least include a feature extraction module and a prediction module. The feature extraction module is used to extract features from the perception training samples, and the prediction model is used to output perception results based on the extracted features. During training, images from the perception training samples can be input into the perception model to obtain the perception results of the images. Then, the prediction loss of the perception model is calculated based on the difference between the perception results and the corresponding label data of the image. The parameters of the perception model are then updated based on the prediction loss until the training termination condition is reached. In one implementation, the training termination condition can be that the training reaches a preset number of epochs. In another implementation, the training termination condition can be that the predicted loss is less than a loss threshold.

[0084] Roadside change detection models can detect changes in acquired road images. In other words, roadside change detection models can highlight areas of change in road images. Drawing on the concept of change detection in remote sensing, since roadside equipment is usually located on both sides of the road and in a fixed position, the background of the road images captured by the cameras mounted on them is relatively fixed. Therefore, moving objects such as traffic participants on the road can be quantitatively analyzed and their changes can be determined over different time periods.

[0085] In one embodiment, the roadside change detection model can be a trained neural network model, such as a Siamese network, or other types of models, such as template matching operators. In this embodiment, the specific structure and type of the roadside change detection model are not limited.

[0086] In one embodiment, the roadside change detection model can be trained using preset change detection training samples. These preset training samples can be multiple road images collected at different times by the same roadside device. Then, in chronological order, adjacent road images are used as historical and current images, respectively, and the changed areas in the current image relative to the historical images are labeled as annotation data. The preset training samples typically include multiple sets of historical and current images. The historical and current images are input into the change detection model. Through feature extraction and comparison, the change detection result is obtained, i.e., the changed areas in the current image relative to the historical images. Then, the prediction loss of the change detection model is calculated based on the difference between the change detection result and the corresponding label data of the set of images. The parameters of the change detection model are then updated based on this prediction loss until the training termination condition is met. It should be noted that existing change detection models in the remote sensing field can also be fine-tuned to obtain the roadside change detection model in this embodiment. This embodiment does not specifically limit the training method of the roadside change detection model.

[0087] In one embodiment, the cloud device can use the aforementioned preset perception training samples to train a first perception model, and then distribute the first perception model to the roadside device for deployment.

[0088] In one embodiment, the cloud device can use the aforementioned preset change detection training samples to train a roadside change detection model, and then distribute the roadside change detection model to the roadside device for deployment.

[0089] S203: The roadside equipment determines the target dataset based on the perception results of the first perception model on the road image dataset and the detection results of the roadside change detection model on the road image dataset.

[0090] The target dataset represents a set of road images where the perception results are missing.

[0091] Roadside equipment can simultaneously perform roadside perception and change detection on acquired road images. The perception results of the first perception model include the type and location of objects appearing in the road image. Typically, due to the diversity of object types and numbers in road images, the perception results often include multiple object information. The detection results of the roadside change detection model include the change area between two adjacent road images. By comparing whether there is object information corresponding to the change area in the perception results, it can be determined whether the first perception model has missed any object information when detecting objects in the road image. When the perception results obtained by using the first perception model to perceive a road image have omissions, the road image can be stored. It should be understood that the existence of omissions in the perception results indicates that the first perception model cannot accurately perceive the scene of the road image. In terms of training data, the scene of this road image appears less frequently in the training data; the scene of this road image is a long-tail data that can be used to expand the scenes of the training data. The stored set of road images is denoted as the target dataset, which can be regarded as a long-tail dataset.

[0092] S205: The roadside equipment sends the target dataset to the cloud equipment.

[0093] The roadside equipment sends the target dataset to the cloud equipment, enabling the cloud equipment to iteratively update its perception model. Optionally, the target dataset can be sent to the cloud equipment using a communication unit, transceiver, interface circuit, etc.

[0094] S207: The cloud device uses the target dataset to fine-tune and train the first perception model to obtain the second perception model, and updates the stored first perception model to the second perception model.

[0095] The cloud-based device pre-stores a first perception model. Upon receiving the target dataset sent by the roadside device, the cloud-based device retrains the first perception model using the target dataset to obtain a second perception model. It should be understood that the second perception model has the same structure as the first perception model, but different parameters. Then, the stored first perception model is replaced with the trained second perception model.

[0096] S209: The cloud device sends the second perception model to the roadside device.

[0097] Preferably, the cloud device can use Over-the-Air Technology (OTA) to enable the second sensing model to be sent and deployed to the roadside device.

[0098] S211: The roadside equipment updates the first perception model to the second perception model.

[0099] The second perception model is obtained by fine-tuning and training the first perception model using the target dataset on cloud devices. After receiving the second perception model, the roadside devices can update their perception models. Compared to the first perception model, the second perception model has higher perception capabilities because its training data includes long-tail data collected by roadside devices, making the training data more diverse and thus improving its generalization ability.

[0100] S213: Roadside equipment uses a second perception model to perform target detection on real-time acquired road images.

[0101] After the second perception model is deployed, the roadside equipment can use the second perception model to perceive the road images captured by the camera, and can perform road target detection, image segmentation or multi-task perception according to actual needs.

[0102] It should be noted that after the above steps, the second perception model can be used as the new first perception model. The target dataset can be acquired again within a preset time period according to the actual application scenario, uploaded to the cloud device for training, and the perception model can be iteratively updated to continuously optimize the performance of the perception model deployed on the roadside device.

[0103] In this embodiment, the vehicle-mounted device acquires road images and then compares the perception results of the perception model deployed on it with the detection results of the change detection model to filter out road images whose perception results are missing. This set of road images is then sent to the cloud device to further enrich the training data of the perception model. The cloud device uses the dataset sent by the vehicle-mounted device as training samples to fine-tune the perception model and then sends a new perception model to the vehicle-mounted device. This iterative update process continuously improves the road-mounted perception capability in the vehicle-road cooperative scenario.

[0104] Figure 3 This is a schematic block diagram illustrating the determination of target data provided in an embodiment of this application.

[0105] like Figure 3 As shown, the roadside equipment perceives roadside images through a deployed roadside perception model and detects regional changes in the roadside images through a deployed roadside image change detection model. The roadside perception model can be an artificial intelligence (AI) perception model used to perceive relevant information about objects contained in the road image, such as the object's category and location. Specifically, the roadside perception model can be an object detection model. It should be noted that the roadside perception model can be viewed as... Figure 2This is a concrete example of the first perception model. The roadside image change detection model can be viewed as... Figure 2 This is a specific example of a roadside change detection model. Since roadside equipment is fixed in location, changes in the area can reflect the movement of pedestrians and vehicles on the road. The targets for object detection are also pedestrians and vehicles, and moving pedestrians and vehicles, compared to stationary ones, may exhibit anomalies such as motion blur, making detection more difficult. Taking advantage of the non-moving nature of the roadside equipment, by comparing the target detection results with area changes, we can focus on situations where changes occur in the area, but the roadside perception model fails to detect the objects—cases where perception results are missed. The road images with missed perception results and the corresponding information of the missed objects can be considered as long-tail data. Figure 3 Long-tail data in the middle can be seen as Figure 2 A specific example of target data.

[0106] Figure 4 This is a flowchart illustrating the process of determining the target dataset provided in an embodiment of this application. Figure 4 This can be seen as a specific example of step S203. In one embodiment, the road image dataset includes road images at multiple times. Determining the target dataset based on the perception results of the first perception model and the detection results of the road end change detection model on the road image dataset may include the following steps:

[0107] S401: Target detection is performed on the first image using the first perception model to obtain a first perception result; the first image is a road image at time Tn of the road image dataset; the first perception result includes the object type and the region information of the corresponding area of ​​the object in the first image; n is a positive integer.

[0108] Preferably, the road images acquired by the roadside equipment include timestamps. The first image is the road image at time Tn. For example, the first image could be the road image at time T1, where T1 could be the time / moment of January 1, 2024, 17:00:00. It should be understood that the time / moment in this example is merely an illustration and is not a limitation.

[0109] The first image is input into the first perception model. After the first perception model performs target detection, the first perception result corresponding to the first image is obtained. The first perception result includes at least information about traffic participants such as vehicles and pedestrians in the first image.

[0110] S403: Target detection is performed on the second image using the first perception model to obtain a second perception result; the second image is the road image at time Tn+1 of the road image dataset; the second perception result includes the object type and the region information of the corresponding area of ​​the object in the second image.

[0111] The second image is the road image at time Tn+1. The second image is temporally adjacent to the first image, i.e., the next frame after the first image. The time difference between the second and first images can be determined based on the frame rate of the roadside camera. By comparing each frame, it is determined whether the perception results of the first perception model have any omissions. In one example, the second image could be the road image at time T2, where T2 could be the time / moment of January 1, 2024, 17:00:01. It should be understood that the time / moment in this example is merely illustrative and not a limitation.

[0112] The second image is input into the first perception model. After the first perception model performs object detection, the second perception result corresponding to the second image is obtained. Similar to the first perception result, the second perception result includes at least information about traffic participants such as vehicles and pedestrians in the second image.

[0113] S405: Process the first image and the second image using the road end change detection model to obtain the region information of the changed area of ​​the second image relative to the first image;

[0114] After the roadside camera captures the second image, the first and second images can be input into the roadside change detection model for processing to obtain the changed region in the second image relative to the first image. The region information of this changed region can include the pixel position occupied by the changed region in the second image. It should be noted that in practical applications, the number of changed regions can be one or more, or there may be no changed regions. This application does not limit the number of detected changed regions.

[0115] S407: Compare the area information of the changed area with the area information of the corresponding area of ​​the object in the second perception result to obtain the target data;

[0116] When there are multiple changing regions, for each changing region, the second perception result is filtered to see if there is region information corresponding to the object in that changing region. If the filtering result is yes, it means that the first perception model can detect the object in the changing region; if the filtering result is no, it means that the first perception model cannot detect the object in the changing region. In this case, the second image and the changing regions are jointly determined as a set of target data.

[0117] In one embodiment, road images without vehicles, pedestrians, or other traffic participants can be pre-collected as template images. Then, a road-end change detection model is used to detect change regions between the road images collected by the road-end device and the template images. Simultaneously, a first perception model is used to perform target detection on the road images collected by the road-end device, obtaining perception results. Then, the change regions and perception results are compared; that is, for each change region, the perception results are filtered to determine if there is region information corresponding to an object in that change region. If the filtering result is negative, the road image and the change region are collectively identified as a set of target data.

[0118] In one embodiment, comparing the region information of the changed region with the region information of the object corresponding region in the second perception result to obtain the target data may include: determining the overlap between the object corresponding region in the second perception result and the changed region based on the region information of the changed region and the region information of the object corresponding region in the second perception result; when the overlap is less than a preset threshold, determining the second image as the target image and the changed region as the target region; and determining the target image and the target region as the target data.

[0119] Figure 5 This is a schematic diagram illustrating the principle of determining the overlap between the corresponding region and the changing region of the object in the second perception result, as provided in the embodiments of this application.

[0120] Figure 5 (A) and Figure 5 In the diagram, (B) represents two cases: overlap greater than the preset threshold and overlap less than the preset threshold. Rectangle ABCD represents one of the changing regions, and rectangle EFGH represents the region corresponding to one of the objects in the second perception result. The preset threshold can be set to 0.5; this application does not limit the specific value of the preset threshold.

[0121] In this embodiment, the overlap can be calculated using the Intersection over Union (IOU). The IOU is the ratio of the intersection to the union of two bounding boxes; the intersection is the overlapping portion of the two bounding boxes, such as... Figure 5 (A) and Figure 5 The slashed portion of (B) in the diagram. The union of the two bounding boxes is the sum of the two bounding boxes.

[0122] like Figure 5 As shown in (A), if the intersection-union ratio between rectangle ABCD and rectangle EFGH is greater than or equal to the preset threshold, it indicates that rectangle ABCD and rectangle EFGH are the same area, and that there is an object corresponding to the changed area in the second perception result at this time.

[0123] like Figure 5 As shown in (B), if the intersection-union ratio (IUGR) between rectangles ABCD and EFGH is less than a preset threshold, it indicates that rectangles ABCD and EFGH are not the same region. If the overlap of the corresponding regions of objects in the second perception result is less than the preset threshold, it means that the first perception model did not detect objects in the changing region of the second image. At this time, the second image is determined as the target image, and the changing region is determined as the target region; then the target image and the target region are determined as the target data.

[0124] S409: When the target data meets the preset conditions, the set of target data is determined as the target dataset.

[0125] In one embodiment, the preset condition can be a preset time period, that is, within the preset time period, the determined set of target data is packaged into a target dataset. The preset time period can be one month. This application does not limit the specific length of the preset time period and can set it according to actual application needs.

[0126] In one embodiment, the preset condition can also be a preset quantity, that is, when the target data in the set of target data accumulates to a preset quantity, the set of target data is determined as the target dataset. This preset quantity can be 1000. This application does not limit the specific value of the preset quantity and it can be set according to actual application needs.

[0127] In one embodiment, the method of this application may further include: compressing the target dataset to obtain a compressed target dataset; desensitizing the compressed dataset to obtain a desensitized target dataset; encrypting the desensitized dataset to obtain an encrypted target dataset; and sending the encrypted target dataset to a cloud device.

[0128] Figure 6 This is a schematic diagram of the process by which a cloud device uses a target dataset to fine-tune and train a first perception model, as provided in an embodiment of this application. Figure 6 This can be seen as a specific example of step S207.

[0129] S601: Label the target dataset to obtain labeled data;

[0130] The target dataset includes road images and information on areas where no objects were detected. In one implementation, a cloud-based annotation model can be used to automatically annotate each road image in the target dataset. Alternatively, based on the automatic annotation, labels can be manually added to the corresponding areas of the road image information to obtain labeled data.

[0131] S603: Using the labeled data, fine-tune and train the first perception model to obtain the second perception model.

[0132] Using the aforementioned labeled data, the first perception model stored on the cloud device is fine-tuned and trained. During training, road images from the target dataset can be input into the first perception model to obtain image perception results. Then, the prediction loss of the first perception model is calculated based on the difference between the perception results and the labeled data. The parameters of the perception model are then updated based on this prediction loss until the training termination condition is met, allowing the parameters of the perception model to adapt to the target dataset. This fine-tuning training method uses the parameters of the first perception model as initialization parameters for backpropagation of the loss. In another implementation, some parameters of the first perception model can be frozen, and only the remaining parameters are adjusted.

[0133] Compared to retraining using large datasets, fine-tuning training methods are faster, and the proposed solution utilizes the target dataset uploaded by roadside devices, making it more targeted and able to accurately improve the perception capabilities of the perception model.

[0134] Figure 7 This is a schematic diagram of the interactive process for generating version information provided in an embodiment of this application.

[0135] S701: The roadside device sends the version information of the first perception model to the cloud device.

[0136] After sending the target dataset, the version information of the currently deployed first-sensory model can be sent to the cloud device. This application does not specify a particular format for representing the version information; for example, the version of the first-sensory model can be represented by a number.

[0137] S703: After the cloud device fine-tunes and trains the first perception model using the target dataset to obtain the second perception model, it generates the version information of the second perception model based on the version information of the first perception model.

[0138] After receiving the second sensing model, the cloud device generates version information for the second sensing model. This version information is determined by adding one to the version information of the first sensing model. This step can be performed before the cloud device sends the second sensing model to the roadside device.

[0139] S705: The cloud device sends the version information of the second perception model to the roadside device.

[0140] By sending version information of the second sensing model, the cloud device can notify the roadside device to receive the second sensing model that the cloud device is about to send. In one embodiment, after receiving the version information of the second sensing model, the roadside device can send a confirmation message to the cloud device. After receiving the confirmation message from the roadside device, the cloud device then sends the second sensing model to the roadside device.

[0141] S707: Deploy the second perception model received from the cloud device based on the version information of the second perception model.

[0142] After receiving the version information of the second perception model sent by the cloud device, it can first check whether the version information of the second perception model is higher than that of the first perception model. If the detection result is yes, the received second perception model can be deployed.

[0143] In this embodiment of the application, by transmitting version information between the roadside device and the cloud device, it can be ensured that the version of the perception model stored in the cloud device is always higher than or equal to the version of the perception model deployed in the roadside device, thereby realizing the update and iteration of the roadside and cloud perception models.

[0144] This application also provides a sensing system. The system includes roadside equipment and cloud-based equipment;

[0145] The roadside device is used to acquire road images to obtain a road image dataset; and to determine a target dataset based on the perception results of the first perception model on the road image dataset and the detection results of the roadside change detection model on the road image dataset; the target dataset represents the set of road images in the road image dataset for which the perception results of the first perception model are missing; and to send the target dataset to a cloud device; and to receive a second perception model sent by the cloud device and update the first perception model to the second perception model; and to perform target detection on the real-time acquired road images using the second perception model;

[0146] The cloud device is configured to receive a target dataset sent by the roadside device; fine-tune and train the first perception model using the target dataset to obtain a second perception model, update the stored first perception model to the second perception model; and send the second perception model to the roadside device.

[0147] Figure 8 This is a schematic diagram of a sensing system framework provided in the embodiments of this application. This sensing system can be regarded as a specific example of the sensing system provided in the above embodiments.

[0148] like Figure 8As shown, the system includes a roadside device 810, a cloud device 820, and a deployment module 830. The roadside device 810 includes a data acquisition module 811 and a data transmission module 812. The data acquisition module 811 includes a first perception model, a roadside change detection model, and a data storage program. Based on the comparison between the perception results of the first perception model and the detection results of the roadside change detection model, the image storage program is triggered to store and process the acquired road images to obtain the target dataset. The data transmission module 812 is used to transmit the target dataset and perception version information. The target dataset includes image data and spatiotemporal data, where the spatiotemporal data consists of the timestamps and location data corresponding to the image data. The cloud device 820 includes a data annotation and model training module 821 and an update module 822. The data annotation and model training module 821 is used to perform long-tail data annotation and cloud model training. The update module 822 is used to save the second perception model trained by the data annotation and model training module 821 in the cloud device 820 and replace the previously stored first perception model. Deployment module 830 is used to use the second perception model as a new roadside perception model and distribute it to the roadside devices. Deployment module 830 can be a module that uses OTA (Over-The-Air) technology for model deployment. The roadside devices send the collected dataset to the cloud devices, which then train on the original version of the perception model. Iterative updates can be used to upgrade and optimize the roadside perception model, improving the perception capabilities of the roadside devices.

[0149] As can be seen from the technical solutions provided in the embodiments of this application above, in these embodiments, the vehicle-side device collects road images, and then compares the perception results of the perception model deployed on it with the detection results of the change detection model to filter out road images whose perception results are missed by the perception model. The set of these road images is then sent to the cloud device. The cloud device uses the dataset sent by the road-side device as training samples to fine-tune and train the perception model, and then sends a new perception model to the road-side device. Thus, during the iterative update of the perception model, the road-side perception capability in the vehicle-road cooperative scenario is continuously improved. Furthermore, by focusing on data of object types not detected by the perception model but appearing on the road, long-tail data is obtained as the target dataset to further enrich the training data of the perception model.

[0150] Figure 9 This is a schematic diagram of the structure of the roadside equipment provided in the embodiments of this application.

[0151] like Figure 9 As shown, the roadside equipment 900 may include:

[0152] Acquisition module 901 is used to acquire road images and obtain a road image dataset;

[0153] The processing module 902 is used to determine the target dataset based on the perception results of the first perception model on the road image dataset and the detection results of the road end change detection model on the road image dataset; the target dataset is used to represent the set of road images where the perception results are missing.

[0154] Sending module 903 is used to send the target dataset to the cloud device;

[0155] The receiving module 904 is used to receive the second perception model of the cloud device and update the first perception model to the second perception model; the second perception model is obtained by the cloud device through fine-tuning and training the first perception model using the target dataset;

[0156] The perception module 905 is used to perform target detection on real-time acquired road images using the second perception model.

[0157] In some embodiments, the road image dataset includes road images at multiple times, and the processing module may include:

[0158] The first perception result determination submodule is used to perform object detection on the first image using the first perception model to obtain the first perception result; the first image is a road image at time Tn of the road image dataset; the first perception result includes the object type and the region information of the corresponding area of ​​the object in the first image; n is a positive integer.

[0159] The second perception result determination submodule is used to perform target detection on the second image using the first perception model to obtain the second perception result; the second image is the road image at time Tn+1 of the road image dataset; the second perception result includes the object type and the region information of the corresponding area of ​​the object in the second image.

[0160] The change detection submodule is used to process the first image and the second image using the roadside change detection model to obtain the region information of the changed area in the second image relative to the first image.

[0161] The target data determination submodule is used to compare the regional information of the changed area with the regional information of the corresponding area of ​​the object in the second perception result to obtain the target data;

[0162] The target dataset determination submodule is used to determine the set of target data as the target dataset when the target data meets preset conditions.

[0163] In some embodiments, the target data determination submodule includes:

[0164] The overlap unit is used to determine the overlap between the object-corresponding region in the second perception result and the changed region based on the region information of the changed region and the region information of the object-corresponding region in the second perception result.

[0165] The target region determination unit is used to determine the second image as the target image and the changed region as the target region when the overlap is less than a preset threshold.

[0166] The target data determination unit is used to determine the target image and target region as target data.

[0167] In some embodiments, the target dataset determination submodule includes:

[0168] The quantity accumulation unit is used to determine the set of target data as the target dataset when the target data in the target data set accumulates to a preset quantity.

[0169] In some embodiments, the roadside device 900 further includes: a first sending module, configured to send version information of the first sensing model to a cloud device; a second receiving module, configured to receive version information of the second sensing model sent by the cloud device; and a deployment module, configured to deploy the received second sensing model sent by the cloud device according to the version information of the second sensing model.

[0170] In some embodiments, the roadside device 900 further includes: a compression module for compressing the target dataset to obtain a compressed target dataset; a desensitization module for desensitizing the compressed dataset to obtain a desensitized target dataset; an encryption module for encrypting the desensitized dataset to obtain an encrypted target dataset; and a third sending module for sending the encrypted target dataset to a cloud device.

[0171] Figure 10 This is a schematic diagram of the structure of the cloud device provided in the embodiments of this application.

[0172] like Figure 10 As shown, the cloud device 1000 stores a first perception model. The cloud device 1000 may include: a receiving module 1001, used to receive a target dataset sent by a roadside device; the roadside device is deployed with a first perception model and a roadside change detection model, the roadside device is used to collect road images to obtain a road image dataset; and determines a target dataset based on the perception results of the first perception model on the road image dataset and the detection results of the roadside change detection model on the road image dataset; the target dataset represents the set of road images in the road image dataset where the perception results of the first perception model are missing; and a processing module 1002, used to fine-tune and train the first perception model using the target dataset to obtain a second perception model, and update the stored first perception model to the second perception model.

[0173] The sending module 1003 is used to send the second sensing model to the roadside equipment.

[0174] In some embodiments, the processing module may include: an annotation submodule for annotating the target dataset to obtain annotated data; and a training submodule for fine-tuning the first perceptual model using the annotated data to obtain a second perceptual model.

[0175] The apparatus and method embodiments described herein are based on the same inventive concept.

[0176] This application provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the method provided in the above method embodiments.

[0177] Embodiments of this application also provide a computer storage medium, which can be disposed in a terminal to store at least one instruction or at least one program related to implementing a method as provided in the above method embodiments, wherein the at least one instruction or at least one program is loaded and executed by the processor to implement the method provided in the above method embodiments.

[0178] Embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the above-described method embodiments.

[0179] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0180] The memory described in this application embodiment can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for the functions, etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0181] The methods provided in this application can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Taking running on a server as an example... Figure 11 This is a hardware structure block diagram of a server for a roadside sensing method provided in an embodiment of this application. For example... Figure 11 As shown, server 1100 can be considered a specific example of an electronic device. Server 1100 can vary significantly due to different configurations or performance, and may include one or more Central Processing Units (CPUs) 1110 (CPUs 1110 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 1130 for storing data, and one or more storage media 1120 (e.g., one or more mass storage devices) for storing application programs 1123 or data 1122. The memory 1130 and storage media 1120 can be temporary or persistent storage. The program stored in storage media 1120 may include one or more modules, each module including a series of instruction operations on the server. Furthermore, CPU 1110 may be configured to communicate with storage media 1120 and execute the series of instruction operations in storage media 1120 on server 1100. Server 1100 may also include one or more power supplies 1160, one or more wired or wireless network interfaces 1150, one or more input / output interfaces 1140, and / or one or more operating systems 1121, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0182] The input / output interface 1140 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 1100. In one example, the input / output interface 1140 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 1140 may be a radio frequency (RF) module for wireless communication with the Internet.

[0183] Those skilled in the art will understand that Figure 11 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 1100 may also include... Figure 11 The more or fewer components shown, or having the same Figure 11 The different configurations shown.

[0184] As can be seen from the embodiments of the roadside perception method, device, and storage medium provided in this application, in these embodiments, by comparing the perception results of the perception model deployed thereon with the detection results of the change detection model, road images whose perception results are missed by the perception model are filtered out. A set of these road images is then sent to a cloud device. The cloud device uses the dataset sent by the roadside device as training samples to fine-tune and train the perception model, and then sends a new perception model to the roadside device. Thus, during the iterative update of the perception model, the roadside perception capability in the vehicle-road cooperative scenario is continuously improved. Furthermore, by focusing on data of object types not detected by the perception model but appearing on the road, long-tail data is obtained as the target dataset to further enrich the training data of the perception model.

[0185] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0186] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0187] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer storage medium, such as a read-only memory, a disk, or an optical disk.

[0188] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for roadside sensing, characterized in that, include: Road images were collected at multiple times to obtain a road image dataset; Obtain the perception results of the first perception model on the road image dataset and the detection results of the road end change detection model on the road image dataset; The perception results of the first perception model include the type and location of objects appearing on the road image; the detection results of the road end change detection model include the change area in two adjacent road images. By comparing the perception results with the object information corresponding to the changed areas, it is determined whether the first perception model has missed any object information when detecting objects in the road image; If there is missing object information, the road image is determined to be a road image with missing information, and the target dataset is stored; the target dataset is used to represent the set of road images with missing information in the perception results; Send the target dataset to the cloud device; Receive the second perception model sent by the cloud device, and update the first perception model to the second perception model; The second perception model is obtained by the cloud device through fine-tuning and training the first perception model using the target dataset; The second perception model is used to perform target detection on real-time acquired road images; The fine-tuning training process includes: using the parameters of the first perception model as the initialization parameters of the model, inputting the road images included in the target dataset into the model, calculating the model's prediction loss based on the difference between the model output and the labeled data, updating the model's parameters based on the prediction loss, until the training termination condition is met, and determining the model obtained after training termination as the second perception model.

2. The method according to claim 1, characterized in that, The method further includes: The first perception model is used to perform object detection on the first image to obtain a first perception result; the first image is a road image at time Tn of the road image dataset; the first perception result includes the object type and the region information of the corresponding area of ​​the object in the first image; n is a positive integer. The first perception model is used to perform target detection on the second image to obtain a second perception result; the second image is the road image at time Tn+1 of the road image dataset; the second perception result includes the object type and the region information of the corresponding area of ​​the object in the second image. The first image and the second image are processed using the road end change detection model to obtain the region information of the changed area in the second image relative to the first image. The target data is obtained by comparing the area information of the changed area with the area information of the corresponding area of ​​the object in the second perception result. When the target data meets the preset conditions, the set of target data is determined as the target dataset.

3. The method according to claim 2, characterized in that, The step of comparing the region information of the changed region with the region information of the corresponding region of the object in the second perception result to obtain target data includes: Based on the region information of the changed region and the region information of the object corresponding region in the second perception result, the overlap between the object corresponding region in the second perception result and the changed region is determined. When the overlap is less than a preset threshold, the second image is determined as the target image, and the changed region is determined as the target region; The target image and the target region are defined as target data.

4. The method according to claim 2, characterized in that, The step of determining the set of target data as the target dataset when the target data meets preset conditions includes: When the target data in the target data set accumulates to a preset quantity, the target data set is determined as the target dataset.

5. The method according to claim 1, characterized in that, The method further includes: Send the version information of the first perception model to the cloud device; Receive the version information of the second perception model sent by the cloud device; Based on the version information of the second perception model, deploy the second perception model sent by the cloud device.

6. The method according to claim 1, characterized in that, The method further includes: The target dataset is compressed to obtain the compressed target dataset; The compressed target dataset is de-identified to obtain the de-identified target dataset; The de-identified target dataset is encrypted to obtain the encrypted target dataset. The encrypted target dataset is sent to the cloud device.

7. A method for roadside sensing, characterized in that, Applied to a cloud device, wherein the cloud device stores a first perception model, the method includes: The system receives a target dataset sent by a roadside device. The roadside device collects road images at multiple times to obtain a road image dataset. It also acquires the perception results of a first perception model and the detection results of a roadside change detection model on the road image dataset. The perception results of the first perception model include the type and location of objects appearing in the road images. The detection results of the roadside change detection model include the change areas in two adjacent road images. The roadside device further compares the perception results with the object information corresponding to the change areas to determine whether the first perception model has missed any object information when detecting objects in the road images. If there is any missed object information, the road image is determined to be a road image with missing information and stored to obtain the target dataset. The target dataset represents the set of road images for which the perception results show missing information. The first perception model is fine-tuned and trained using the target dataset to obtain the second perception model, and the stored first perception model is updated to the second perception model. Send the second sensing model to the roadside equipment; The fine-tuning training process includes: using the parameters of the first perception model as the initialization parameters of the model, inputting the road images included in the target dataset into the model, calculating the model's prediction loss based on the difference between the model output and the labeled data, updating the model's parameters based on the prediction loss, until the training termination condition is met, and determining the model obtained after training termination as the second perception model.

8. The method according to claim 7, characterized in that, The method further includes: Receive the version information of the first sensing model sent by the roadside device; After fine-tuning and training the first perception model using the target dataset to obtain the second perception model, the version information of the second perception model is generated based on the version information of the first perception model. The version information of the second sensing model is sent to the roadside equipment.

9. A road-end device, characterized in that, The roadside equipment includes: The acquisition module is used to acquire road images at multiple times to obtain a road image dataset; The processing module is used to acquire the perception results of the first perception model on the road image dataset and the detection results of the road end change detection model on the road image dataset; the perception results of the first perception model include the type and location of objects appearing on the road image; the detection results of the road end change detection model include the change area of ​​two road images at adjacent time points; comparing the perception results and the object information corresponding to the change area, it is determined whether the first perception model has missed object information when detecting objects in the road image; if there is missed object information, the road image is determined to be a road image with missing information, and the target dataset is stored; the target dataset is used to represent the set of road images with missing information in the perception results; The sending module is used to send the target dataset to the cloud device; A receiving module is configured to receive a second perception model sent by the cloud device and update the first perception model to the second perception model; the second perception model is obtained by the cloud device through fine-tuning training of the first perception model using the target dataset; the fine-tuning training process includes: using the parameters of the first perception model as the initialization parameters of the model, inputting the road images included in the target dataset into the model, calculating the prediction loss of the model based on the difference between the model output and the labeled data, updating the parameters of the model based on the prediction loss, until the training termination condition is met, and determining the model obtained after training termination as the second perception model; The perception module is used to perform target detection on real-time acquired road images using the second perception model.

10. A cloud device, characterized in that, The cloud device stores a first perception model, and the cloud device includes: A receiving module is used to receive a target dataset sent by a roadside device. The roadside device is equipped with a first perception model and a roadside change detection model. The roadside device is used to collect road images at multiple times to obtain a road image dataset. It also acquires the perception results of the first perception model and the detection results of the roadside change detection model on the road image dataset. The perception results of the first perception model include the type and location of objects appearing in the road images. The detection results of the roadside change detection model include the change areas in two adjacent road images. By comparing the perception results and the object information corresponding to the change areas, it is determined whether the first perception model has missed any object information when detecting objects in the road images. If there is any missed object information, the road image is determined to be a road image with missing information and is stored to obtain the target dataset. The target dataset represents the set of road images for which the perception results show missing information. The processing module is used to fine-tune the first perception model using the target dataset to obtain a second perception model, and update the stored first perception model to the second perception model. The fine-tuning training process includes: using the parameters of the first perception model as the initialization parameters of the model, inputting the road images included in the target dataset into the model, calculating the prediction loss of the model based on the difference between the model output and the labeled data, updating the parameters of the model based on the prediction loss, until the training termination condition is met, and determining the model obtained after training termination as the second perception model. The sending module is used to send the second sensing model to the roadside equipment.

11. A sensing system, characterized in that, The system includes roadside equipment and cloud-based equipment; The roadside device is used to acquire road images at multiple times to obtain a road image dataset; and to acquire the perception results of the first perception model on the road image dataset and the detection results of the roadside change detection model on the road image dataset. The perception results of the first perception model include the type and location of objects appearing on the road image; the detection results of the road end change detection model include the change area in two adjacent road images. By comparing the perception results with the object information corresponding to the changed areas, it is determined whether the first perception model has missed any object information when detecting objects in the road image; if there is any missing object information, the road image is determined to be a road image with missing information, and the target dataset is stored; the target dataset is used to represent the set of road images with missing information in the perception results. And send the target dataset to the cloud device; The system also receives a second perception model sent by the cloud device and updates the first perception model to the second perception model; and uses the second perception model to perform target detection on real-time acquired road images. The cloud device is used to receive the target dataset sent by the roadside device; And by using the target dataset to fine-tune and train the first perception model to obtain the second perception model, and updating the stored first perception model to the second perception model; The process of fine-tuning training includes sending the second perception model to the roadside device, wherein the parameters of the first perception model are used as the initialization parameters of the model, the road images included in the target dataset are input into the model, the prediction loss of the model is calculated based on the difference between the model output and the labeled data, the parameters of the model are updated based on the prediction loss, until the training end condition is met, and the model obtained after training is determined as the second perception model.

12. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the method as claimed in any one of claims 1-6 or 7-8.

Images