Training method and device of traffic recognition model, equipment, storage medium and product
By updating the traffic identification model using multiple sensor data on edge devices, the problem of existing traffic identification models adapting to the specificity of different road sections is solved, and the identification accuracy and training efficiency are improved.
Patent Information
- Application Number
- CN202310609082.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-11-26
AI Technical Summary
Due to unified training, existing traffic recognition models are difficult to adapt to the specificity of different road sections, resulting in low recognition accuracy.
By acquiring the initial traffic identification model on the edge device, using multiple sensors to collect data, identify and update the model, so that each edge device can customize its model based on its own road segment data.
The accuracy and training efficiency of the traffic identification model are improved, so that the model can better adapt to the characteristics of each road section and enhance the detection performance.
Smart Images

Figure CN119027886A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a training method, device, electronic device, computer-readable storage medium and computer program product for a traffic recognition model. Background Art
[0002] The first prerequisite for smart transportation is traffic perception on the road. Traffic perception plays a role in the digitization of road information. It digitizes physically existing vehicles, roads, and people through artificial intelligence (AI) intelligent learning detection, and represents road elements and road participants in the form of information. In the related technologies, most of the traffic perception technical solutions build points at fixed intervals on the road, and then establish intelligent sensing equipment such as cameras and radars at the corresponding points to perceive and cover the roads. For each covered road section, the road information captured by the sensors at the points will be transmitted to the mobile edge computing (MEC), so as to carry out a prediction process based on the traffic recognition model.
[0003] However, for the above solution, the MEC at each point uses the same traffic recognition model, that is, the detection performance of the MEC at each point is completely dependent on the same traffic recognition model issued by the center and obtained through unified data training on the center side. However, the conditions of different traffic sections are different. Therefore, using the above traffic recognition model to identify road information will have greater performance limitations, resulting in lower accuracy of traffic recognition results. Summary of the invention
[0004] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for training a traffic recognition model, which can improve the accuracy of traffic recognition performed by the traffic recognition model.
[0005] The technical solution of the embodiment of the present application is implemented as follows:
[0006] The embodiment of the present application provides a training method for a traffic recognition model, which is applied to a traffic sensing system. The traffic sensing system includes a plurality of edge devices, each of which is dispersedly arranged at the edge of a road, and different edge devices correspond to different road sections. The method includes:
[0007] The edge device acquires an initial traffic recognition model, where the initial traffic recognition model is used to recognize traffic objects on the road and is trained based on first traffic image data collected by each edge device for a corresponding road section;
[0008] Capturing images of the current road section by using at least two different types of sensors to obtain at least two pieces of second traffic image data;
[0009] Recognize each of the second traffic image data respectively by using the initial traffic recognition model to obtain a recognition result corresponding to each of the second traffic image data;
[0010] When the recognition results corresponding to the second traffic image data are inconsistent, the initial traffic recognition model is updated based on the at least two second traffic image data to obtain a target traffic recognition model.
[0011] The embodiment of the present application provides a training device for a traffic recognition model, which is applied to a traffic perception system. The traffic perception system includes a plurality of edge devices, each of which is dispersedly arranged at the edge of a road, and different edge devices correspond to different road sections. The device includes:
[0012] An acquisition module is used for the edge device to acquire an initial traffic recognition model, where the initial traffic recognition model is used to recognize traffic objects on the road and is trained based on first traffic image data collected by each edge device for the corresponding road section;
[0013] A collection module, used for collecting images of the current road section through at least two different types of sensors to obtain at least two second traffic image data;
[0014] A recognition module, used to recognize each of the second traffic image data respectively through the initial traffic recognition model to obtain a recognition result corresponding to each of the second traffic image data;
[0015] The updating module is used to update the initial traffic recognition model based on the at least two second traffic image data to obtain a target traffic recognition model when the recognition results corresponding to the second traffic image data are inconsistent.
[0016] In the above scheme, the device also includes a coordinate system conversion module, which is used to perform the following processing for the recognition results corresponding to each of the second traffic image data: obtaining the coordinates of the traffic object indicated by the recognition result in the coordinate system where the sensor corresponding to the second traffic image data is located; performing coordinate conversion on the coordinates of the traffic object to obtain the target coordinates of the traffic object in the target coordinate system; determining the target traffic object of the recognition result in the target coordinate system based on the target coordinates of the traffic object in the target coordinate system; when the target traffic objects corresponding to each of the recognition results are inconsistent, determining that the recognition results corresponding to each of the second traffic image data are inconsistent.
[0017] In the above scheme, the updating module is also used to determine the label of the corresponding second traffic image data based on each of the recognition results, and to annotate the corresponding second traffic image data based on the label to obtain an enhanced training sample of the initial traffic recognition model; based on the enhanced training sample, the initial traffic recognition model is updated to obtain a target traffic recognition model.
[0018] In the above scheme, the update module is also used to perform the following processing for each of the recognition results: fuse at least one other recognition result with the recognition result to obtain at least one fused recognition result; wherein the other recognition result is a recognition result other than the recognition result among the recognition results corresponding to the at least two second traffic image data; select a target recognition result from at least one of the fused recognition results, and determine the target recognition result as a label of the corresponding second traffic image data.
[0019] In the above scheme, the device also includes a fusion module, which is used to project each of the other recognition results to the coordinate system where the recognition result is located to obtain at least one standard recognition result; the update module is also used to fuse the at least one standard recognition result with the recognition result to obtain at least one fused recognition result.
[0020] In the above scheme, the update module is also used to obtain the confidence of each fused recognition result when there are multiple fused recognition results; and select the fused recognition result whose confidence is greater than the confidence threshold from the multiple fused recognition results as the target recognition result.
[0021] In the above scheme, the update module is also used to obtain the confidence of each fused recognition result when there are multiple fused recognition results; sort the confidences to obtain the sorting results: based on the sorting results, select the fused recognition result with the largest confidence from the multiple fused recognition results as the target recognition result.
[0022] In the above scheme, the at least two different types of sensors include at least a radar and a camera, and the acquisition module is also used to acquire images of the current road section through the radar to obtain point cloud image data, and to acquire images of the current road section through the camera to obtain plane image data; the point cloud image data and the plane image data are used as the at least two second traffic image data.
[0023] In the above scheme, the update module is also used to obtain the resource consumption of the edge device; in response to the resource consumption of the edge device being less than the resource consumption threshold, the initial traffic recognition model is updated based on the at least two second traffic image data to obtain the target traffic recognition model.
[0024] In the above scheme, the recognition module is also used to perform the following processing on each second traffic image data: through the initial traffic recognition model, the second traffic image data is continuously downsampled at least twice to obtain a downsampled image; the downsampled image is upsampled to obtain an upsampled image, and the upsampled image has the same image size as the downsampled image; the upsampled image is fused with the downsampled image to obtain a fused image, and based on the fused image, the second traffic image data is recognized to obtain a recognition result corresponding to the second traffic image data.
[0025] In the above scheme, the device also includes an application module, which is used to collect images of the road section corresponding to the edge device to obtain at least one traffic image data to be identified; through the target traffic recognition model, each of the traffic image data to be identified is respectively identified to obtain the recognition result corresponding to each of the traffic image data to be identified; based on the recognition result corresponding to each of the traffic image data to be identified, the traffic objects in the corresponding road section and the traffic information of the traffic objects are determined; wherein the traffic information includes at least one of the following: the category, form, identification, position, speed and angle of the traffic object.
[0026] An embodiment of the present application provides an electronic device, including:
[0027] A memory for storing executable instructions;
[0028] The processor is used to implement the training method of the traffic recognition model provided in the embodiment of the present application when executing the executable instructions stored in the memory.
[0029] An embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor will execute the training method of the traffic recognition model provided by the embodiment of the present application.
[0030] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the training method of the traffic recognition model provided in the embodiment of the present application.
[0031] The embodiments of the present application have the following beneficial effects:
[0032] After the edge devices distributed in different sections obtain the same initial traffic recognition model, the edge devices of each section use at least two different types of sensors to collect images of the current section, and then use the initial traffic recognition model to respectively recognize the collected second traffic image data to obtain the recognition results corresponding to each second traffic image data; when the recognition results corresponding to each second traffic image data are inconsistent, the initial traffic recognition model is updated based on at least two second traffic image data to obtain the target traffic recognition model. In this way, each edge device collects the second traffic image data of its own section, and based on the recognition results of each second traffic image data, determines the training data for updating the initial traffic recognition model from multiple second traffic image data to update the acquired initial traffic recognition model, so that the initial traffic recognition model of each edge device can be updated according to the traffic image data of the section where it is located, thereby enhancing the training effect and training efficiency of the model and improving the accuracy of traffic recognition by the traffic recognition model. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 1 is a schematic diagram of the architecture of a training system 100 for a traffic recognition model provided in an embodiment of the present application;
[0034] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0035] Figure 3 It is a flowchart of a training method for a traffic recognition model provided in an embodiment of the present application;
[0036] Figure 4 This is a schematic diagram of the dispersed locations of edge devices provided in an embodiment of the present application;
[0037] Figure 5 is a schematic diagram of a recognition process of second traffic image data provided by an embodiment of the present application;
[0038] Figure 6 is a schematic diagram of the model structure of the initial traffic recognition model provided in an embodiment of the present application;
[0039] Figure 7 is a flowchart of a process of updating an initial traffic recognition model based on at least two second traffic image data to obtain a target traffic recognition model provided by an embodiment of the present application;
[0040] Figure 8 It is a flowchart of the training of the traffic recognition model provided in the embodiment of the present application;
[0041] Fig. 9 This is a system architecture diagram of the edge data mining and incremental learning method in the traffic perception scenario provided by the embodiment of the present application;
[0042] Fig.10 It is a technical architecture diagram of the sample mining process provided by the embodiment of the present application;
[0043] Fig.11 It is a schematic diagram of a joint training process based on an incremental sample set provided in an embodiment of the present application. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.
[0045] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0046] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.
[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0048] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.
[0049] 1) Client, also known as the user end, refers to the program corresponding to the server that provides local services to users. Except for some applications that can only run locally, it is generally installed on an ordinary client and needs to cooperate with the server to run, that is, it requires corresponding servers and service programs in the network to provide corresponding services. In this way, specific communication connections need to be established on the client and server sides to ensure the normal operation of the application.
[0050] 2) Edge and edge computing, in traffic scenarios, specifically refer to a computing method that is different from the center and central computing. It is a computing mode that is sunk to the road. Generally, edge computing units or edge devices are located on the road.
[0051] 3) Mobile Edge Computing (MEC): It refers specifically to computing devices in traffic scenarios. In general, in terms of physical form, it is deployed at each point (pole) or in an edge computer room (such as a toll station computer room).
[0052] 4) In response, it is used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order of execution of the multiple operations executed.
[0053] 5) Incremental learning refers to updating the model by learning new data based on the existing model.
[0054] 6) Data mining refers to discovering and utilizing valuable data based on existing data in the model operation scenario.
[0055] 7) Point: In traffic scenarios, it refers to a location where sensor equipment is installed, such as a signal control pole, gantry or other high-level location. If a sensor is installed there for perception, it is called a point.
[0056] See also Figure 1 , Figure 1 It is a schematic diagram of the architecture of the training system 100 for the traffic recognition model provided in an embodiment of the present application. To realize the application scenario of the training of the traffic recognition model, the terminal (the terminal 400 is shown as an example) is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The terminal 400 is used for the user to use the client 401, which is displayed on the display interface (the display interface 401-1 is shown as an example). The terminal 400 and the server 200 are connected to each other via a wired or wireless network.
[0057] The server 200 is used to send the initial traffic recognition model to the terminal 400;
[0058] The terminal 400 is used to receive the initial traffic recognition model sent by the server 200; the edge device obtains the initial traffic recognition model, which is used to recognize traffic objects on the road and is trained based on the first traffic image data collected by each edge device for the corresponding road section; the image of the current road section is collected by at least two different types of sensors to obtain at least two second traffic image data; the second traffic image data are respectively recognized by the initial traffic recognition model to obtain the recognition results corresponding to the second traffic image data; when the recognition results corresponding to the second traffic image data are inconsistent, the initial traffic recognition model is updated based on at least two second traffic image data to obtain the target traffic recognition model
[0059] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Deliver Network), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a set-top box, an intelligent voice interaction device, a smart home appliance, a virtual reality device, a vehicle-mounted terminal, an aircraft, and a mobile device (e.g., a mobile phone, a portable music player, a personal digital assistant, a dedicated messaging device, a portable gaming device, an intelligent speaker, and a smart watch), etc., but is not limited thereto. The terminal device and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.
[0060] The electronic device (ie, edge device) for implementing the training method of the traffic recognition model of the embodiment of the present application is described. Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. In practical applications, the electronic device can be Figure 1 The server 200 or the terminal 400 shown in FIG. 4 is a terminal. Figure 2 , Figure 2 The electronic device shown includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 440 is not shown in FIG. Figure 2Various buses are labeled as bus system 440 .
[0061] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.
[0062] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.
[0063] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.
[0064] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.
[0065] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.
[0066] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;
[0067] A network communication module 452, used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;
[0068] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);
[0069] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.
[0070] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A training device 455 of a traffic recognition model stored in a memory 450 is shown, which may be software in the form of a program or a plug-in, and includes the following software modules: an acquisition module 4551, a collection module 4552, an identification module 4553, and an update module 4554. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.
[0071] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the training device for the traffic recognition model provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the training method for the traffic recognition model provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.
[0072] In some embodiments, the terminal or server can implement the training method of the traffic recognition model provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as an instant messaging APP, a web browser APP; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.
[0073] Based on the above description of the training system and electronic device (edge device) of the traffic recognition model provided in the embodiment of the present application, the following describes the training method of the traffic recognition model provided in the embodiment of the present application. In actual implementation, the training method of the traffic recognition model provided in the embodiment of the present application can be implemented by a terminal or a server alone, or by a terminal and a server in collaboration. Taking the edge device as an example, the training method of the traffic recognition model provided in the embodiment of the present application is described. Figure 3 , Figure 3 is a flow chart of the training method of the traffic recognition model provided in the embodiment of the present application. Figure 3 The steps shown are explained.
[0074] Step 101: The edge device obtains an initial traffic recognition model, which is used to recognize traffic objects on the road and is trained based on first traffic image data collected by each edge device for the corresponding road section.
[0075] It should be noted that the training method of the traffic recognition model provided in the embodiment of the present application is applied to the traffic sensing system. The traffic sensing system includes multiple edge devices. Each edge device is dispersedly arranged at the edge of the road. Different edge devices operate independently of each other, and different edge devices correspond to different road sections. They are physically installed at each point of a pole and are responsible for the reasoning of the image data collected by the sensors (cameras, radars, etc.) installed on the road section. For example, for a 100-kilometer section of road, there is one edge device every 1 kilometer (or 500 meters), then this section of road has 100 edge devices (or 200), and each edge device can detect the road conditions within the field of view of the section where the edge device is located, such as vehicle detection, so as to detect the road conditions of the entire section of the road based on these edge devices. For example, see Figure 4 , Figure 4 This is a schematic diagram of the dispersed locations of edge devices provided in an embodiment of the present application, based on Figure 4 ,against Figure 4 For a 10-kilometer road, if there is an edge device every 500 meters, then there are 20 edge devices on this section of road, among which the dashed boxes 401, 402, 403 and 404 are the four edge devices shown.
[0076] In actual implementation, the traffic perception system also includes a central cloud device (which can be a server). The central cloud device here is used to train an initial traffic recognition model based on the first traffic image data collected by each edge device for the corresponding road section, and then send the trained initial traffic recognition model to each edge device. Specifically, the central cloud device obtains the first traffic image data. Here, when the central cloud device receives the traffic image data collected by each edge device for the corresponding road section, it uniformly determines these traffic image data as the first traffic image data, and then annotates the first traffic image data to obtain a full sample data set, thereby training the traffic recognition model based on the full sample data set to obtain an initial traffic recognition model, and then using this model as a base model and distributing it to each edge device.
[0077] It should be noted that the full sample data set can also be pre-stored annotated traffic image data. Specifically, after the central cloud device obtains the pre-stored traffic image data, it annotates these traffic image data to obtain the full sample data set, and then trains the traffic recognition model based on the full sample data set to obtain the initial traffic recognition model. Here, the traffic image data used as the basis for the initial traffic recognition model training can be pre-stored or collected by each edge device for the corresponding road section. This is not limited in the embodiments of the present application.
[0078] Step 102: collect images of the current road section by using at least two different types of sensors to obtain at least two pieces of second traffic image data.
[0079] It should be noted that the different types here can indicate different types of sensors, such as radar and camera, or different collection ranges of sensors, for example, Figure 4 As shown, Figure 4 The dotted boxes 405, 406, 407 and 408 show two sensors with different acquisition ranges. For the two sensors in each dotted box, the acquisition ranges are opposite. Accordingly, different types of sensors can acquire different images of the current road section, and the at least two second traffic image data can be traffic image data acquired by sensors of different types for the same traffic object. Next, taking two types of sensors as an example, the process of acquiring images of the current road section by at least two different types of sensors to obtain at least two second traffic image data is described.
[0080] In actual implementation, at least two different types of sensors include at least radar and camera. The process of acquiring images of the current road section through at least two different types of sensors to obtain at least two second traffic image data can be: acquiring images of the current road section through radar to obtain point cloud image data, and acquiring images of the current road section through camera to obtain plane image data; and using the point cloud image data and the plane image data as at least two second traffic image data.
[0081] Step 103 : Recognize each second traffic image data respectively through the initial traffic recognition model to obtain a recognition result corresponding to each second traffic image data.
[0082] In actual implementation, the edge device will identify each second traffic image data separately through the initial traffic recognition model, so as to obtain the recognition result corresponding to each second traffic image data. Specifically, the following processing is performed for each second traffic image data: the second traffic image data is continuously downsampled at least twice through the initial traffic recognition model to obtain a downsampled image; the downsampled image is upsampled to obtain an upsampled image, and the image size of the upsampled image and the downsampled image are the same; the upsampled image and the downsampled image are fused to obtain a fused image, and based on the fused image, the second traffic image data is recognized to obtain the recognition result corresponding to the second traffic image data.
[0083] For example, see Figure 5 , Figure 5 is a schematic diagram of the recognition process of the second traffic image data provided by the embodiment of the present application, based on Figure 5 , through the initial traffic recognition model, the second traffic image data is continuously down-sampled four times to obtain four down-sampled images; the down-sampled image obtained by the third down-sampling process is up-sampled to obtain an up-sampled image, the up-sampled image is fused with the down-sampled image obtained by the fourth down-sampling process to obtain a first fused image, and the first fused image is fused with the down-sampled image obtained by the second down-sampling process to obtain a second fused image, and then the second fused image is fused with the down-sampled image obtained by the first down-sampling process to obtain a third fused image, so that the second traffic image data is recognized based on the third fused image to obtain a recognition result corresponding to the second traffic image data.
[0084] In actual implementation, for the model structure of the initial traffic recognition model, see Figure 6 , Figure 6 is a schematic diagram of the model structure of the initial traffic recognition model provided in the embodiment of the present application, based on Figure 6The initial traffic recognition model includes a feature extraction layer, a feature fusion layer and an output layer, so that the process of continuously downsampling the second traffic image data at least twice to obtain the downsampled image can be, through the feature extraction layer, continuously downsampling the second traffic image data at least twice to obtain the downsampled image; upsampling the downsampled image to obtain the upsampled image can be, through the feature extraction layer, upsampling the downsampled image to obtain the upsampled image; fusing the upsampled image with the downsampled image to obtain the fused image can be, through the feature fusion layer, fusing the upsampled image with the downsampled image to obtain the fused image; based on the fused image, the second traffic image data is recognized and the recognition result corresponding to the second traffic image data is obtained, which can be, through the output layer, based on the fused image, the second traffic image data is recognized and the recognition result corresponding to the second traffic image data is obtained.
[0085] Step 104 : when the recognition results corresponding to the second traffic image data are inconsistent, the initial traffic recognition model is updated based on at least two pieces of second traffic image data to obtain a target traffic recognition model.
[0086] It should be noted that, due to the different types of sensors, the coordinate systems of the second traffic image data collected by different types of sensors are different. Thus, the coordinate systems of the recognition results corresponding to each second traffic image data are also different. Based on this, the process of determining whether the recognition results corresponding to each second traffic image data are consistent may be to perform the following processing respectively for the recognition results corresponding to each second traffic image data: in the coordinate system where the sensor corresponding to the second traffic image data is located, obtain the coordinates of the traffic object indicated by the recognition result; perform coordinate conversion on the coordinates of the traffic object to obtain the target coordinates of the traffic object in the target coordinate system; determine the target traffic object of the recognition result in the target coordinate system based on the target coordinates of the traffic object in the target coordinate system; when the target traffic objects corresponding to each recognition result are inconsistent, determine that the recognition results corresponding to each second traffic image data are inconsistent.
[0087] Exemplarily, when at least two different types of sensors are radar and camera, at least two different types of sensors are used to collect images of the current road section to obtain at least two second traffic image data, namely, point cloud image data collected by the radar and plane image data collected by the camera. Thus, the coordinates of the recognition result corresponding to the point cloud image data, such as (x2, y2, v, yaw), are in the radar coordinate system, where x2 and y2 are the positions of the traffic objects in the radar coordinate system, v is the speed of the traffic object, and yaw is the angle of the traffic object; and the coordinates of the recognition result corresponding to the plane image data, such as (x1, y1, w, h), are in the camera coordinate system, where x1 and y1 are the pixel coordinates of the traffic object, and w and h are the width and height of the frame of the traffic object. In this way, after obtaining the coordinates of the traffic object in the recognition result in the radar coordinate system and the camera coordinate system, the target coordinates of the traffic object in the target coordinate system are obtained by performing coordinate conversion on the coordinates of the traffic object with different recognition results; thereby, based on the target coordinates of the traffic object with different recognition results in the target coordinate system, the target traffic objects with different recognition results in the target coordinate system are determined; when the target traffic objects corresponding to the recognition results are inconsistent, it is determined that the recognition results corresponding to the second traffic image data are inconsistent.
[0088] It should be noted that the target coordinate system here can be the coordinate system where the target sensor is located among at least two different types of sensors, or it can be a coordinate system established separately from at least two different types of sensors. This embodiment of the application does not limit this.
[0089] It should be noted that, for the process of inconsistency of the target traffic objects corresponding to each recognition result, the target coordinates of the target traffic objects corresponding to each recognition result in the target coordinate system may be matched, so that when the matching result represents the target traffic objects corresponding to each recognition result, and the target coordinates in the target coordinate system match, it is determined that the target traffic objects corresponding to each recognition result are consistent; when the matching result represents the target traffic objects corresponding to each recognition result, and the target coordinates in the target coordinate system do not match, it is determined that the target traffic objects corresponding to each recognition result are inconsistent.
[0090] In actual implementation, after the initial traffic recognition model is used to recognize each second traffic image data and obtains the recognition result corresponding to each second traffic image data, the initial traffic recognition model can be updated based on at least two second traffic image data to obtain the target traffic recognition model. Specifically, see Figure 7 , Figure 7 is a flowchart of a process of updating an initial traffic recognition model based on at least two second traffic image data to obtain a target traffic recognition model according to an embodiment of the present application, based on Figure 7, step 104 can be implemented as follows.
[0091] Step 1041 , based on each recognition result, determine the label of the corresponding second traffic image data, and based on the label, annotate the corresponding second traffic image data to obtain enhanced training samples of the initial traffic recognition model.
[0092] In actual implementation, the process of determining the label of the corresponding second traffic image data based on each recognition result may be, for each recognition result, performing the following processing: fusing at least one other recognition result with the recognition result to obtain at least one fused recognition result; wherein the other recognition result is a recognition result other than the recognition result among the recognition results corresponding to at least two pieces of second traffic image data; selecting a target recognition result from at least one fused recognition result, and determining the target recognition result as the label of the corresponding second traffic image data.
[0093] It should be noted that in order to fuse at least one other recognition result with the recognition result to obtain at least one fused recognition result, at least one other recognition result is aligned with the recognition result, and then after the alignment, the aligned at least one other recognition result is fused with the recognition result to obtain at least one fused recognition result.
[0094] In some embodiments, as described above, due to the different types of sensors, the coordinate systems of the second traffic image data collected by different types of sensors are different. Thus, the coordinate systems of the recognition results corresponding to each second traffic image data are also different. Based on this, before at least one other recognition result is fused with the recognition result to obtain at least one fused recognition result, each recognition result needs to be converted into the same coordinate system. Specifically, each other recognition result is projected to the coordinate system where the recognition result is located to obtain at least one standard recognition result. Thus, the process of fusion of at least one other recognition result with the recognition result to obtain at least one fused recognition result may be to fuse at least one standard recognition result with the recognition result to obtain at least one fused recognition result.
[0095] It should be noted that projecting each other recognition result to the coordinate system where the recognition result is located to obtain at least one standard recognition result can be done by obtaining the coordinates of the traffic object indicated by each recognition result in the coordinate system where the corresponding sensor is located, performing coordinate transformation on the coordinates of the traffic object to obtain the target coordinates of the traffic object in the coordinate system where the recognition result is located, and determining the target traffic object of the recognition result in the target coordinate system based on the target coordinates of the traffic object in the target coordinate system, thereby determining the standard recognition result based on the target traffic object.
[0096] In actual implementation, for the process of selecting a target recognition result from at least one fused recognition result, here, based on the difference in the number of fused recognition results, the process of selecting the target recognition result is also different. Next, the process of selecting the target recognition result when the number of fused recognition results is one or more is described.
[0097] In some embodiments, when the number of fused recognition results is one, the fused recognition result can be directly determined as the target recognition result; alternatively, the confidence of the fused recognition result can be obtained, and when the confidence is greater than a confidence threshold, the fused recognition result can be determined as the target recognition result.
[0098] In other embodiments, when there are multiple fused recognition results, the process of selecting the target recognition result may be to obtain the confidence of each fused recognition result; and select, from the multiple fused recognition results, a fused recognition result with a confidence greater than a confidence threshold as the target recognition result; or, when there are multiple fused recognition results, to obtain the confidence of each fused recognition result; and sort the confidences to obtain a sorted result; and based on the sorted result, select, from the multiple fused recognition results, a fused recognition result with the largest confidence as the target recognition result.
[0099] It should be noted that the confidence threshold here may be preset, such as ninety-five percent.
[0100] In actual implementation, after the target recognition result is determined as the label of the corresponding second traffic image data, that is, the label of the corresponding second traffic image data is determined, the corresponding second traffic image data is labeled based on the label to obtain an enhanced training sample of the initial traffic recognition model. Here, the enhanced training sample is used to indicate the second traffic image data carrying the label.
[0101] Step 1042: based on the enhanced training samples, the initial traffic recognition model is updated to obtain a target traffic recognition model.
[0102] In actual implementation, the process of updating the initial traffic recognition model based on the enhanced training samples to obtain the target traffic recognition model is to update the initial traffic recognition model in combination with the label and the corresponding second traffic image data to obtain the target traffic recognition model. Specifically, the recognition result corresponding to the corresponding second traffic image data and the difference between the recognition result and the label are obtained; based on the difference, the model parameters of the initial traffic recognition model are updated to obtain the target traffic recognition model.
[0103] In some embodiments, traffic objects on the road are not constant, and there are obvious busy and idle time characteristics. For example, the traffic volume drops sharply during the early morning hours, and the resource consumption at this time is greatly reduced. The computing resources freed up at this time are used for model training. Specifically, the initial traffic recognition model is updated based on at least two second traffic image data to obtain the target traffic recognition model. The process can also be to obtain the resource consumption of the edge device; in response to the resource consumption of the edge device being less than the resource consumption threshold, the initial traffic recognition model is updated based on at least two second traffic image data to obtain the target traffic recognition model.
[0104] It should be noted that the resource consumption here may refer to the resource consumption of the image processor or the resource consumption of the central processing unit; for obtaining the resource consumption of the edge device, the resource consumption of the edge device may be obtained in real time, or the resource consumption of the edge device may be obtained periodically, such as obtaining the resource consumption of the edge device every 6 hours. The embodiments of the present application do not limit this.
[0105] In some embodiments, after the initial traffic recognition model is updated based on at least two second traffic image data to obtain the target traffic recognition model, the target traffic recognition model can also be used for actual application. Specifically, the road section corresponding to the edge device is imaged to obtain at least one traffic image data to be recognized; each traffic image data to be recognized is recognized by the target traffic recognition model to obtain the recognition result corresponding to each traffic image data to be recognized; based on the recognition result corresponding to each traffic image data to be recognized, the traffic object in the corresponding road section and the traffic information of the traffic object are determined; wherein the traffic information includes at least one of the following: the category, form, identification, position, speed and angle of the traffic object. Exemplarily, the category of the traffic object may refer to whether the traffic object is a person, a car or an animal, etc., the form of the traffic object may be the color, model, etc. of the car, the identification of the traffic object may refer to the license plate number of the vehicle, etc., or the ID number of a person, the position of the traffic object may refer to the longitude and latitude of the traffic object, and the angle of the traffic object may refer to the current driving direction of the traffic object.
[0106] In actual implementation, the process of collecting images of the road section corresponding to the edge device to obtain at least one piece of traffic image data to be identified can also be to collect images of the corresponding road section through at least two different types of sensors to obtain at least two pieces of traffic image data to be identified. Therefore, the training method of the above-mentioned traffic recognition model can be carried out in real time, that is, after the recognition results corresponding to each piece of traffic image data to be identified, the recognition results corresponding to each piece of traffic image data to be identified can also be matched. When the matching result indicates that the recognition results corresponding to each piece of traffic image data to be identified are inconsistent, the target traffic recognition model is updated again based on at least one piece of traffic image data to be identified.
[0107] It should be noted that the process of matching the recognition results corresponding to each traffic image data to be recognized is similar to the process of determining whether the recognition results corresponding to each second traffic image data are consistent, and this embodiment of the present application will not be elaborated on this.
[0108] Next, the training method of the traffic recognition model provided in the embodiment of the present application is introduced. Figure 8 , Figure 8 It is a flow chart of the training of the traffic recognition model provided in the embodiment of the present application. Here, the training method of the traffic recognition model provided in the embodiment of the present application is implemented collaboratively by the terminal, i.e., the edge device, and the server, i.e., the central server.
[0109] Step 201: Each edge device collects traffic image data for the corresponding road section.
[0110] In actual implementation, each edge device is dispersedly set at the edge of the road. Different edge devices operate independently of each other, and different edge devices correspond to different road sections, so that the traffic image data collected by different edge devices are different.
[0111] Step 202: Send the traffic image data to the central server.
[0112] In step 203, the central server receives the traffic image data sent by each edge device, and trains an initial traffic recognition model based on the traffic image data sent by each edge device.
[0113] It should be noted that after the central server receives the traffic image data sent by each edge device, it uniformly determines these traffic image data as the first traffic image data, and then labels the first traffic image data to obtain the full sample data set, and then trains the traffic recognition model based on the full sample data set to obtain the initial traffic recognition model.
[0114] In step 204 , the central server sends the initial traffic recognition model to each edge device.
[0115] Step 205 : Each edge device obtains an initial traffic recognition model and its own resource consumption.
[0116] It should be noted that the resource consumption here may refer to the resource consumption of the image processor or the resource consumption of the central processing unit; for obtaining the resource consumption of the edge device, the resource consumption of the edge device may be obtained in real time, or the resource consumption of the edge device may be obtained periodically, such as obtaining the resource consumption of the edge device every 6 hours. The embodiments of the present application do not limit this.
[0117] Step 206, detecting resource consumption, when the detection result indicates that the resource consumption of the edge device is less than the resource consumption threshold, collecting images of the current road section of the edge device through at least two different types of sensors to obtain at least two second traffic image data.
[0118] Step 207 : Recognize each of the second traffic image data using the initial traffic recognition model to obtain a recognition result corresponding to each of the second traffic image data.
[0119] Step 208 , when the recognition results corresponding to the second traffic image data are inconsistent, determine the labels of the corresponding second traffic image data based on the recognition results, and annotate the corresponding second traffic image data based on the labels to obtain enhanced training samples of the initial traffic recognition model.
[0120] Step 209: Based on the enhanced training samples, the initial traffic recognition model is updated to obtain a target traffic recognition model.
[0121] In step 210, each edge device collects images for the corresponding road section to obtain at least one piece of traffic image data to be identified, and identifies each piece of traffic image data to be identified through the target traffic recognition model to obtain a recognition result corresponding to each piece of traffic image data to be identified.
[0122] In actual implementation, each edge device collects images for the corresponding road section to obtain at least one piece of traffic image data to be identified, and may also collect images for the corresponding road section through at least two different types of sensors to obtain at least two pieces of traffic image data to be identified. Thus, the training method of the above-mentioned traffic recognition model may be carried out in real time, that is, after the recognition results corresponding to each piece of traffic image data to be identified are obtained, the recognition results corresponding to each piece of traffic image data to be identified may also be matched. When the matching result indicates that the recognition results corresponding to each piece of traffic image data to be identified are inconsistent, the target traffic recognition model is updated again based on at least one piece of traffic image data to be identified.
[0123] Step 211 : determining the traffic objects in the corresponding road section and the traffic information of the traffic objects based on the recognition results corresponding to the traffic image data to be recognized.
[0124] It should be noted that the traffic information includes at least one of the following: the type, form, identification, position, speed and angle of the traffic object.
[0125] Step 212: output the traffic objects in the corresponding road section and the traffic information of the traffic objects.
[0126] In actual implementation, the edge device can display the identified traffic objects in the corresponding road section and the traffic information of the traffic objects in the human-computer interaction interface, and can also save the identified traffic objects in the corresponding road section and the traffic information of the traffic objects locally on the terminal, and can also send the identified traffic objects in the corresponding road section and the traffic information of the traffic objects to other devices that are communicatively connected to the terminal.
[0127] By applying the above-mentioned embodiment of the present application, after the edge devices distributed in different sections obtain the same initial traffic recognition model, for the edge devices of each section, at least two different types of sensors are used to collect images of the current section, and then the collected second traffic image data are respectively recognized through the initial traffic recognition model to obtain the recognition results corresponding to each second traffic image data; when the recognition results corresponding to each second traffic image data are inconsistent, the initial traffic recognition model is updated based on at least two second traffic image data to obtain the target traffic recognition model. In this way, each edge device collects the second traffic image data of its own section, and based on the recognition results of each second traffic image data, determines the training data for updating the initial traffic recognition model from multiple second traffic image data to update the acquired initial traffic recognition model, so that the initial traffic recognition model of each edge device can be updated according to the traffic image data of the section where it is located, thereby enhancing the training effect and training efficiency of the model and improving the accuracy of the traffic recognition results.
[0128] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.
[0129] The first prerequisite for smart transportation is traffic perception on the road. As the role of road information digitization, traffic perception digitizes the physically existing vehicles, roads, and people through AI intelligent learning detection, and represents road elements and road participants in the form of information. The premise of road perception is the need to establish intelligent sensing equipment on the road, such as cameras, radars, and MEC. A typical single-point intelligent sensing equipment and road section layout method. Traffic perception technical solutions in related technologies, such as Figure 4As shown, the construction premise is the same, that is, to build points at fixed intervals to perceive and cover the road. For each covered road, the road information captured by the sensor will be transmitted to the MEC for the next AI calculation; that is, the MEC at each point will access the sensor data of this point, and the MEC at each point will deploy an AI model for calculation. For example, a 100-kilometer section of highway is constructed with one point every 1 kilometer (or 500 meters). If there is a MEC at each of the 100 points (or 200), there will be 100 MECs (or 200), and each MEC will run an AI model to detect the road conditions within the field of view of the current point, such as vehicle detection. At the same time, the traffic perception technical solution in the relevant technology is to train the AI model (traffic recognition model) on the center side, and then send and deploy the AI model to each MEC separately. The data required for training the AI model on the center side is generally other scene data accumulated in advance, as well as the newly added annotation data of the target road section.
[0130] However, the main problem with the above solution is that the MEC at each point uses the same AI model, that is, the AI model on the central side, which is sent to each MEC indiscriminately; then the detection performance of the MEC at each point is completely dependent on the same AI model sent by the central side. Since this AI model is obtained through unified data training on the central side, it lacks customization capabilities. Therefore, the recognition results obtained based on this model will have greater performance limitations.
[0131] Based on this, the embodiment of the present application provides an edge data mining and incremental learning method in a traffic perception scenario, which can provide a refined modeling method for traffic roadside perception, so that the model of each traffic perception point has the characteristics of scene customization, so as to achieve better perception performance at each point. Specifically, compared with each MEC using the same AI model sent by the center side, the present application starts from another perspective and customizes different AI models for each MEC, that is, the AI model on each MEC is customized and updated through the data collected from its own sensors, so that the model can perform well in the current sensor scenario without having to perform well in other scenarios. In this way, the model at each point is no longer the same model sent by the center side, but a model customized according to its own sensor data; the model at each point will only run in its own point sensor scenario, it only needs to ensure that the detection performance of the road section it is responsible for is excellent, and there is no need to worry about the negative effects, that is, the problem of poor performance in other point scenarios.
[0132] In actual implementation, see Fig. 9 , Fig. 9 This is a system architecture diagram of the edge data mining and incremental learning method in the traffic perception scenario provided by the embodiment of the present application, based on Fig. 9 , this application first uses the labeled full sample data set to perform model training in the central cloud to obtain a central cloud AI model (initial traffic recognition model) with good performance for the full sample data set, which is called the base model. The base model is then sent to each edge computing unit (edge device), such as edge computing unit A, edge computing unit B, etc. Each edge computing unit corresponds to a point on the road section and is responsible for vehicle detection within the field of view of the point. Afterwards, for the base model of each edge computing unit, such as model A (initial traffic recognition model) corresponding to edge computing unit A, the sensor data (second traffic image data) within the field of view of the point will be inferred, and this part of the data will be recorded as A field of view data (second traffic image data). After the A field of view data is inferred by model A (initial traffic recognition model), the inference result (recognition result) of each sample will be obtained. Next, valuable samples are screened and stored based on the inference results, and an incremental sample set (enhanced training sample) is gradually formed. Then, the incremental learning module inside the edge computing unit A is used to perform incremental learning on the incremental sample set to obtain a model (target traffic recognition model) with better sensor data performance within the field of view of the edge computing unit A at this point, and the model A is updated with this model. This cycle is repeated and eventually converges to the point where sample mining no longer mines more valuable samples and the performance of model A at this point is optimal.
[0133] It should be noted that for the full sample data set, the full sample data set can be pre-stored annotated traffic image data. Here, after the central cloud obtains the pre-stored traffic image data, it uniformly determines these traffic image data as traffic image data, and then annotates the traffic image data to obtain the full sample data set, and then trains the initial model based on the full sample data set to obtain the central cloud AI model, and then uses the model as the base model and distributes it to each edge computing unit.
[0134] For the AI model trained in the central cloud, the full sample data set is used as the model training data. After convergence on the data set, the central cloud AI model is obtained. This model, as the base model, will be distributed to each edge computing unit and used as the initial model in each edge computing unit to infer the data within the field of view of each point sensor; it will be replaced by a new incremental model through sample mining and incremental learning (that is, based on the enhanced training samples, the initial traffic recognition model is trained to obtain the target traffic recognition model).
[0135] For edge computing units, computing units (industrial computers, such as AGXJETSON XAVIER) with complete computing, storage, and communication capabilities are scattered at the edge of the road. Different edge computing units operate independently of each other, and different edge computing units correspond to different road sections. They are physically installed on each pole and are responsible for the reasoning of image data collected by sensors (cameras, radars, etc.) installed on the road section, and output the recognition results within the point range, that is, vehicle information, such as category, color, ID, location, speed, angle, etc. During initialization, the base model (initial traffic recognition model) issued by the central cloud is used for reasoning operations, and the incremental model (target traffic recognition model) obtained by sample mining and incremental learning will be used later.
[0136] Model A, also known as the logical unit, represents the inference model currently used by the edge computing unit. There are two stages in the life cycle. The first stage is the initial stage, in which Model A is the base model issued by the central cloud; the second stage is the incremental stage, in which Model A is an incremental model obtained through sample mining and incremental learning.
[0137] For field of view data, it refers to the sensor data at that point (camera video stream, radar point cloud, etc.), and the field of view data for each point is different; A field of view data refers to the sensor data at the point where edge computing unit A is located.
[0138] The sample mining process refers to automatically screening out sample data that is valuable to edge computing unit A based on the result data obtained by inferring A field of view data by model A. In this way, the aforementioned process is repeated so that the sample data is further accumulated and constitutes an incremental sample set.
[0139] The incremental learning process refers to using the data set of the incremental sample set as training data, adopting the incremental learning method, and using the computing power of the idle edge computing unit to train and update the model. Among them, the incremental sample set is the sample set obtained through sample mining, and the data in it is the sample data produced by sample mining that is considered valuable for the improvement of model A. This part of the data will be further used for incremental model training.
[0140] In actual implementation, the sample mining process is difficult because the sample mining in traffic scenarios has no true value. In other words, in traditional sample mining methods, mining is performed on a closed set. The true value labels of the sample data in the training set are known, and sample mining is performed by comparing the difference between the inference result and the true value. However, in traffic scenarios, the coverage data of the deployed edge computing units is an open set without a true value. Therefore, the problem of how to perform sample mining on open set data without a true value arises. The embodiments of the present application make full use of the multi-source sensing characteristics of traffic scenarios and use sensor A to mine the data of sensor B, and vice versa. Specifically, see Fig.10 , Fig.10 This is a technical architecture diagram of the sample mining process provided by the embodiment of the present application, based on Fig.10 For sensors A, B, ..., N at the same point (road section), each sensor has its own observation (second traffic image data) A, B, ..., N, such as the observation of the camera is an image, the observation of the radar is a frame of point cloud, etc. Through the recognition algorithm corresponding to each sensor, the observation data is converted into recognition results. For example, the recognition result of the image is the coordinate position and category of each target in the image. After obtaining each structured data, the recognition results of each sensor are projected to the same coordinate system through calibration projection (coordinate system transformation), that is, the recognition results of the objects obtained under each sensor are transformed into coordinate systems to obtain the recognition results of the objects in the same coordinate system. After projecting to the same coordinate system, each structured result is associated and matched. When the match is successful, the group of samples is skipped and it is considered that this group of samples is in line with expectations. When the matching fails (that is, when the recognition results corresponding to each of the second traffic image data are inconsistent), it is considered that this set of samples does not meet expectations for the current model and is therefore valuable. This set of sensor observations is then accumulated as a sample in the incremental sample set, and this is repeated to eventually obtain an incremental sample set.
[0141] In actual implementation, compared with the incremental learning methods in related technologies, the incremental learning process in traffic scenarios has two characteristics: one is the lack of true value characteristics mentioned above, and the other is the shortage of computing resources. For the lack of true value, because there is no supervision signal, it is impossible to directly use the incremental sample set for training; for the shortage of resources, because the actual operation of the business system must be considered, incremental learning cannot be performed at any time. Based on this, the embodiments of the present application respectively propose methods for mutual verification of true values and utilization during idle time. The mutual verification of the true value means that, because each sample in the incremental sample set is composed of multiple groups of observations, and it is known that these observations have failed to match; therefore, through the projection method (coordinate system transformation), the recognition result A of one observation A is projected onto the recognition result B of another observation B (that is, the other recognition results are projected to the coordinate system where the recognition results are located to obtain at least one standard recognition result), and the reasoning is performed by alignment (that is, the other recognition results are aligned with the recognition results, and then after alignment, the aligned other recognition results are fused with the recognition results to obtain a fused recognition result), and the confidence of the reasoning result (fused recognition result) is observed to see whether it meets the reasonable threshold; if it does, it is considered that the projection of the recognition result A on the observation B is a more accurate target object, and the target object information of the recognition result B is updated, that is, the recognition result A is used as the label of the observation B (that is, the target recognition result is selected from at least one fused recognition result, and the target recognition result is determined as the label of the corresponding second traffic image data). Repeat this process and verify each other to obtain the corrected sample set as the true value of incremental learning. In order to solve the problem of resource shortage and guarantee business system, the embodiment of the present application proposes a method of training during idle time, that is, the traffic objects on the road are not constant, and there are obvious busy and idle time characteristics. For example, the traffic volume drops sharply during the early morning, and the resource consumption at this time is greatly reduced. The computing resources freed up at this time are used for model training, and dynamic scheduling and dynamic recording are performed. Among them, dynamic scheduling refers to the fact that the camera working conditions are unavailable at night, and the corresponding image processor (GPU, Graphic Processing Unit) does not run, so the idle GPU is used to perform model training during the idle time of the GPU at night. Dynamic recording refers to the scheduling of GPU detection, performing incremental training when the GPU is idle, and storing the latest training model results at any time.
[0142] In some embodiments, see Fig.11 , Fig.11 is a schematic diagram of a joint training process based on an incremental sample set provided in an embodiment of the present application, based on Fig.11,The GPU detection trigger detects the GPU (edge device resource) usage of the edge computing unit. ,When the GPU is idle, the model training is triggered. Specifically, the acquired incremental sample set (enhanced training samples) is used to ,perform supervised training based on the deep learning inference model (initial traffic recognition model), so as to obtain the final edge deep learning ,model (target traffic recognition model).
[0143] By applying the above-mentioned embodiment of the present application, after the edge devices distributed in different sections obtain the same initial traffic recognition model, the edge devices of each section use at least two different types of sensors to collect images of the current section, and then use the initial traffic recognition model to respectively recognize the collected second traffic image data to obtain the recognition results corresponding to each second traffic image data; when the recognition results corresponding to each second traffic image data are inconsistent, the initial traffic recognition model is updated based on at least two second traffic image data to obtain the target traffic recognition model. In this way, each edge device collects the second traffic image data of its own section, and based on the recognition results of each second traffic image data, determines the training data for updating the initial traffic recognition model from multiple second traffic image data to update the acquired initial traffic recognition model, so that the initial traffic recognition model of each edge device can be updated according to the traffic image data of the section where it is located, thereby enhancing the training effect and training efficiency of the model and improving the accuracy of traffic recognition by the traffic recognition model.
[0144] The following is an exemplary structure of a software module implemented as a training device 455 for a traffic recognition model provided in an embodiment of the present application. The device is applied to a traffic sensing system, and the traffic sensing system includes a plurality of edge devices, each of which is dispersedly arranged at the edge of a road, and different edge devices correspond to different road sections. In some embodiments, for example Figure 2 As shown, the software modules stored in the training device 455 of the traffic recognition model in the memory 450 may include:
[0145] An acquisition module 4551 is used for the edge device to acquire an initial traffic recognition model, where the initial traffic recognition model is used to recognize traffic objects on the road and is trained based on first traffic image data collected by each edge device for the corresponding road section;
[0146] The acquisition module 4552 is used to acquire images of the current road section through at least two different types of sensors to obtain at least two pieces of second traffic image data;
[0147] The recognition module 4553 is used to recognize each of the second traffic image data by using the initial traffic recognition model to obtain a recognition result corresponding to each of the second traffic image data;
[0148] The updating module 4554 is used to update the initial traffic recognition model based on the at least two second traffic image data to obtain a target traffic recognition model when the recognition results corresponding to the second traffic image data are inconsistent.
[0149] In some embodiments, the device also includes a coordinate system conversion module, which is used to perform the following processing for the recognition results corresponding to each of the second traffic image data: obtaining the coordinates of the traffic object indicated by the recognition result in the coordinate system where the sensor corresponding to the second traffic image data is located; performing coordinate conversion on the coordinates of the traffic object to obtain the target coordinates of the traffic object in the target coordinate system; determining the target traffic object of the recognition result in the target coordinate system based on the target coordinates of the traffic object in the target coordinate system; when the target traffic objects corresponding to each of the recognition results are inconsistent, determining that the recognition results corresponding to each of the second traffic image data are inconsistent.
[0150] In some embodiments, the update module 4554 is also used to determine the label of the corresponding second traffic image data based on each of the recognition results, and based on the label, annotate the corresponding second traffic image data to obtain an enhanced training sample of the initial traffic recognition model; based on the enhanced training sample, update the initial traffic recognition model to obtain a target traffic recognition model.
[0151] In some embodiments, the update module 4554 is also used to perform the following processing for each of the recognition results: fusing at least one other recognition result with the recognition result to obtain at least one fused recognition result; wherein the other recognition result is a recognition result other than the recognition result among the recognition results corresponding to the at least two second traffic image data; selecting a target recognition result from at least one of the fused recognition results, and determining the target recognition result as a label of the corresponding second traffic image data.
[0152] In some embodiments, the device also includes a fusion module, which is used to project each of the other recognition results to the coordinate system where the recognition result is located, so as to obtain at least one standard recognition result; the update module 4554 is also used to fuse the at least one standard recognition result with the recognition result, so as to obtain at least one fused recognition result.
[0153] In some embodiments, the update module 4554 is also used to obtain the confidence of each fused recognition result when there are multiple fused recognition results; and select the fused recognition result whose confidence is greater than the confidence threshold from the multiple fused recognition results as the target recognition result.
[0154] In some embodiments, the update module 4554 is also used to obtain the confidence of each fused recognition result when there are multiple fused recognition results; sort the confidences to obtain a sorting result: based on the sorting result, select the fused recognition result with the largest confidence from the multiple fused recognition results as the target recognition result.
[0155] In some embodiments, the at least two different types of sensors include at least a radar and a camera, and the acquisition module 4552 is also used to acquire images of the current road section through the radar to obtain point cloud image data, and to acquire images of the current road section through the camera to obtain plane image data; and use the point cloud image data and the plane image data as the at least two second traffic image data.
[0156] In some embodiments, the update module 4554 is also used to obtain the resource consumption of the edge device; in response to the resource consumption of the edge device being less than a resource consumption threshold, the initial traffic recognition model is updated based on the at least two second traffic image data to obtain a target traffic recognition model.
[0157] In some embodiments, the recognition module 4553 is also used to perform the following processing on each second traffic image data: using the initial traffic recognition model, continuously downsampling the second traffic image data at least twice to obtain a downsampled image; upsampling the downsampled image to obtain an upsampled image, and the upsampled image has the same image size as the downsampled image; fusing the upsampled image with the downsampled image to obtain a fused image, and based on the fused image, identifying the second traffic image data to obtain an identification result corresponding to the second traffic image data.
[0158] In the above scheme, the device also includes an application module, which is used to collect images of the road section corresponding to the edge device to obtain at least one traffic image data to be identified; through the target traffic recognition model, each of the traffic image data to be identified is respectively identified to obtain the recognition result corresponding to each of the traffic image data to be identified; based on the recognition result corresponding to each of the traffic image data to be identified, the traffic objects in the corresponding road section and the traffic information of the traffic objects are determined; wherein the traffic information includes at least one of the following: the category, form, identification, position, speed and angle of the traffic object.
[0159] The present application embodiment provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the training method of the traffic recognition model in the present application embodiment, for example, Figure 3 The training method of the traffic recognition model is shown.
[0160] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the training method of the traffic recognition model provided by the present application embodiment, for example, Figure 3 The training method of the traffic recognition model is shown.
[0161] In some embodiments, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disk, or a CD-ROM; or it may be various devices including one or any combination of the above memories.
[0162] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.
[0163] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).
[0164] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.
[0165] It should be noted that in the embodiments of the present application, it involves obtaining image data collected by sensors and other related data. When the embodiments of the present application are applied to specific products or technologies, corresponding permissions or consents need to be obtained, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0166] In summary, the embodiments of the present application have the following beneficial effects:
[0167] (1) Each edge device collects the second traffic image data of its own road section, and based on the recognition results of each second traffic image data, determines the training data for updating the initial traffic recognition model from multiple second traffic image data, so as to update the acquired initial traffic recognition model, so that the initial traffic recognition model of each edge device can be updated according to the traffic image data of the road section where it is located, thereby enhancing the training effect and training efficiency of the model and improving the accuracy of traffic recognition by the traffic recognition model.
[0168] (2) Compared with each MEC using the same AI model sent by the center, this application starts from another perspective and customizes different AI models for each MEC, that is, the AI model on each MEC is customized and updated through the data collected from its own sensors, so that the model can perform well in the current sensor scenario without having to perform well in other scenarios. In this way, the model at each point is no longer the same model sent by the center, but a model customized according to its own sensor data; the model at each point will only run in its own point sensor scenario, it only needs to ensure that the detection performance of the road section it is responsible for is excellent, and there is no need to worry about the negative effects, that is, the problem of poor performance in other point scenarios.
[0169] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.
Claims
1. A method for training a traffic recognition model, characterized in that: Applied to a traffic sensing system, the traffic sensing system includes a plurality of edge devices, each of the edge devices is dispersedly arranged at the edge of a road, and different edge devices correspond to different road sections. The method includes: The edge device acquires an initial traffic recognition model, where the initial traffic recognition model is used to recognize traffic objects on the road and is trained based on first traffic image data collected by each edge device for a corresponding road section; Capturing images of the current road section by using at least two different types of sensors to obtain at least two pieces of second traffic image data; Recognize each of the second traffic image data respectively by using the initial traffic recognition model to obtain a recognition result corresponding to each of the second traffic image data; When the recognition results corresponding to the second traffic image data are inconsistent, the initial traffic recognition model is updated based on the at least two second traffic image data to obtain a target traffic recognition model.
2. The method according to claim 1, characterized in that After the initial traffic recognition model is used to recognize each of the second traffic image data and a recognition result corresponding to each of the second traffic image data is obtained, the method further includes: For each recognition result corresponding to the second traffic image data, the following processing is performed respectively: Acquire the coordinates of the traffic object indicated by the recognition result in the coordinate system of the sensor corresponding to the second traffic image data; Performing coordinate transformation on the coordinates of the traffic object to obtain target coordinates of the traffic object in a target coordinate system; Determining the target traffic object of the recognition result in the target coordinate system based on the target coordinates of the traffic object in the target coordinate system; When the target traffic objects corresponding to the recognition results are inconsistent, it is determined that the recognition results corresponding to the second traffic image data are inconsistent.
3. The method according to claim 1, characterized in that The updating of the initial traffic recognition model based on the at least two second traffic image data to obtain a target traffic recognition model includes: Determine a label of the corresponding second traffic image data based on each of the recognition results, and annotate the corresponding second traffic image data based on the label to obtain an enhanced training sample of the initial traffic recognition model; Based on the enhanced training samples, the initial traffic recognition model is updated to obtain a target traffic recognition model.
4. The method according to claim 3, characterized in that The step of determining a label of the corresponding second traffic image data based on each of the recognition results includes: For each of the recognition results, the following processing is performed: Fusing at least one other recognition result with the recognition result to obtain at least one fused recognition result; The other recognition results are recognition results other than the recognition results among the recognition results corresponding to the at least two second traffic image data; A target recognition result is selected from at least one of the fused recognition results, and the target recognition result is determined as a label of the corresponding second traffic image data.
5. The method according to claim 4, characterized in that The method further comprises: Projecting each of the other recognition results to the coordinate system where the recognition result is located to obtain at least one standard recognition result; The step of fusing at least one other recognition result with the recognition result to obtain at least one fused recognition result comprises: The at least one standard recognition result is fused with the recognition result to obtain at least one fused recognition result.
6. The method according to claim 4, characterized in that The step of selecting a target recognition result from at least one of the fused recognition results comprises: When there are multiple fusion recognition results, obtaining the confidence level of each fusion recognition result; From the multiple fused recognition results, a fused recognition result whose confidence is greater than a confidence threshold is selected as the target recognition result.
7. The method according to claim 4, characterized in that The step of selecting a target recognition result from at least one of the fused recognition results comprises: When there are multiple fusion recognition results, obtaining the confidence level of each fusion recognition result; Sorting the confidence levels to obtain a sorting result; Based on the sorting result, the fused recognition result with the highest confidence is selected from the multiple fused recognition results as the target recognition result.
8. The method according to claim 1, characterized in that The at least two different types of sensors include at least a radar and a camera, and the at least two different types of sensors are used to collect images of the current road section to obtain at least two second traffic image data, including: The radar is used to collect images of the current road section to obtain point cloud image data, and the camera is used to collect images of the current road section to obtain plane image data; The point cloud image data and the plane image data are used as the at least two pieces of second traffic image data.
9. The method according to claim 1, characterized in that The updating of the initial traffic recognition model based on the at least two second traffic image data to obtain a target traffic recognition model includes: Obtain resource consumption of the edge device; In response to the resource consumption of the edge device being less than a resource consumption threshold, the initial traffic recognition model is updated based on the at least two second traffic image data to obtain a target traffic recognition model.
10. The method according to claim 1, characterized in that The step of identifying each of the second traffic image data by using the initial traffic recognition model to obtain a recognition result corresponding to each of the second traffic image data includes: The following processing is performed on each of the second traffic image data: Using the initial traffic recognition model, continuously downsampling the second traffic image data at least twice to obtain a downsampled image; Performing upsampling processing on the downsampled image to obtain an upsampled image, wherein the upsampled image has the same image size as the downsampled image; The up-sampled image and the down-sampled image are fused to obtain a fused image, and based on the fused image, the second traffic image data is recognized to obtain a recognition result corresponding to the second traffic image data.
11. The method according to claim 1, characterized in that The method further comprises: Capturing images of the road section corresponding to the edge device to obtain at least one piece of traffic image data to be identified; Using the target traffic recognition model, each of the traffic image data to be recognized is recognized, and a recognition result corresponding to each of the traffic image data to be recognized is obtained; Determine the traffic object in the corresponding road section and the traffic information of the traffic object based on the recognition result corresponding to each of the traffic image data to be recognized; The traffic information includes at least one of the following: the type, form, identification, position, speed and angle of the traffic object.
12. A training device for a traffic recognition model, characterized in that: Applied to a traffic sensing system, the traffic sensing system includes a plurality of edge devices, each of which is dispersedly arranged at the edge of a road, and different edge devices correspond to different road sections. The device includes: An acquisition module is used for the edge device to acquire an initial traffic recognition model, where the initial traffic recognition model is used to recognize traffic objects on the road and is trained based on first traffic image data collected by each edge device for the corresponding road section; A collection module, used for collecting images of the current road section through at least two different types of sensors to obtain at least two second traffic image data; A recognition module, used to recognize each of the second traffic image data respectively through the initial traffic recognition model to obtain a recognition result corresponding to each of the second traffic image data; The updating module is used to update the initial traffic recognition model based on the at least two second traffic image data to obtain a target traffic recognition model when the recognition results corresponding to the second traffic image data are inconsistent.
13. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the training method for the traffic recognition model described in any one of claims 1 to 11 when executing the executable instructions stored in the memory.
14. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to cause the processor to execute and implement the training method of the traffic recognition model described in any one of claims 1 to 11.
15. A computer program product comprising a computer program or computer executable instructions, characterized in that: When the computer program or computer executable instructions are executed by a processor, the training method of the traffic recognition model described in any one of claims 1 to 11 is implemented.
