Multi-terminal collaborative vehicle model updating method and device, vehicle and equipment

Abstract

The present disclosure relates to a multi-terminal collaborative vehicle model updating method and device, a vehicle and equipment. The method comprises: obtaining a training sample; training a first model of the vehicle through the training sample to obtain a first parameter value of a model parameter of the vehicle at the end of training; obtaining a gradient value of the model parameter according to the first parameter value of the model parameter and a second parameter value of the model parameter at the beginning of training; sending the gradient value to a multi-terminal collaborative equipment for collaborative updating of the model; receiving a third parameter value of the model parameter provided by the multi-terminal collaborative equipment, and updating the first model of the vehicle through the third parameter value; wherein the third parameter value of the model parameter is obtained according to the gradient value of the model parameter and a parameter value of a corresponding model parameter in a public model, the public model and the first model have the same network structure, and the parameter value of the model parameter of the public model is updated based on the value of the corresponding model parameter provided by a plurality of vehicles after the gradient value.

Description

Technical Field

[0001] This disclosure relates to the field of model updating, and more specifically, to a multi-terminal collaborative vehicle model updating method, a multi-terminal collaborative vehicle model updating device, a vehicle, and a multi-terminal collaborative equipment. Background Technology

[0002] Intelligent vehicles are comprehensive systems integrating environmental perception, planning and decision-making, and multi-level assisted driving functions. They require various artificial intelligence models to implement these functions; for example, a target recognition model is needed to identify objects from images captured by cameras. The accuracy of these models is crucial for intelligent vehicles, determining the reliable implementation of their intelligent functions. Therefore, intelligent vehicle models need continuous learning and updating. In existing technologies, after a model is deployed, the vehicle can train and update it locally based on acquired samples. However, the limited number of samples and the vehicle's computing power restrict the improvement of model performance. Summary of the Invention

[0003] One objective of this disclosure is to provide a new technical solution for updating a vehicle model by combining the local training results of multiple vehicles to improve model performance.

[0004] According to a first aspect of this disclosure, an embodiment of a multi-terminal collaborative vehicle model update method is provided, the method being implemented by a vehicle, the method comprising:

[0005] Obtain training samples for the vehicle;

[0006] The first model of the vehicle is trained using the training samples to obtain the first parameter values ​​of the model parameters at the end of training.

[0007] The gradient value of the model parameter is obtained based on the first parameter value of the model parameter and the second parameter value of the model parameter at the beginning of training;

[0008] The gradient values ​​of the model parameters are sent to a multi-terminal collaborative device for collaborative model updates.

[0009] The system receives a third parameter value of the model parameters obtained by the multi-terminal collaborative device through the collaborative update; wherein the third parameter value of the model parameters is obtained by the multi-terminal collaborative device based on the gradient value of the model parameters and the parameter value of the corresponding model parameters in the common model, the common model having the same network structure as the first model, and the parameter value of the model parameters of the common model being the value updated by the multi-terminal collaborative device based on the gradient value of the corresponding model parameters provided by multiple vehicles;

[0010] The first model of the vehicle is updated using the third parameter value.

[0011] Optionally, the third parameter value of the model parameters is obtained by the multi-terminal collaborative device based on the gradient value of the model parameters and the parameter value of the corresponding model parameters in the common model, including:

[0012] The third parameter value of the model parameter is obtained by the multi-terminal collaborative device performing a first weighted calculation on the gradient value of the model parameter and the parameter value of the model parameter; wherein, the weight of the gradient value of the model parameter in the first weighted calculation is greater than or equal to the weight of the parameter value of the model parameter in the first weighted calculation.

[0013] Optionally, the public model is updated by the multi-terminal collaborative device after obtaining the third parameter value of the model parameters, based on the gradient value of the model parameters and the parameter value of the corresponding model parameter in the public model.

[0014] Optionally, the public model is updated by the multi-terminal collaborative device after obtaining the third parameter value of the model parameters, based on the gradient value of the model parameters and the parameter value of the corresponding model parameter in the public model, including:

[0015] The public model is updated by the multi-terminal collaborative device after obtaining the third parameter value of the model parameter, based on the calculation result of a second weighted calculation of the gradient value of the model parameter and the parameter value of the corresponding model parameter in the public model; wherein, the weight of the gradient value of the model parameter in the second weighted calculation is less than the weight of the parameter value of the model parameter in the second weighted calculation.

[0016] Optionally, after sending the gradient values ​​of the model parameters to a multi-terminal collaborative device for collaborative model updates, the method further includes:

[0017] The system receives parameter values ​​of the model parameters of the first application model obtained by the multi-terminal collaborative device through the collaborative update; wherein, the first application model is the model used by the vehicle when performing a set task, the first model is a training model used to update the first application model, and the parameter values ​​of the model parameters of the first application model are obtained by the multi-terminal collaborative device based on the third parameter value of the model parameters;

[0018] The first application model is updated based on the parameter values ​​of the model parameters received from the first application model.

[0019] Optionally, the first model is a model corresponding to a target recognition task, wherein the target recognition task includes recognizing the location of the target object and recognizing the category of the target object, and before obtaining the training samples of the vehicle, the method further includes:

[0020] Acquire images captured by the vehicle's camera;

[0021] The image is input to the first model or a first application model corresponding to the first model to obtain detection box information of the detection boxes corresponding to the target objects identified in the image; wherein, the first application model is a distillation model of the first model, and the detection box information includes the position information of the detection box in the image and the confidence level of the target object category corresponding to the detection box;

[0022] Obtain bounding box information for multiple frames of images within a set time period;

[0023] Based on the confidence scores of each detection box corresponding to the same target object category in the multi-frame images, the associated data corresponding to each detection box is fitted, and valid detection boxes are selected from each detection box based on the fitting results; wherein, the associated data corresponding to the detection box includes the position information of the detection box in the image and the acquisition time of the image where the detection box is located;

[0024] Obtain the highest confidence level corresponding to all valid bounding boxes;

[0025] The confidence level of each valid detection frame is relabeled as the highest confidence level;

[0026] Training samples for the vehicle are generated based on the confidence scores of the relabeled valid detection boxes.

[0027] Optionally, before acquiring the detection box information of multiple frames of images within a set time period, the method further includes:

[0028] If the highest confidence level corresponding to the detection box of the image is greater than the first threshold, the detection box information of the image is set and processed to obtain the detection boxes of the image that need to be retained.

[0029] If the highest confidence level corresponding to the detection box in the image is less than or equal to the first threshold, all detection boxes in the image are retained.

[0030] The setting process includes:

[0031] Filter out detection boxes with a confidence level less than the first threshold from all detection boxes in the image to obtain a candidate set of detection boxes;

[0032] From the candidate set of detection boxes, the first detection box corresponding to the highest confidence level is selected;

[0033] Traverse the candidate detection boxes in the detection box set except for the first detection box, and calculate the intersection-union ratio between each other detection box and the first detection box;

[0034] Filter out other detection boxes in the candidate detection box set that make the intersection-union ratio greater than the second threshold, and take out the first detection box from the candidate detection box set and store it in the first set;

[0035] If the number of detection boxes in the first set reaches a set value, the detection boxes in the second set are selected as the detection boxes to be retained in the image.

[0036] If the number of detection boxes in the first set is less than the set value, the step of filtering out the first detection box with the highest confidence from the candidate detection box set is performed again.

[0037] Optionally, fitting the associated data corresponding to each detection box based on the confidence level of each detection box corresponding to the same target object category in the multi-frame images includes:

[0038] In each detection frame, a set number of detection frames with a confidence level greater than the third threshold are randomly selected as interior points;

[0039] A set curve is obtained by fitting the associated data corresponding to the inlier points; wherein, the set curve is a curve that reflects the change of the position information of the identified target object in the image over time.

[0040] If each detection box has an outlier that does not conform to the set curve, the outlier is filtered out from each detection box, and the step of randomly selecting a set number of detection boxes with a confidence level greater than the third threshold as inliers is performed again.

[0041] The fitting process ends when none of the detection boxes have the stated outer point, and the remaining detection boxes are considered valid detection boxes.

[0042] According to a second aspect of this disclosure, another embodiment of a multi-terminal collaborative vehicle model update method is provided, the method comprising:

[0043] The gradient values ​​of the model parameters of the first model sent by the vehicle are received; wherein the gradient values ​​of the model parameters are obtained based on the first parameter values ​​of the model parameters at the end of the first model training and the second parameter values ​​of the model parameters at the beginning of the first model training.

[0044] The third parameter value of the model parameter is obtained based on the gradient value of the model parameter and the parameter value of the corresponding model parameter in the common model;

[0045] The third parameter value of the model parameters is sent to the vehicle to update the first model.

[0046] According to a third aspect of this disclosure, an embodiment of a multi-terminal collaborative vehicle model update apparatus is provided, the apparatus comprising:

[0047] A sample acquisition module is used to acquire training samples of the vehicle;

[0048] A local training module is used to train a first model of the vehicle using the training samples, and to obtain the first parameter values ​​of the model parameters at the end of training.

[0049] The gradient calculation module is used to obtain the gradient value of the model parameters based on the first parameter value of the model parameters and the second parameter value of the model parameters at the beginning of training;

[0050] The sending module is used to send the gradient values ​​of the model parameters to a multi-terminal collaborative device for collaborative model updates.

[0051] The receiving module receives the third parameter value of the model parameters obtained by the multi-terminal collaborative device through the collaborative update; wherein the third parameter value of the model parameters is obtained by the multi-terminal collaborative device based on the gradient value of the model parameters and the parameter value of the corresponding model parameters in the common model, the common model having the same network structure as the first model, and the parameter value of the model parameters of the common model being the updated value by the multi-terminal collaborative device based on the gradient values ​​of the corresponding model parameters provided by multiple vehicles; and,

[0052] The model update module is used to update the first model of the vehicle using the third parameter value.

[0053] According to a fourth aspect of the present disclosure, a vehicle is provided that includes the vehicle model updating apparatus according to a third aspect of the present disclosure; or, the vehicle includes a memory and a processor, the memory storing executable instructions for controlling the processor to operate to perform the method according to a first aspect of the present disclosure.

[0054] According to a fifth aspect of the present disclosure, a multi-terminal collaborative device is also provided, comprising a memory and a processor, the memory storing executable instructions for controlling the processor to operate in order to perform the method according to a second aspect of the present disclosure.

[0055] One beneficial effect of this disclosure is that, according to the vehicle model update method of this disclosure, after the vehicle trains the first model locally, it can obtain a gradient value that reflects the degree of change of the parameter values ​​of the first model during the current training, and upload the gradient value to a multi-terminal collaborative device for collaborative model update, so as to make use of the training results of other vehicles and improve the performance of the vehicle model. Furthermore, in the method of this embodiment, the multi-terminal collaborative device maintains a common model, which has the same network structure as the first model, that is, the two have the same model parameters, but the parameter values ​​of the model parameters are usually different. When the multi-terminal collaborative device performs collaborative updates, it determines the third parameter value of the vehicle's model parameters based on the gradient value of the model parameters provided by the vehicle and the parameter value of the corresponding model parameters in the common model. The model parameters of the common model are jointly maintained and updated by the training results of multiple vehicles. That is, the parameter value of the model parameters of the common model is the updated value based on the gradient value of the corresponding model parameters provided by multiple vehicles. In this way, when the vehicle updates its local first model based on the third parameter value of the model parameters provided by the multi-terminal collaborative device, it can achieve the purpose of integrating the local training results of multiple vehicles to update the vehicle model, thereby improving model performance with higher model update efficiency and economical model update cost.

[0056] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0057] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0058] Figure 1 This is a schematic diagram of the vehicle system composition structure that can be used to implement the vehicle model update method of the embodiments of this disclosure;

[0059] Figure 2 This is a flowchart illustrating a vehicle model update method according to some embodiments;

[0060] Figure 3 This is a flowchart illustrating a vehicle model update method according to other embodiments;

[0061] Figure 4 This is a schematic diagram of the process of obtaining training samples for a vehicle according to some embodiments;

[0062] Figure 5 This is a schematic diagram of the process of obtaining training samples for a vehicle according to some other embodiments;

[0063] Figure 6This is a flowchart illustrating a method for updating a vehicle model using multi-terminal collaborative devices, according to some embodiments.

[0064] Figure 7 This is a structural schematic diagram of a vehicle model updating device according to some embodiments;

[0065] Figure 8 This is a structural diagram of the vehicle based on some embodiments;

[0066] Figure 9 These are structural schematic diagrams of a vehicle according to other embodiments;

[0067] Figure 10 This is a structural schematic diagram of a multi-terminal collaborative device according to some embodiments. Detailed Implementation

[0068] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0069] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0070] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0071] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0072] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0073] <Implementation Scenarios and Hardware Configuration>

[0074] Figure 1 This is a schematic diagram of the composition structure of a vehicle system 100 that can be used to implement the vehicle model update method of the embodiments of this disclosure. The vehicle system 100 can be applied as a whole to multi-terminal collaborative vehicle model update scenarios, wherein... Figure 1 The vehicle system 100 shown is merely illustrative and is by no means intended to limit the invention, its application, or its uses.

[0075] like Figure 1As shown, the vehicle system 100 includes a multi-terminal collaborative device 1000 and m vehicles 2000, namely vehicles 2001, 2002, ..., 200m, where the j-th vehicle is 200j, m is an integer greater than 1, and j is an integer from 1 to m.

[0076] Each vehicle 2000 is connected to the multi-terminal collaborative device 1000. The networks on which different vehicles 2000 communicate with the multi-terminal collaborative device can be the same network or different networks.

[0077] Vehicle 2000 has a pre-set model for performing a defined task. The model for vehicle 200j is referred to as the j-th model in this paper; that is, the model for vehicle 2001 is the first model, the model for vehicle 2002 is the second model, and the model for vehicle 200m is the m-th model. These models have the same network structure, meaning they have the same model parameters. The model parameters of the j-th model are referred to as model parameters θ in this paper. ji , where i is an integer from 1 to N, and N is the number of model parameters for these models. For example, if the j-th model has 1000 model parameters, then N is 1000. Here, since different vehicles have different usage scenarios, the model parameters of different vehicle models usually have different parameter values. Usage scenarios involve, for example, road conditions and external environment. Different usage scenarios differ in at least one aspect of road conditions and external environment.

[0078] Vehicle 200j can periodically train the j-th model locally using training samples it has acquired, thereby obtaining the model parameters θ. ji The first parameter value at the end of training, and based on the model parameter θ ji The first parameter value and the model parameter θ ji The second parameter value at the start of training yields the model parameters θ that reflect the j-th model. ji The gradient value of the parameter values ​​during this training is then uploaded to the multi-device collaborative device 1000 for collaborative model updates. Model parameters θ ji For any model parameter among N model parameters, the same processing is applied to each model parameter. Taking the j-th model with 1000 model parameters as an example, after training the j-th model, vehicle 200j will obtain the first parameter value of each of the 1000 model parameters at the end of training, and then obtain the gradient value of each model parameter. Figure 1 As shown, the multi-terminal collaborative device 1000 maintains a common model, which has the same network structure and model parameters as the local model of the vehicle 2000, referred to in this paper as model parameters θ. i Among them, the model parameters θ of the common modeli With model parameters θ ji Correspondingly, both use the same model parameters, but typically take different values. Here, the same vehicle 2000 can upload the gradient values ​​obtained through local training to the multi-device collaborative device 1000 multiple times at different time points to continuously and collaboratively update the local model, thereby improving its performance. Furthermore, different vehicles 2000 can upload the gradient values ​​obtained through local training to the multi-device collaborative device 1000 at the same time point or at different time points; this is not limited here.

[0079] like Figure 1 As shown, taking vehicle 2001 as an example, at time node t, the model parameters θ of the first model of vehicle 2001 are... 1i After the gradient value is sent to the multi-terminal collaborative device 1000, the multi-terminal collaborative device 1000 will process the model parameter θ. 1i The gradient value and the corresponding model parameter θ in the common model i The parameter values ​​are fused to obtain the vehicle model parameters θ. 1i The third parameter value is obtained and returned to vehicle 2001. After receiving the third parameter value, vehicle 2001 can adjust the model parameters θ of the first model. 1i The parameter value is updated to the corresponding third parameter value, thus completing the update of the first model. Here, the multi-terminal collaborative device 1000 receives the model parameter θ 1i Before the time node t for the gradient value, model parameters θ from multiple vehicles have been received. ji The gradient value is obtained by considering multiple vehicles, which may or may not include vehicle 2001; this is not limited here. The gradient value is obtained by considering the model parameters θ provided by multiple vehicles. ji The gradient value, with respect to the model parameters θ of the common model. i The parameter values ​​were updated; for example, the model parameter θ was changed. ji The gradient value and the model parameters θ of the common model i The parameter values ​​are combined to update the model parameter θ. i The value of θ, therefore, the model parameter θ i The parameter values ​​are fused from the training results of multiple vehicles. Thus, at time node t, the multi-device collaborative system 1000 can use the model parameters θ... i For model parameters θ 1i During the update process, the training results of multiple vehicles are transferred to vehicle 2001 at a certain ratio, thereby achieving multi-terminal collaborative model update.

[0080] In some embodiments, the multi-terminal collaboration device 1000 can be a server. This server can be a monolithic server, a distributed server spanning multiple computers, a computer data center, a cloud server, or a server cluster deployed in the cloud, etc. In some embodiments, each server can include hardware, software, or embedded logic components for performing suitable functions supported or implemented by the server, or a combination of two or more such components. For example, servers may be blade servers, cloud servers, etc., or a server group may consist of multiple servers, including one or more of the types of servers described above, etc.

[0081] In some embodiments, such as Figure 1 As shown, the multi-terminal collaborative device 1000 may include a processor 1100, a memory 1200, an interface device 1300, and a communication device 1400.

[0082] The processor 1100 is used to execute computer programs, which can be written using instruction sets of architectures such as x86, Arm, RISC, MIPS, and SSE. The memory 1200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as a hard disk. The interface device 1300 includes, for example, various bus interfaces and network interfaces. The communication device 1400 is capable of wired or wireless communication.

[0083] The memory 1200 of server 1000 stores a computer program that controls processor 1100 to perform collaborative update processing of the vehicle model. This computer program can be designed by those skilled in the art according to the disclosed scheme of this invention. How this computer program controls the processor is well known in the art and will not be described in detail here.

[0084] Vehicle 2000 can be any type of vehicle. In terms of power source, it can be a fuel vehicle, an electric vehicle, or other new energy vehicle. In terms of seating capacity, it can be a 5-seater vehicle, a 7-seater vehicle, etc. Vehicle system 100 can include different types of vehicles, without limitation.

[0085] like Figure 1As shown, in some embodiments, the vehicle 2000 may include a processor 2100, a memory 2200, an interface device 2300, a communication device 2400, an output device 2500, an input device 2600, a positioning device 2700, and various sensors 2800, etc. The processor 2100 is used to execute computer programs, which may be written using instruction sets of architectures such as x86, Arm, RISC, MIPS, and SSE. The memory 2200 includes, for example, ROM (Read-Only Memory), RAM (Random Access Memory), and non-volatile memory such as a hard disk. The interface device 2300 includes, for example, a charging interface and a fault detection interface. The communication device 2400 includes at least one communication module, for example, capable of wired or wireless communication, and for example, capable of short-range and long-range communication. The output device 2500 may be, for example, a device that outputs signals, a display device such as a liquid crystal display screen or a touch screen, or a speaker that outputs voice information. The input device 2600 may include, for example, a touch screen or buttons. Positioning device 2700 may include, for example, a receiver for a Global Navigation Satellite System (GNSS), Global Positioning System (GPS), or BeiDou Navigation Satellite System (BDS). The receiver can determine the position and velocity vector of vehicle 2000 based on signals received from satellites such as GNSS. Various sensors 2800 include, for example, wheel speed sensors, accelerometers, angular velocity sensors, radar, cameras, etc.

[0086] In this embodiment, the memory 2200 of the vehicle 2000 is used to store a computer program that controls the processor 2100 to operate and execute the vehicle model update method according to an embodiment of this disclosure. Those skilled in the art can design this computer program based on the disclosed solution. How this computer program controls the processor to operate is well known in the art and will not be described in detail here.

[0087] <Method Example 1>

[0088] Figure 2 This is a flowchart illustrating a multi-terminal collaborative vehicle model update method according to some embodiments. Now, let's take... Figure 1 The method of this embodiment is illustrated by taking the vehicle 2001 shown as an example. The vehicle 2001 can be any vehicle in the vehicle system 100. That is, each vehicle in the vehicle system 100 can perform vehicle model collaborative updates according to this method.

[0089] like Figure 2As shown, the vehicle model update method of this embodiment may include the following steps S210 to S260:

[0090] Step S210: Obtain training samples for vehicles.

[0091] The training samples can be generated by vehicle 2001 based on its own operating data, or they can be samples from a general dataset; there is no limitation here.

[0092] In an embodiment where vehicle 2001 generates training samples based on its own operational data, taking the target recognition model used for object recognition of images collected by vehicle 2001 as an example, when vehicle 2001 is running, it inputs images into the first model or the corresponding first application model of the first model for target recognition, and obtains target recognition results. Vehicle 2001 can select highly accurate parts of the target recognition results as training samples based on the correlation between the target recognition results of the image sequence, and execute the method of this embodiment when the number of collected training samples reaches a set number. Here, the first application model is a lightweight model obtained by distilling the first model. The network structure of the first application model is simpler, and the number of model parameters is also smaller. Performing the target recognition task through the first application model is beneficial to improving the processing speed, and training the model through the first model to synchronously update the first application model is beneficial to obtaining better training results.

[0093] When vehicle 2001 generates training samples based on its own operating data, these training samples are more in line with the usage scenario of vehicle 2001. Therefore, training the vehicle model with these training samples will enable the trained model to perform the corresponding tasks more accurately, which will help improve the accuracy of model application.

[0094] Step S220: Train the first model of the vehicle using training samples to obtain the model parameters θ of the first model. 1i The first parameter value at the end of training.

[0095] When vehicle 2001 trains the first model using training samples, if the training meets the set convergence condition, the training ends, and the model parameters θ of the first model are set. 1i The parameter value at the end of this training session is the first parameter value, and the model parameter θ is... 1i The parameter value at the beginning of this training is the second parameter value, which is also the model parameter θ. 1i The initial values ​​used in this training. The convergence condition could be, for example, reaching a set number of iterations, or the loss value being less than or equal to a set threshold; this is not limited here. If this is not the first training iteration, the model parameters θ... 1iThe parameter values ​​at the start of this training are the parameter values ​​after collaborative updates based on the previous training.

[0096] In this embodiment, i takes an integer from 1 to N, and N is the number of model parameters in the first model. For example, N takes a value of 1000, and the model parameters θ 1i This represents any one of the N model parameters. Thus, after training the first model using the training samples, vehicle 2001 will obtain the first parameter value of each of the N model parameters at the end of training.

[0097] Step S230, based on model parameters θ 1i The first parameter value and the model parameter θ 1i The model parameters θ are obtained from the second parameter value at the start of training. 1i The gradient value.

[0098] Model parameters θ 1i The gradient value is also the model parameter θ. 1i The second parameter value and the model parameter θ 1i The difference between the values ​​of the first parameter.

[0099] In this embodiment, after obtaining the first parameter value of each of the N model parameters at the end of training, the vehicle 2001 can further obtain the gradient value of each of the N model parameters.

[0100] Step S240, set the model parameters θ 1i The gradient values ​​are sent to multiple collaborative devices for collaborative model updates.

[0101] See Figure 1 and Figure 2 The collaborative updating of the model by the multi-terminal collaborative device 1000 may include: based on the model parameters θ 1i The gradient value and the corresponding model parameter θ in the common model i The parameter values ​​are used to obtain the vehicle model parameters θ. 1i The third parameter value.

[0102] The common model maintained by the multi-terminal collaborative device 1000 has the same network structure as the first model of vehicle 2001. When performing this collaborative update process on the first model of vehicle 2001, the multi-terminal collaborative device 1000 has already based it on the corresponding model parameters θ provided by multiple vehicles. i The gradient value of the model parameter θ i The parameter values ​​have been updated, that is, the model parameter θ i The parameter values ​​are based on the corresponding model parameters θ provided by multiple vehicles. i The gradient values ​​are updated values ​​for these multiple vehicles in vehicle system 100. For example, such as Figure 1 As shown, multiple vehicles include vehicle 2 and vehicle m, where vehicle 2 provides the corresponding model parameter θ. i The gradient value is the model parameter θ 2i The gradient value, and the corresponding model parameters θ provided by the m-th vehicle. i The gradient value is the model parameter θ mi The gradient value, when the multi-terminal collaborative device 1000 performs this collaborative update process on the first model of vehicle 2001, is related to the model parameter θ. i The parameter values ​​are based on the model parameter θ 2i gradient values ​​and model parameters θ mi The updated gradient value.

[0103] The multi-terminal collaborative device 1000 is based on the corresponding model parameters θ provided by multiple vehicles. i The gradient value updates the model parameters θ i When determining the parameter values, the corresponding model parameters θ provided by the vehicle can be used. i gradient values ​​and model parameters θ i The parameter values ​​are combined with a set weight ratio to obtain the model parameters θ. i The updated parameter values, here, are the corresponding model parameters θ provided by the vehicle. i The gradient value can have a small weight in the fusion to control the degree of influence of the training results of other vehicles on the first model of vehicle 2001.

[0104] In some embodiments, the multi-terminal collaborative device 1000, based on model parameters θ 1i The gradient value and the corresponding model parameter θ in the common model i The parameter values ​​are used to obtain the vehicle model parameters θ. 1i When the third parameter value is obtained, the model parameter θ can be... 1i gradient values ​​and model parameters θ i The parameter values ​​are combined with a set weight ratio; that is, a first weighted calculation is performed with a set weight ratio to obtain the model parameter θ. 1i The third parameter value. In another embodiment, the model parameter θ can also be based on other fusion algorithms. 1i gradient values ​​and model parameters θ i The parameter values ​​are fused, or aggregated, to obtain the model parameters θ. 1i The value of the third parameter is not limited here.

[0105] For example, let the model parameters θ 1i The gradient value in the fusion weights of the first model update is λ, and the model parameters are θ. i If the fusion weight of the parameter values ​​in the first model update is (1-λ), then the model parameter θ1i The value of the third parameter can be expressed as:

[0106]

[0107] In formula 1, Represents the model parameters θ 1i The third parameter value, d 1i Represents the model parameters θ 1i gradient value, Represents the model parameters θ i The parameter value.

[0108] In these embodiments, λ can be a value greater than or equal to 0.5 and less than 1, so that the training results of vehicle 2001 play a dominant role in the update of the first model, while the training results of other vehicles play a secondary role in the update of the first model. In this way, the first model can better highlight the personalized features of vehicle 1002, which is conducive to improving the recognition accuracy of the first model or the first application model in the application.

[0109] In some embodiments, such as Figure 3 As shown, the collaborative updating of the model by the multi-terminal collaborative device 1000 may further include: after obtaining the model parameters θ 1i After obtaining the third parameter value, based on the model parameter θ 1i The gradient value and the corresponding model parameter θ in the common model i The parameter values ​​are used to update the public model. This allows the training results of vehicle 2001 in this training to be integrated into the public model, so that the multi-terminal collaborative device 1000 can perform collaborative updates of other vehicle models after time node t, and / or perform the next collaborative update of its own vehicle model. In other words, the public model can be obtained by the multi-terminal collaborative device 1000 after obtaining the model parameters θ. 1i After the third parameter value, based on the model parameter θ 1i The gradient value and the corresponding model parameter θ in the common model i The parameter values ​​are updated.

[0110] In these embodiments, the multi-terminal collaborative device 1000 can transmit the model parameters θ provided by the vehicle 2001. 1i gradient values ​​and model parameters θ i The parameter values ​​are combined with a set weight ratio; that is, the gradient value and parameter value are calculated with a set weight ratio, and the common model is updated based on the calculation result. In other words, this common model can be obtained by the multi-terminal collaborative device 1000 after obtaining the model parameters θ. 1i After the third parameter value, based on the model parameter θ 1i The gradient value and the corresponding model parameter θ in the common model iThe calculation results of the second weighted calculation of the parameter values ​​are updated, where the model parameter θ 1i The gradient value has a smaller weight in the second weighted calculation than the model parameter θ. i The parameter values ​​have their weights in the second weighted calculation. Here, the model parameters θ provided by vehicle 2001... 1i The gradient value can have a small weight in the fusion process to control the influence of the training results of vehicle 2001 on the model parameter values ​​of the common model. Additionally, the multi-device collaborative device 1000 can also use other fusion algorithms to adjust the model parameters θ. 1i gradient values ​​and model parameters θ i The parameter values ​​are fused to update the common model, but no restrictions are imposed here.

[0111] For example, let the model parameters θ 1i The gradient value is fused into a weight μ in the common model update, and the model parameters θ i If the fusion weight of the parameter values ​​in the public model update is (1-μ), then regarding the model parameter θ i The fusion result, that is, the model parameters θ i The updated parameter value can be represented as:

[0112]

[0113] In formula 2, Represents the model parameters θ i In the updated parameter value, d 1i Represents the model parameters θ 1i gradient value, Represents the model parameters θ i The parameter values ​​before the update. In these embodiments, μ is less than λ and μ is less than 0.5, so as to control the degree of influence of the training results of vehicle 2001 on the model parameter values ​​of the common model, and thus control the degree of influence of the training results of vehicle 2001 on the model updates of other vehicles, so as to reflect the personalized characteristics of the vehicle model, and improve the training effect of the first model of vehicle 2001 by utilizing the training results of other vehicles while ensuring data security.

[0114] Step S250: Receive model parameters θ obtained by multi-terminal collaborative devices through collaborative updates. 1i The third parameter value.

[0115] The model parameter θ 1i The third parameter value is determined by the multi-terminal collaborative device based on the model parameter θ. 1i The gradient value and the corresponding model parameter θ in the common model i The parameter values ​​are obtained. For example, the model parameter θ 1iThe third parameter value is determined by the multi-terminal collaborative device for the model parameter θ. 1i gradient values ​​and model parameters θ i The parameter values ​​are obtained by performing a first weighted calculation.

[0116] In this embodiment, after the multi-terminal collaborative device completes the collaborative update of the first model of vehicle 2001, it updates the model parameters θ obtained through collaborative update. 1i The third parameter value is sent to vehicle 2001 so that vehicle 2001 can use it according to the model parameter θ. 1i The third parameter value is used to update the first model locally.

[0117] Step S260: Update the first model of the vehicle using the third parameter value.

[0118] In step S260, the model parameter θ is used. 1i Updating the first model with the third parameter value can be understood as updating the model parameter θ of the first model. 1i Assign the value to the model parameter θ 1i The third parameter value, that is, the model parameter θ of the first model. 1i The second parameter value at the start of this training is updated to the third parameter value provided by the multi-terminal collaborative device 1000.

[0119] As can be seen from steps S210 to S260 above, in the vehicle model update method of this embodiment, vehicle 2001 trains a first model locally using training samples, and updates the model parameters θ obtained through model training. 1i The gradient values ​​are sent to the multi-terminal collaborative device 1000 for collaborative model updates, which can perform multi-terminal collaborative updates while ensuring the data security of training samples and other data on the vehicle's local device.

[0120] In the vehicle model update method of this embodiment, since the model parameters θ of the common model at one end of the multi-terminal collaborative device 1000 are... i It is jointly maintained and updated by the training results of multiple vehicles. In this way, vehicle 2001 uses the model parameters θ provided by the multi-terminal collaborative device 1000. 1i When the third parameter value is used to update the local first model, the training results of multiple vehicles can be integrated locally to update the vehicle model, thereby improving model performance with higher model update efficiency and economical model update cost.

[0121] In some embodiments, the vehicle 2001 not only has a first model, but also a first application model corresponding to the first model. The first application model is the model used by the vehicle when performing a defined task. The first model is a training model used to update the first application model. Here, the first application model can be a lightweight model obtained by distilling the first model. In these embodiments, when the multi-terminal collaborative device 1000 performs collaborative model updates for the first model of the vehicle 2001, it obtains the vehicle's model parameters θ. 1i After obtaining the third parameter value, the following steps may also be included: based on the vehicle's model parameters θ 1i The third parameter value is used to obtain the model parameter values ​​of the first application model of the vehicle. Correspondingly, in step S240 above, the method calculates the model parameter θ. 1i After the gradient value is sent to the multi-terminal collaborative device 1000 for collaborative model update, the following steps may also be included: the vehicle 2001 receives the parameter values ​​of the model parameters of the first application model obtained by the multi-terminal collaborative device 1000 through collaborative update; wherein, the parameter values ​​of the model parameters of the first application model are determined by the multi-terminal collaborative device according to the model parameter θ. 1i The third parameter value is obtained; and the first application model is updated based on the parameter values ​​of the model parameters of the received first application model.

[0122] In these embodiments, the goal of updating the first application model can be achieved by training the first model, thereby improving the accuracy of the first application model when performing the set tasks in the application.

[0123] In these embodiments, the multi-terminal collaborative device 1000 can obtain the model parameters θ of the vehicle 2001. 1i After determining the third parameter value, the network structure and model parameters θ of the common model are combined. 1i The third parameter value yields the personalized model of vehicle 2001. Then, the personalized model is distilled to obtain a lightweight model of the personalized model. The model parameters of the lightweight model are also the model parameters of the first application model. The parameter values ​​of the lightweight model or the first application model are then sent to vehicle 2001 so that vehicle 2001 can update its local first application model using these parameter values.

[0124] In another embodiment, the model parameter θ may also be received by vehicle 2001. 1i After the third parameter value, first based on the model parameter θ 1i The third parameter value is used to update the first model, and then the first model is distilled to update the first application model, which is not limited here.

[0125] In some embodiments, the training samples obtained in step S210 above are training samples generated by vehicle 2001 based on its own operating data, which is beneficial to improving the accuracy of model application. In these embodiments, taking the first model as an example of the model corresponding to the target recognition task, such as... Figure 4 As shown, before obtaining the training samples of the vehicle in step S210 above, the method further includes the following steps S410 to S440:

[0126] Step S410: Acquire the image captured by the vehicle's camera.

[0127] Step S420: Input the image into the first model or the first application model corresponding to the first model to obtain the detection box information of the detection box corresponding to the target object identified in the image.

[0128] The detection box information includes the location of the detection box in the image and the confidence level of the target object category identified by the detection box. The location information can be represented by the center coordinates of the detection box, as well as its length and width. Since the first model or the first application model can identify multiple categories of target objects, it outputs the confidence levels of the detection boxes that identify the target objects for different categories based on the input image. The category corresponding to the highest confidence level is taken as the category of the target object identified by the detection box. Therefore, the confidence level of the target object category identified by the detection box here is the highest confidence level corresponding to that detection box.

[0129] Step S430: Obtain the detection box information of multiple frames of images within a set time period.

[0130] Step S440: Based on the confidence of each detection box corresponding to the same target object category in multiple frames of images, fit the associated data corresponding to each detection box, and select the effective detection boxes from each detection box based on the fitting results.

[0131] The associated data for each detection box includes the location information p(t) of the detection box in the image, and the acquisition time t of the image containing the detection box. For the same physical object in the external environment of the vehicle, the location information of the target object corresponding to the physical object in multiple frames of images (i.e., image sequence) will change with the acquisition time, and this change should conform to a certain pattern. Therefore, by fitting, effective detection boxes corresponding to the same physical object can be selected.

[0132] In some embodiments, the polynomial p(t) = ∑ can be fitted using the Random Sample Consensus (RANSAC) algorithm. i a i ρ(t0)(t-t0) i+b, t0 represents the time corresponding to the latest frame (most recent) in the multi-frame image, i represents the polynomial exponent, i ranges from 1 to N, where N is the highest power, and can be a preset fixed value, a i Let be the coefficient corresponding to the i-th term, p(t) be the position information of the object in the image at time t, that is, the position information of the detection box in the image at time t, b be the polynomial offset, and b and a be the coefficients. i Determined through fitting.

[0133] In some embodiments, such as Figure 5 As shown, in step S440, the associated data corresponding to each detection box is fitted based on the confidence level of each detection box corresponding to the same target object category in multiple frames of images. This may include the following steps S4411 to S4413:

[0134] Step S4411: In each detection box, randomly select a set number of detection boxes with a confidence level greater than the third threshold as interior points.

[0135] Step S4412: The set curve is obtained by fitting the correlation data of the interior points.

[0136] In this embodiment, an interior point is a point located on a defined curve.

[0137] The set curve is a curve reflecting the change of the target object's position information in the image over time. In step S4412, for example, the above polynomial p(t) = ∑ i a i p(t0)(t-t0) i Fitting with +b yields b and a. i A defined curve.

[0138] Step S4413: If each detection box has an outside point that does not conform to the set curve, filter out the outside point in each detection box, and execute step S4411 again, randomly selecting a set number of detection boxes with a confidence level greater than the third threshold as inside points in each detection box.

[0139] In this embodiment, the outer point is the detection box that is not on the set curve, or in other words, the correlation data of the outer point does not conform to the correlation relationship defined by the set curve.

[0140] Step S4414: End the fitting process if none of the detection boxes have an outlier, and take the remaining detection boxes as valid detection boxes.

[0141] According to step S4414, the interior points on the set curve corresponding to the end of the fitting are the valid detection boxes.

[0142] Step S450: Obtain the highest confidence level corresponding to all valid detection boxes.

[0143] Taking the detection box corresponding to pedestrian A as an example, the clarity of pedestrian A in the image will vary depending on the distance between the vehicle and pedestrian A. When the vehicle and pedestrian A are close, the environmental influence is less, resulting in higher clarity and a higher confidence level for the "pedestrian category" of the effective detection box. However, if the vehicle and pedestrian A are far apart, due to the limited camera pixels and the influence of environmental factors such as weather, the confidence level for the "pedestrian category" of the effective detection box will be relatively lower. In other words, the confidence levels of these effective detection boxes for the "pedestrian category" will differ. In step S450, the highest confidence level of all effective detection boxes for the same target object category can be obtained.

[0144] Step S460: Relabel the confidence of these valid detection boxes with the maximum confidence.

[0145] Regarding the model's recognition results, the confidence levels of these valid detection boxes for the "target object category" may differ. However, in reality, the actual category of the target object identified by these valid detection boxes is "pedestrian category". In this case, these valid detection boxes can be re-labeled according to the highest confidence level among all valid detection boxes, and the re-labeled valid detection boxes can be used as training samples. In this way, the accuracy of the model in performing the target recognition task can be continuously improved.

[0146] Step S470: Based on the confidence level of the relabeled valid detection boxes, obtain the training samples for the vehicles.

[0147] According to steps S410 to S470 above, in the method of this embodiment, the vehicle selects the parts with highly accurate target recognition results as training samples based on the images collected during actual operation and the recognition results of the vehicle model on the images. Since the training samples are more in line with the usage scenario of the vehicle 2001, training the vehicle model with these training samples will enable the trained model to perform the corresponding tasks more accurately, which is beneficial to improving the accuracy of model application.

[0148] Because the detection bounding box data in the vehicle model's image recognition results is large, different detection bounding boxes in the same image may correspond to the same target object, which increases the amount of data processing required to obtain training samples. Therefore, in some embodiments, after step S420 above, where the image is input into the first model or the first application model corresponding to the first model to obtain the detection bounding box information of the target object in the image, and before step S430, where the detection bounding box information of multiple frames within a set time period is obtained, the method may further include the following steps S610 and S620:

[0149] Step S610: If the highest confidence level corresponding to the detection box of the image is greater than the first threshold, the detection box information of the image is subjected to the first processing to obtain the detection boxes of the image that need to be retained for fitting.

[0150] In step S610, the first processing of setting the detection box information of the image includes the following steps S6111 to S6115:

[0151] Step S6111: Filter out detection boxes with confidence scores less than the first threshold from all detection boxes in the image to obtain a candidate set of detection boxes.

[0152] Step S6112: Select the first detection box with the highest confidence from the candidate detection box set.

[0153] Step S6113: Traverse the candidate detection boxes in the detection box set except for the first detection box, and calculate the intersection-union ratio (IoU) between each other detection box and the first detection box.

[0154] In step S6113, the intersection-union ratio is the result of dividing the area of ​​the overlapping region of the two detection boxes being compared by the area of ​​the merged region of the two detection boxes.

[0155] Step S6114: Filter out other detection boxes in the candidate detection box set that make the intersection-union ratio greater than the second threshold, and take out the first detection box from the candidate detection box set and store it in the first set.

[0156] Step S6115: Determine whether the number of detection boxes in the first set has reached the set value. If yes, then the detection boxes in the first set are the detection boxes that need to be retained in the image. If no, then execute step S6112 again to select the first detection box with the highest confidence from the candidate detection box set.

[0157] Step S620: If the maximum confidence level corresponding to the detection box in the image is less than or equal to the first threshold, retain all detection boxes in the image for fitting.

[0158] Through the above steps S610 and S620, the detection boxes of the image can be effectively screened, and a reasonable number of detection boxes can be retained to participate in the subsequent fitting, thereby generating training samples and improving the processing efficiency of generating training samples.

[0159] <Method Example 2>

[0160] Figure 6 This diagram illustrates a flowchart of a multi-terminal collaborative vehicle model update method according to some other embodiments, the method comprising, for example, Figure 1 1000 multi-terminal collaborative devices were implemented.

[0161] like Figure 1 and Figure 6 As shown, in these embodiments, the vehicle model update method may include the following steps S610 to S630:

[0162] Step S610: Receive the model parameters θ of the first model sent by the vehicle. 1i The gradient value.

[0163] In step S610, the model parameter θ 1i The gradient value is determined by the vehicle based on the model parameters θ. 1i The first parameter values ​​and the model parameters θ at the end of the first model training 1i The second parameter value is obtained at the start of the first model training. The vehicle trains the first model locally using training samples to obtain the gradient value.

[0164] Step S620, based on model parameters θ 1i The gradient value and the corresponding model parameter θ in the common model i The parameter values ​​are used to obtain the model parameters θ. 1i The third parameter value.

[0165] See Figure 1 The common model has the same network structure as the first model, and the model parameters θ of the common model are... i The parameter values ​​are based on the corresponding model parameters θ provided by multiple vehicles. i The updated gradient value.

[0166] In some embodiments, the multi-terminal collaborative device 1000 can adjust the model parameters θ 1i The gradient value and the corresponding model parameter θ in the common model i The parameter values ​​are then subjected to a first weighted calculation to obtain the model parameters θ. 1i The third parameter value can be found in the description of the corresponding part of the method embodiment one, and will not be repeated here.

[0167] Step S630, set the model parameters θ 1i The third parameter value is sent to the vehicle to update the first model.

[0168] The vehicle receives the model parameter θ 1i The third parameter value will be used to update the first model.

[0169] According to steps S610 to S630, in the vehicle model update method of this embodiment, the model parameters θ of the common model maintained by the multi-terminal collaborative device 1000 are updated. i It is maintained and updated jointly by the training results of multiple vehicles, thus, based on the model parameters θ i And the model parameters θ of the first model1i Obtain the model parameters θ 1i When the third parameter value is obtained and sent to the vehicle to update the first model, the training results of multiple vehicles can be integrated locally to update the first model of the vehicle, thereby improving model performance with higher model update efficiency and economical model update cost.

[0170] In some embodiments, the multi-terminal collaborative device 1000 obtains the model parameters θ 1i After determining the third parameter value, the method may further include the following steps: based on the model parameter θ 1i The gradient value and the corresponding model parameter θ in the common model i The public model is updated by adjusting the parameter values. For example, the multi-device collaborative device 1000 can update the model parameter θ. 1i The gradient value and the corresponding model parameter θ in the common model i The parameter values ​​are then used for a second weighted calculation to update the common model. This can be found in the description of the corresponding section of Method Implementation Example 1, and will not be repeated here.

[0171] In some embodiments, the multi-terminal collaborative device 1000 obtains the model parameters θ 1i After determining the third parameter value, the method may further include the following steps: based on the model parameter θ 1i The third parameter value is used to obtain the parameter value of the model parameter of the first application model of the vehicle, and the parameter value of the model parameter of the first application model is sent to the vehicle to update the first application model.

[0172] <Device Embodiment>

[0173] This disclosure also provides a multi-terminal collaborative vehicle model update device, such as... Figure 7 As shown, the device 7000 may include a sample acquisition module 7100, a local training module 7200, a gradient calculation module 7300, a sending module 7400, a receiving module 7500, and a model update module 7600.

[0174] The sample acquisition module 7100 is used to acquire training samples of the vehicle and provide them to the local training module 7200.

[0175] The local training module 7200 is used to train the first model of the vehicle using training samples, and to obtain the first parameter values ​​of the model parameters at the end of training.

[0176] The gradient calculation module 7300 is used to obtain the gradient value of the model parameters based on the first parameter value of the model parameters provided by the local training module 7200 and the second parameter value of the model parameters at the beginning of training.

[0177] The sending module 7400 is used to send the gradient values ​​of the model parameters provided by the gradient calculation module 7300 to, for example, Figure 1 The multi-terminal collaborative device 1000 shown performs collaborative updates of the model.

[0178] The receiving module 7500 is used to receive the third parameter value of the model parameters obtained by the multi-terminal collaborative device 1000 through collaborative updates. Here, the third parameter value of the model parameters is obtained by the multi-terminal collaborative device based on the gradient value of the model parameters and the parameter value of the corresponding model parameters in the common model. The common model has the same network structure as the first model, and the parameter value of the model parameters of the common model is the value updated by the multi-terminal collaborative device based on the gradient value of the corresponding model parameters provided by multiple vehicles.

[0179] The model update module 7600 is used to update the first model of the vehicle using a third parameter value.

[0180] In some embodiments, the receiving module 6500 is further configured to receive parameter values ​​of the model parameters of the first application model obtained by the multi-terminal collaborative device through collaborative updating. The model updating module 6600 is further configured to update the first application model using the obtained parameter values ​​of the model parameters of the first application model. The parameter values ​​of the model parameters of the first application model are obtained by the multi-terminal collaborative device based on the third parameter value of the model parameters after obtaining the third parameter value of the vehicle model parameters.

[0181] In some embodiments, the first model is a model corresponding to a target recognition task, the target recognition task including recognizing the location of a target object and recognizing the category of the target object. The device 6000 may further include a sample generation module, which is used to acquire images captured by the vehicle's camera before the sample acquisition module 6100 acquires training samples of the vehicle; input the images into the first model or a first application model corresponding to the first model to obtain detection box information corresponding to the detection boxes of the target objects identified in the images; acquire detection box information of multiple frames of images within a set time period; fit the associated data corresponding to each detection box according to the confidence level of each detection box corresponding to the same target object category in the multiple frames of images, and filter out valid detection boxes from each detection box according to the fitting result; acquire the highest confidence level corresponding to all valid detection boxes; re-label the confidence level of each valid detection box as the highest confidence level; and generate training samples of the vehicle based on the re-labeled confidence levels of the valid detection boxes.

[0182] In some embodiments, the sample generation module is further configured to, before acquiring the detection box information of multiple frames of images within a set time period, perform setting processing on the detection box information of the image if the highest confidence level corresponding to the detection box of the image is greater than a first threshold, to obtain the detection boxes of the image that need to be retained; and, if the highest confidence level corresponding to the detection box of the image is less than or equal to the first threshold, retain all the detection boxes of the image.

[0183] In these embodiments, the sample generation module performs the following processing: filtering out detection boxes with confidence levels less than a first threshold from all detection boxes in the image to obtain a candidate set of detection boxes; selecting a first detection box with the highest confidence level from the candidate set of detection boxes; traversing all other detection boxes in the candidate set of detection boxes except the first detection box, and calculating the intersection-union ratio (IUR) between each other detection box and the first detection box; filtering out other detection boxes in the candidate set that make the IUR greater than a second threshold, and retrieving the first detection box from the candidate set of detection boxes and storing it in a first set; when the number of detection boxes in the first set reaches a set value, selecting the detection boxes in the second set as the detection boxes to be retained in the image; and when the number of detection boxes in the first set is less than the set value, performing the step of selecting the first detection box with the highest confidence level from the candidate set of detection boxes again.

[0184] In some embodiments, when the sample generation module fits the associated data corresponding to each detection box based on the confidence level of each detection box corresponding to the same target object category in the multi-frame images, it can be used to: randomly select a predetermined number of detection boxes with a confidence level greater than a third threshold as inliers in each detection box; fit a predetermined curve using the associated data corresponding to the inliers; wherein the predetermined curve is a curve reflecting the change of the position information of the identified target object in the image over time; if each detection box has outliers that do not conform to the predetermined curve, filter out the outliers in each detection box, and repeat the step of randomly selecting a predetermined number of detection boxes with a confidence level greater than the third threshold as inliers in each detection box; and end the fitting when each detection box does not have the outliers, and take the remaining detection boxes as valid detection boxes.

[0185] <Equipment Example>

[0186] This disclosure also provides a vehicle, in some embodiments, such as Figure 8As shown, the vehicle 8000 includes a vehicle model updating device 7000 according to any embodiment of this disclosure. The modules of the vehicle model updating device 7000 can be implemented by a processor running a computer program, or they can be implemented in conjunction with other hardware; no limitation is made herein.

[0187] In other embodiments, such as Figure 9 As shown, the vehicle 9000 may include a memory 9200 and a processor 9100. The memory 9200 stores a computer program that controls the processor 9100 to operate in order to execute the vehicle model update method according to any method embodiment of this disclosure.

[0188] In other embodiments, a multi-terminal collaborative device is also provided, such as Figure 10 As shown, the multi-terminal collaborative device 100 may include a memory 120 and a processor 110. The memory 120 stores a computer program for controlling the processor 110 to operate and execute the vehicle model update method according to any of the method embodiments of this disclosure.

[0189] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0190] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0191] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0192] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0193] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0194] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0195] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0196] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0197] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A multi-terminal collaborative vehicle model update method, characterized in that, The method, implemented by the vehicle, includes: Obtain training samples for the vehicle; The first model of the vehicle is trained using the training samples to obtain the first parameter values ​​of the model parameters at the end of training. The gradient value of the model parameter is obtained based on the first parameter value of the model parameter and the second parameter value of the model parameter at the beginning of training; The gradient values ​​of the model parameters are sent to a multi-terminal collaborative device for collaborative model updates. The system receives a third parameter value of the model parameters obtained by the multi-terminal collaborative device through the collaborative update; wherein the third parameter value of the model parameters is obtained by the multi-terminal collaborative device based on the gradient value of the model parameters and the parameter value of the corresponding model parameters in the common model, and the common model has the same network structure as the first model. The first model of the vehicle is updated using the third parameter value.

2. The method according to claim 1, characterized in that, The third parameter value of the model parameters is obtained by the multi-terminal collaborative device based on the gradient value of the model parameters and the parameter value of the corresponding model parameters in the common model, including: The third parameter value of the model parameter is obtained by the multi-terminal collaborative device performing a first weighted calculation on the gradient value of the model parameter and the parameter value of the model parameter; wherein, the weight of the gradient value of the model parameter in the first weighted calculation is greater than or equal to the weight of the parameter value of the model parameter in the first weighted calculation.

3. The method according to claim 1, characterized in that, The public model is updated by the multi-terminal collaborative device after obtaining the third parameter value of the model parameters, based on the gradient value of the model parameters and the parameter value of the corresponding model parameter in the public model.

4. The method according to claim 3, characterized in that, The public model is updated by the multi-terminal collaborative device after obtaining the third parameter value of the model parameters, based on the gradient value of the model parameters and the parameter value of the corresponding model parameter in the public model, including: The public model is updated by the multi-terminal collaborative device after obtaining the third parameter value of the model parameter, based on the calculation result of a second weighted calculation of the gradient value of the model parameter and the parameter value of the corresponding model parameter in the public model; wherein, the weight of the gradient value of the model parameter in the second weighted calculation is less than the weight of the parameter value of the model parameter in the second weighted calculation.

5. The method according to claim 1, characterized in that, After sending the gradient values ​​of the model parameters to the multi-terminal collaborative device for collaborative model updates, the method further includes: The system receives parameter values ​​of the model parameters of the first application model obtained by the multi-terminal collaborative device through the collaborative update; wherein, the first application model is the model used by the vehicle when performing a set task, the first model is a training model used to update the first application model, and the parameter values ​​of the model parameters of the first application model are obtained by the multi-terminal collaborative device based on the third parameter value of the model parameters; The first application model is updated based on the parameter values ​​of the model parameters received from the first application model.

6. The method according to any one of claims 1 to 5, characterized in that, The first model is a model corresponding to a target recognition task, wherein the target recognition task includes recognizing the location of a target object and recognizing the category of the target object. Before obtaining the training samples of the vehicle, the method further includes: Acquire images captured by the vehicle's camera; The image is input to the first model or a first application model corresponding to the first model to obtain detection box information of the detection boxes corresponding to the target objects identified in the image; wherein, the first application model is a distillation model of the first model, and the detection box information includes the position information of the detection box in the image and the confidence level of the target object category corresponding to the detection box; Obtain bounding box information for multiple frames of images within a set time period; Based on the confidence scores of each detection box corresponding to the same target object category in the multi-frame images, the associated data corresponding to each detection box is fitted, and valid detection boxes are selected from each detection box based on the fitting results; wherein, the associated data corresponding to the detection box includes the position information of the detection box in the image and the acquisition time of the image where the detection box is located; Obtain the highest confidence level corresponding to all valid bounding boxes; The confidence level of each valid detection frame is relabeled as the highest confidence level; Training samples for the vehicle are generated based on the confidence scores of the relabeled valid detection boxes.

7. The method according to claim 6, characterized in that, Before acquiring the detection bounding box information of multiple frames of images within a set time period, the method further includes: If the highest confidence level corresponding to the detection box of the image is greater than the first threshold, the detection box information of the image is set and processed to obtain the detection boxes of the image that need to be retained. If the highest confidence level corresponding to the detection box in the image is less than or equal to the first threshold, all detection boxes in the image are retained. The setting process includes: Filter out detection boxes with a confidence level less than the first threshold from all detection boxes in the image to obtain a candidate set of detection boxes; From the candidate set of detection boxes, the first detection box corresponding to the highest confidence level is selected; Traverse the candidate detection boxes in the detection box set except for the first detection box, and calculate the intersection-union ratio between each other detection box and the first detection box; Filter out other detection boxes in the candidate detection box set that make the intersection-union ratio greater than the second threshold, and take out the first detection box from the candidate detection box set and store it in the first set; If the number of detection boxes in the first set reaches a set value, the detection boxes in the first set are selected as the detection boxes that need to be retained in the image. If the number of detection boxes in the first set is less than the set value, the step of filtering out the first detection box with the highest confidence from the candidate detection box set is performed again.

8. The method according to claim 6, characterized in that, The step of fitting the associated data corresponding to each detection box based on the confidence level of each detection box corresponding to the same target object category in the multi-frame images includes: In each detection frame, a set number of detection frames with a confidence level greater than the third threshold are randomly selected as interior points; A set curve is obtained by fitting the associated data corresponding to the inlier points; wherein, the set curve is a curve that reflects the change of the position information of the identified target object in the image over time. If each detection box has an outlier that does not conform to the set curve, the outlier is filtered out from each detection box, and the step of randomly selecting a set number of detection boxes with a confidence level greater than the third threshold as inliers is performed again. The fitting process ends when none of the detection boxes have the stated outer point, and the remaining detection boxes are considered valid detection boxes.

9. A multi-terminal collaborative vehicle model update method, characterized in that, include: The gradient values ​​of the model parameters of the first model sent by the vehicle are received; wherein the gradient values ​​of the model parameters are obtained based on the first parameter values ​​of the model parameters at the end of the first model training and the second parameter values ​​of the model parameters at the beginning of the first model training. The third parameter value of the model parameter is obtained based on the gradient value of the model parameter and the parameter value of the corresponding model parameter in the common model; The third parameter value of the model parameters is sent to the vehicle to update the first model.

10. A multi-terminal collaborative vehicle model update device, characterized in that, include: A sample acquisition module is used to acquire training samples of the vehicle; A local training module is used to train a first model of the vehicle using the training samples, and to obtain the first parameter values ​​of the model parameters at the end of training. The gradient calculation module is used to obtain the gradient value of the model parameters based on the first parameter value of the model parameters and the second parameter value of the model parameters at the beginning of training; The sending module is used to send the gradient values ​​of the model parameters to a multi-terminal collaborative device for collaborative model updates. The receiving module receives the third parameter value of the model parameters obtained by the multi-terminal collaborative device through the collaborative update; wherein the third parameter value of the model parameters is obtained by the multi-terminal collaborative device based on the gradient value of the model parameters and the parameter value of the corresponding model parameters in the common model, the common model having the same network structure as the first model, and the parameter value of the model parameters of the common model being the updated value by the multi-terminal collaborative device based on the gradient values ​​of the corresponding model parameters provided by multiple vehicles; and, The model update module is used to update the first model of the vehicle using the third parameter value.

11. A vehicle, characterized in that, The vehicle includes the vehicle model update apparatus of claim 10; or, the vehicle includes a memory and a processor, the memory storing executable instructions for controlling the processor to operate to perform the method according to any one of claims 1-8.

12. A multi-terminal collaborative device, characterized in that, It includes a memory and a processor, the memory storing executable instructions for controlling the processor to operate in order to perform the method according to claim 9.

Images