End-cloud cooperative data mining method, device, system and computer cluster

Abstract

An end-cloud cooperative data mining method, comprising: determining a target text and a task configuration file according to a business requirement by a cloud end; encoding the target text by a text encoder to obtain a text feature; placing the text feature in the task configuration file and issuing it to a vehicle end together; encoding image data by a first picture encoder to obtain an image feature by the vehicle end; calculating a value of a similarity of the text feature and the image feature; determining a target picture according to the value of the similarity and the task configuration file; and uploading the target picture to the cloud end; wherein the first picture encoder is obtained by compressing and optimizing a second picture encoder; and the text encoder and the second picture encoder are two modules of a picture-text multimodal large model. The application is applied to an automatic driving shadow mode, can complete data mining of any interesting target text by using one large model, and does not need to design and develop detection rules for each type of interesting target. The picture encoder of the vehicle end can be continuously updated and optimized, and the model iteration efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving, and in particular to an edge-cloud collaborative data mining method, device, system and computer cluster. Background Technology

[0002] With the development of autonomous driving technology, the gap between sensing and monitoring solutions for autonomous driving is narrowing, and the mass production and deployment of vehicles equipped with autonomous driving algorithms are accelerating. The key to the continuous optimization of autonomous driving algorithms lies in efficiently filtering out the necessary effective data from a large amount of data from mass-produced vehicles to drive the automatic iteration of these algorithms. Currently, the approach to vehicle-side data mining involves designing and developing task models or rules based on the needs of autonomous driving, publishing the task models, and then distributing them to the vehicle for data mining of the corresponding tasks. Task models developed in this way require collecting a large amount of training data and conducting targeted training and optimization. Before testing the task models, a large amount of training data is typically labeled using automatic or manual methods. Testing the task models requires significant human, material, and financial resources. Furthermore, the rules and task models designed in this way are only applicable to specific needs and lack scalability. Summary of the Invention

[0003] To address the aforementioned issues, embodiments of this application provide an edge-cloud collaborative data mining method, system, and computer cluster.

[0004] In a first aspect, embodiments of this application provide an edge-cloud collaborative data mining method, the method comprising: the cloud determining target text and a task configuration file according to business needs; encoding the target text using a text encoder to obtain text features; placing the text features in the task configuration file and sending them to the vehicle terminal; the vehicle terminal encoding image data using a first image encoder to obtain image features; calculating the similarity value between the text features and the image features; determining a target image based on the similarity value and the task configuration file; and uploading the target image to the cloud; wherein the first image encoder is obtained by compressing and optimizing a second image encoder; and the text encoder and the second image encoder are two modules of a large-scale image-text multimodal model.

[0005] Therefore, by leveraging the strong generalization performance of the image-text multimodal large model, the text and image branches of the image-text multimodal large model are deployed separately in the cloud and on the vehicle. The cloud encodes the target text to obtain text features, and the vehicle encodes any image data to obtain image features, realizing end-to-cloud collaborative mining of target images. The first image encoder, which has been compressed and optimized, can meet the needs of running the multimodal large model on resource-constrained devices.

[0006] In one possible implementation, the method further includes the step of image-text multimodal large model compression: the cloud performs large model compression by extracting the supervision information of the second image encoder to obtain the first image encoder.

[0007] Therefore, by compressing the complex large-scale image and text multimodal model in the cloud, a smaller image encoder model with similar performance and simpler structure can be obtained. This reduces the complexity of the network model and meets the requirement of running the large-scale multimodal model on resource-constrained devices. The compressed smaller image encoder model has the same performance as the large image encoder model.

[0008] In one possible implementation, the large model compression includes distillation, quantization, and / or operator tuning.

[0009] Therefore, large model compression can be further achieved through the above-mentioned distillation, quantization and / or operator tuning, simplifying the network model.

[0010] In one possible implementation, the method further includes a step of fine-tuning the parameters of the image-text multimodal large model: inputting labeled samples into the image-text multimodal large model for training, and obtaining fine-tuning parameters when the image-text alignment loss between the second image encoder and the text encoder converges; the fine-tuning parameters include visual fine-tuning parameters and / or text fine-tuning parameters.

[0011] Therefore, the accuracy of large-scale multimodal image and text models can be improved by using a small number of samples and fine-tuning a small number of parameters. This reduces the human, material, and financial costs of training the model. A small number of fine-tuned parameters can be trained to meet different needs, thereby improving data mining accuracy.

[0012] In one possible implementation, the cloud updates the text encoder based on the text fine-tuning parameters to generate new text features.

[0013] Therefore, the accuracy of the text encoder can be improved by adding only a small number of text parameters without changing the text encoder itself.

[0014] In one possible implementation, the task configuration file includes the visual fine-tuning parameters, and the vehicle-mounted device updates the first image encoder based on the visual fine-tuning parameters.

[0015] Therefore, the accuracy of the image encoder can be improved by adding only a small number of visual fine-tuning parameters without changing the image encoder itself.

[0016] In one possible implementation, the task configuration file includes an algorithm name, and the vehicle-mounted device starts the corresponding first image encoder based on the algorithm name.

[0017] Therefore, corresponding mining algorithms can be launched for different tasks, which is scalable.

[0018] In one possible implementation, the task configuration file includes a similarity threshold, and determining the target image based on the similarity value and the task configuration file includes: if the similarity value between the text features and the image features is greater than the similarity threshold, determining the corresponding image data as the target image.

[0019] Therefore, the vehicle can efficiently filter out the required target images from a large amount of data based on the tasks issued by the cloud, realizing data mining through end-to-end cloud collaboration.

[0020] In one possible implementation, the cloud determines the target text and task configuration file according to business requirements, including: the cloud updates the task configuration file according to business requirements, the task configuration file including an updated similarity threshold.

[0021] Therefore, similarity thresholds can be set for different tasks to drive automatic iteration of the algorithm, continuously optimize the algorithm, improve the generalization performance of large models, and have scalability.

[0022] Secondly, embodiments of this application provide an edge-cloud collaborative data mining device, comprising: a demand processing module located in the cloud, which determines target text and task configuration file according to business requirements; encodes the target text using a text encoder to obtain text features; and sends the text features along with the task configuration file to the vehicle terminal. An image processing module located in the vehicle terminal encodes image data using a first image encoder to obtain image features; calculates the similarity between text features and image features; determines a target image based on the similarity value and the task configuration file; and uploads the target image to the cloud. The first image encoder is obtained by compressing and optimizing a second image encoder; the text encoder and the second image encoder are two modules of a large-scale image-text multimodal model.

[0023] In one possible implementation, the edge-cloud collaborative data mining device further includes: a compression module, located in the cloud, which performs compression training by extracting supervised information from the large image encoder model to compress the large model and obtain a first image encoder; a parameter update module, which inputs labeled samples into the image-text multimodal large model for prompt-tuning training, and obtains fine-tuning parameters when the image-text alignment loss between the second image encoder and the text encoder converges; the fine-tuning parameters include visual fine-tuning parameters and / or text fine-tuning parameters; and a model update module, where the cloud updates the text encoder based on the text fine-tuning parameters to generate new text features, and the vehicle updates the first image encoder based on the visual fine-tuning parameters to generate new image features.

[0024] Thirdly, embodiments of this application provide an edge-cloud collaborative data mining system, comprising: a demand processing device, located in the cloud, used to determine target text and task configuration file according to business requirements; to encode the target text using the text encoder to obtain text features; and to send the text features together with the task configuration file to the vehicle terminal; and an image-text processing device, located in the vehicle terminal, used to encode image data using the first image encoder to obtain image features; to calculate the similarity value between the text features and the image features; to determine a target image based on the similarity value and the task configuration file; and to upload the target image to the cloud; wherein the first image encoder is obtained by compressing and optimizing the second image encoder; and the text encoder and the second image encoder are two modules of a large image-text multimodal model.

[0025] Fourthly, embodiments of this application provide a computing device cluster including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method as described in any one of the first aspects.

[0026] Fifthly, embodiments of this application provide a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in any of the first aspects.

[0027] In a sixth aspect, embodiments of this application provide a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method as described in any of the first aspects. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0030] Figure 1 Software architecture diagram provided for embodiments of this application;

[0031] Figure 2 A schematic diagram illustrating the edge-cloud collaborative data mining method provided in an embodiment of this application;

[0032] Figure 3 A schematic diagram of edge-cloud collaborative updates for a small model of an image encoder;

[0033] Figure 4 This is a schematic diagram of large model compression in the edge-cloud collaborative data mining method provided in Embodiment 1 of this application;

[0034] Figure 5 This is a flowchart of the edge-cloud collaborative data mining method according to Embodiment 2 of this application;

[0035] Figure 6 This is a flowchart of the edge-cloud collaborative data mining method according to Embodiment 3 of this application;

[0036] Figure 7 A schematic diagram of the edge-cloud collaborative data mining system provided in the embodiments of this application;

[0037] Figure 8 A schematic diagram of the edge-cloud collaborative data mining device provided in the embodiments of this application;

[0038] Figure 9 This is a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0039] Figure 10 This is a schematic diagram illustrating the connection method between computing device clusters provided in an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.

[0041] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.

[0042] In the description of the embodiments in this application, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple terminals refer to two or more terminals.

[0043] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.

[0044] In the description of the embodiments in this application, "some embodiments" are mentioned, which describe a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0045] In the description of the embodiments of this application, the terms "first, second, third, etc." or module A, module B, module C, etc. are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that, where permitted, a specific order or sequence can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0046] In the description of the embodiments of this application, the reference numerals for the steps, such as S110, S120, etc., do not necessarily indicate that the steps will be executed in this manner. Where permissible, the order of the steps can be interchanged or executed simultaneously.

[0047] Model compression refers to simplifying a large model by utilizing the redundancy of neural network parameters and structure, thereby enabling it to run on resource-constrained devices. This includes two types of constraints: limited memory space and limited computing power. Model compression methods include network pruning, knowledge distillation, parameter quantization, architecture design, and dynamic computation. Resource-constrained devices include mobile phones, watches, aircraft, smart glasses, and autonomous driving systems.

[0048] Knowledge distillation is the process of compressing a large, complex model (teacher) into a smaller, simpler model (student) with similar performance. The reduced complexity of the network model leads to improved performance of the compressed smaller model.

[0049] Parameter quantization uses fewer bits to store values: for example, if 32 bits were used to store parameters, 16 bits can be used instead, which directly reduces the network size by half.

[0050] The prompt technique involves inserting a learnable parameter into the input token, and then fine-tuning the model by training this parameter. When a new task has only a small number of training samples, conventional fine-tuning often leads to overfitting due to the large number of model parameters. The prompt technique can fix the model parameters and add a small number of additional parameters for training.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0052] The first approach involves setting up a target recognition algorithm based on the type of interest target. The algorithm is defined according to the type of interest target, and a corresponding seed trigger is set for each type. This algorithm is deployed on the vehicle. The cloud, based on business needs, triggers the corresponding target recognition algorithm on the vehicle, outputting the interest target data results. After the interest target data results are collected, they are labeled, and the labeled data is input into the corresponding seed trigger for iterative algorithm training, forming a data closed loop for optimizing the target recognition algorithm. The types of interest targets include target vehicles, target non-motorized vehicles and pedestrians, signs, obstacles, traffic signals, etc.

[0053] This approach requires designing and training a target recognition algorithm for each interest target, which necessitates collecting a large amount of training data. The seed trigger deployed on the vehicle-side cannot be updated iteratively, hindering the improvement of the vehicle-side's accuracy in mining that interest target. While the vehicle-side can upgrade the entire seed trigger by updating the model or interface before performing data mining for that interest target, the update process is cumbersome.

[0054] In autonomous driving perception systems, data mining is crucial for continuous capability improvement. The hardware of an autonomous driving perception system includes sensors and processors; the vehicle collects vast amounts of data, including video and image information, through sensors; the processor performs data mining and processing, uploading the mined target data to the cloud; the software includes the operating system, sensor drivers, and various functional units.

[0055] The edge-cloud collaborative data mining method provided in this application is mainly applied to the perception system of autonomous driving. The model is trained in the cloud, and newly added parameters are sent to the vehicle along with the task. These new parameters are then used for data mining of the new task. This method solves the problems of limited accuracy in mining the target of interest on the vehicle side, the need to collect large amounts of training data, and the cumbersome update and upgrade process.

[0056] Figure 1 This is a cloud software architecture diagram provided for an embodiment of this application. Figure 1 As shown, the cloud-based software architecture includes: a data mining unit 11 that obtains target data from a large amount of data according to business needs; a data processing unit 12 that pre-labels or manually labels the target data; and a model pre-training unit 13 that inputs the labeled target data into the target model for pre-training, continuously improving the performance of the target model. The target data includes target text and target images.

[0057] The target model is a large-scale image-text multimodal model, used to measure the image-text relevance between text and image data in image-text information retrieval scenarios. The large-scale image-text multimodal model has two modal branches: image and text. A corresponding text encoder model is set up for the text modality, and a corresponding image encoder model is set up for the image modality.

[0058] A small image encoder model is constructed using a large model compression method. This small model has the same performance as the large image encoder model and is configured on the vehicle side. The small model can be designated as the first image encoder, and the large model as the second image encoder. The first image encoder is a small model obtained by compressing and optimizing the second image encoder.

[0059] Methods for compressing large models include knowledge distillation and parameter quantization.

[0060] Figure 2 This is a schematic diagram illustrating the edge-cloud collaborative data mining method provided in an embodiment of this application. Figure 2 As shown, the text encoder 21 is configured on the cloud processor (hereinafter referred to as the cloud), and the vehicle-side image encoder mini-model is configured on the vehicle-side processor (hereinafter referred to as the vehicle-side). The method includes: the cloud determines the target text and task configuration file according to business requirements, and encodes the target text to obtain text features through the text encoder 21; the text features and task configuration file are sent to the vehicle-side; the image data is encoded by the image encoder mini-model 22 on the vehicle-side to obtain image features; the similarity value between the text features and the image features is calculated; the target image is determined according to the similarity and task configuration file; and the target image is uploaded to the cloud. Among them, the image encoder mini-model is obtained by compressing and optimizing the image encoder large model; the text encoder and the image encoder large model are two modules of the image-text multimodal large model.

[0061] The edge-cloud collaborative data mining method provided in this application embodiment is described in detail below. The edge-cloud collaborative data mining method provided in this application embodiment includes the following steps S21-S27.

[0062] S21, the cloud determines the target text and task configuration file based on business needs.

[0063] Business requirements are business interests or scenarios, such as vehicles, non-motorized vehicles and pedestrians, signs, obstacles, traffic signal glare, rainy days, etc.

[0064] For example, if the business requirement is to find images of sun glare, the cloud determines the target text as "A photo of a sun glare" based on this business requirement.

[0065] The task configuration file is set according to business requirements. The task-related configurations set in the task configuration file include the similarity threshold, the name of the algorithm to be activated, and fine-tuning parameters. The similarity threshold is used to ensure the accuracy of image-text alignment in data mining; the similarity between the image features of the target image and the text features of the target text must be greater than the set similarity threshold. The algorithm name indicates which algorithm to activate on the vehicle side. Fine-tuning parameters are used to update the parameters of the text encoder and / or the first image encoder.

[0066] S22, Text features are obtained by encoding the target text through text encoder 21.

[0067] The text encoder 21 is used to encode a variable-length target text into a fixed-length feature vector, which is denoted as the text feature.

[0068] S23, the text features are placed in the task configuration file and sent to the vehicle.

[0069] The task configuration file includes a similarity threshold; the cloud sends the text features obtained in step S22 and the similarity threshold together to the vehicle.

[0070] In some feasible implementations, the task configuration file also includes the name of the algorithm to be started, such as the large model of the image encoder, the small model of the image encoder, etc.

[0071] S24, the vehicle-mounted image encoder mini-model 22 is activated according to the task requirements. This image encoder mini-model 22 encodes the image data received by the sensor into image features. Specifically, it includes the following steps:

[0072] S241, the vehicle receives the task configuration file sent from the cloud, parses the task configuration file to obtain the name of the algorithm to be started, the similarity threshold and text features; and starts the vehicle mining algorithm according to the algorithm name.

[0073] The vehicle-side mining algorithm can be the image encoder small model 22 in this application, or it can be other image or radar mining algorithms, such as the non-standard vehicle mining algorithm, etc.

[0074] S242, the vehicle obtains image data through sensors.

[0075] Sensors include in-vehicle cameras, radar, etc. The vehicle acquires image data through its in-vehicle cameras.

[0076] S243, the image data is encoded by the small image encoder model 22 to obtain the corresponding image features.

[0077] Image encoder mini-model 22 is used to encode arbitrary image data into a fixed-length feature vector; the feature vector corresponding to the image data can be denoted as image features.

[0078] The image data obtained in step 242 is input into the small image encoder model 22 for encoding, and the corresponding image features are output. The image features are Y = (y1, y2, y3, ..., yn). The length of the image features is the same as the length of the text features, which is n.

[0079] S25, calculate the similarity between image features and text features, and obtain the similarity value.

[0080] Similarity refers to the degree of similarity between two things. It is obtained by calculating the distance between the feature vectors of the two things. If the distance between the features of the two things is small, the similarity is large; if the distance between the features of the two things is large, the similarity is small.

[0081] For text features X = (x1, x2, x3, ..., xn) and image features Y = (y1, y2, y3, ..., yn), the similarity can be evaluated using cosine similarity, which is the cosine value of the angle between the two vectors.

[0082] S26, Determine the target image based on similarity and similarity threshold.

[0083] The similarity value is compared with the similarity threshold. If the similarity value is greater than or equal to the similarity threshold, the image is determined to be the target image; if the similarity value is less than the similarity threshold, the image is discarded.

[0084] For example, the similarity threshold obtained by parsing the task configuration file on the vehicle side is 0.8; the similarity between image feature X = (x1, x2, x3, ..., xn) and text feature Y = (y1, y2, y3, ..., yn) calculated by the cosine similarity method is 0.86. Since the similarity is greater than the similarity threshold, the image data corresponding to this image feature is the target image.

[0085] S27, upload the target image to the cloud.

[0086] To improve the accuracy of data mining, the cloud uses a prompt tuning method to efficiently fine-tune parameters and sends the new parameters to the vehicle along with the task for data mining of new tasks.

[0087] Cue-based learning methods are used to improve model accuracy with a small number of samples and minimal parameter fine-tuning. Employing cue-based learning for efficient parameter fine-tuning allows for the addition of a small number of parameters to enhance model accuracy without altering the text and image encoders.

[0088] Figure 3 This is a schematic diagram illustrating edge-cloud collaborative updates for a small model of an image encoder. (Example) Figure 3 As shown, the cloud uses a small number of samples or mined target images to train a text encoder and a second image encoder through prompting learning, obtaining text fine-tuning parameters and / or visual fine-tuning parameters. The visual fine-tuning parameters are added to the task configuration file and sent to the vehicle; the vehicle adds the visual fine-tuning parameters to the image encoder mini-model 22, while the cloud can also add text fine-tuning parameters to the text encoder to generate new text features.

[0089] The model optimization and update method provided in this application embodiment includes the following steps S31-SS36.

[0090] S31, the cloud-based text encoder and second image encoder are optimized and trained through prompting learning to obtain text fine-tuning parameters and visual fine-tuning parameters.

[0091] Training data can be target images uploaded by the vehicle or a small number of labeled training sample images from a database.

[0092] S32, the visual fine-tuning parameters are placed in the task configuration file and sent to the vehicle. In some feasible implementations, the task configuration file also includes a new similarity threshold.

[0093] S33, the vehicle receives the task configuration file, parses the task configuration file to obtain visual fine-tuning parameters and / or new similarity thresholds.

[0094] S34, add the visual fine-tuning parameters to the image encoder mini-model 22, update the image encoder mini-model 22, and obtain the optimized image encoder mini-model 22.

[0095] When processing new image data received by the sensor, the corresponding image features are obtained by encoding with an optimized image encoder small model 22. The similarity between the image features and the text features encoded by the cloud model is calculated, and the image data of interest is mined according to whether the similarity is greater than a new similarity threshold.

[0096] Example 1

[0097] The edge-cloud collaborative data mining method provided in Embodiment 1 of this application compresses and optimizes a large, complex multimodal image-text model in the cloud to obtain a smaller image encoder model 22 with similar performance and a simpler structure, thereby reducing the complexity of the network model. Then, the text encoder 21 is deployed in the cloud, and the image encoder model 22 is deployed on the vehicle end for edge-cloud collaborative data mining training.

[0098] Figure 4This is a schematic diagram of large model compression in the edge-cloud collaborative data mining method provided in Embodiment 1 of this application. Figure 4 As shown, large model compression includes the following steps S41-S42.

[0099] S41, compress and optimize the large image encoder model in the large image-text multimodal model to obtain the small image encoder model 22.

[0100] The knowledge distillation method can be used to extract the supervision information of the large image encoder model in the large image-text multimodal model, and construct a small image encoder model 22. The small image encoder model 22 has the same performance as the large image encoder model.

[0101] S42, input the labeled small number of samples into the large image-text multimodal model for fine-tuning. Calculate the cross-entropy loss based on the similarity between the text features output by the text encoder 21 and the image features output by the image encoder small model 22, and the labeled categories. If the cross-entropy loss converges, obtain the fine-tuned text encoder 21 and image encoder small model 22.

[0102] In some feasible implementations, parameter quantization can be used to fine-tune and optimize the small image encoder model 22.

[0103] The steps for edge-cloud collaborative data mining based on the small image encoder model 22 obtained after compression are as follows: S43-S46.

[0104] S43. Based on business requirements, determine the target text to be mined and the mining algorithm to be used.

[0105] S44, the finely tuned text encoder 21 and image encoder mini-model 22 are deployed separately, with the text encoder 21 deployed in the cloud and the image encoder mini-model 22 deployed on the vehicle.

[0106] S45, the cloud sets similarity thresholds according to business needs, and writes the mining algorithm name and similarity threshold into the task configuration file.

[0107] For example, the business requirement could be to mine solar glare, with a similarity threshold of 0.80. The algorithm name "General Text Retrieval" and the similarity threshold of 0.80 would be written into the task configuration file.

[0108] S46. Perform edge-cloud collaborative data mining based on new business needs. For specific implementation details, please refer to steps S21-S27. Further details will not be provided here.

[0109] Example 2

[0110] Figure 5 This is a flowchart of the edge-cloud collaborative data mining method according to Embodiment 2 of this application. Figure 5 As shown, based on the above embodiment 1, the text encoder 21 and the image encoder mini-model 22 of the large-scale multimodal image-text model are deployed separately. The text encoder 21 is deployed in the cloud, and the image encoder mini-model 22 is deployed on the vehicle. The image encoder mini-model 22 is obtained by compressing the large-scale image encoder model. The edge-cloud collaborative data mining method includes the following steps:

[0111] S51, Determine business requirements, and determine the target text based on business requirements.

[0112] Business needs can be the target of exploration.

[0113] For example, if the cloud determines the target of interest as a garbage bag on the road based on the current autonomous driving model corner case scenario, then the target text is "a photo of garbage bag on the road".

[0114] S52, input the target text determined in step S51 into text encoder 21, and text encoder 21 outputs the corresponding text features.

[0115] The text encoder 21 is the text module of the multimodal graph-text large model.

[0116] For example, the cloud inputs the target text "a photo of garbage bag on the road" into the text encoder 21 to generate a text vector X = (x1, x2, x3, ..., xn) with a length of 512 bits, where x1, x2, x3, ..., xn are natural numbers.

[0117] S53: The cloud distributes the task configuration file to the vehicle. The task configuration file includes text features, the name of the algorithm to be activated, and the similarity threshold.

[0118] For example, the cloud determines the algorithm to be activated as "General Text Retrieval" based on the business requirement of digging out garbage bags on the road. The similarity value between the target image and the target text must be greater than the set similarity threshold of 0.80. The cloud sends a 512-bit text vector X = (x1,x2,x3,……,xn), the algorithm name "General Text Retrieval", and 0.80 to the vehicle.

[0119] S54: The vehicle receives the task configuration file sent to it, parses the task configuration file to obtain text features, the name of the algorithm to be started, and the similarity threshold.

[0120] For example, the algorithm to be started is named "General Text Retrieval", the similarity threshold is 0.80, and the text vector is X = (x1, x2, x3, ..., xn).

[0121] S55: The vehicle can acquire frame-by-frame image data through sensors, and then input it into the small image encoder model 22 for encoding to obtain a fixed-length feature vector corresponding to each frame of image data.

[0122] For example, the vehicle-mounted system inputs the image data collected by the sensors frame by frame into an image encoder named "General Text Retrieval" for encoding, thereby obtaining the image features corresponding to each frame of image data. Each frame of image features is a fixed-length feature vector with a length of n.

[0123] S56, calculate the similarity between the image features and text features X for each frame, and obtain the similarity value between the image features and text features for each frame.

[0124] For example, the similarity between the image features Y1, Y2, ..., Y100 corresponding to each frame of image data and the text feature X is calculated to obtain the similarity value between the image features and the text features of each frame.

[0125] S57, compare the similarity value of each frame of image data with the similarity threshold, and obtain multiple target images if the similarity value is greater than the similarity threshold.

[0126] For example, the similarity values ​​of each frame of image data are: 0.71, 0.74, 0.61, 0.88, ..., 0.87; the similarity threshold is 0.80. The similarity value of each frame of image data is compared with the similarity threshold. Image data with a similarity value greater than 0.80 is the target image, and there are multiple target images.

[0127] S58 packages and uploads multiple target images with a similarity greater than a similarity threshold within a certain time period to the cloud.

[0128] The edge-cloud collaborative data mining method provided in Embodiment 2 of this application is based on a large image-text multimodal model. It utilizes the strong generalization performance of the large image-text multimodal model to separately deploy the text and image branches of the large image-text multimodal model on the cloud and the vehicle. The cloud encodes the target text to obtain text features, and the vehicle encodes any image data to obtain image features. The target image is mined based on the text features, image features, and similarity threshold.

[0129] Example 3

[0130] Figure 6 This is a flowchart of the edge-cloud collaborative data mining method according to Embodiment 3 of this application. Figure 6As shown, the edge-cloud collaborative data mining method provided in Embodiment 3 of this application combines model compression and prompt technology in the cloud to accurately and effectively adjust new task-specific parameters using the mined target images, thereby improving the accuracy of mining target images; and pushes them to the vehicle end to adapt to new mining tasks.

[0131] S61 uses a prompting tuning method in the cloud for efficient parameter fine-tuning, adding one or more text fine-tuning parameters and / or image fine-tuning parameters without changing the large model of the text encoder and the small model of the image encoder.

[0132] The cloud-based approach employs a prompting tuning method, utilizing the mined target images to efficiently fine-tune the pre-trained model and obtain fine-tuned parameters that meet the specific task requirements.

[0133] For example, the target image mined might be a picture of a large garbage bag full of trash on the roadside, which doesn't match the target text "aphoto of garbage bag on the road" well. To improve the matching degree or adapt to new mining tasks, the cloud uses a prompting learning method to efficiently fine-tune the image-text multimodal model and obtain one or more relevant text fine-tuning parameters and image fine-tuning parameters that better match the target image.

[0134] S62 generates new text features based on one or more text fine-tuning parameters and adds the new text features to the task configuration file.

[0135] S63 adds one or more image fine-tuning parameters to the task configuration file and sends it to the vehicle. The task configuration file containing one or more image fine-tuning parameters and / or new text features can be designated as the second task configuration file.

[0136] By introducing new fine-tuning parameters, classification accuracy can be readjusted. In some feasible implementations, a new similarity threshold can be set according to specific task requirements, and the new similarity threshold, algorithm name, and one or more visual fine-tuning parameters can be added to a second task configuration file.

[0137] S64, the vehicle receives the new task configuration file, parses the new task configuration file, and obtains visual fine-tuning parameters and / or a new similarity threshold.

[0138] S65, replace or add the visual fine-tuning parameters to the image encoder mini-model 22 to obtain the optimized image encoder mini-model 22.

[0139] S66, the new image data received by the sensor is encoded using the optimized image encoder small model 22 to obtain the second image features.

[0140] S67, calculate the similarity between the second image features and the new text features, and determine the new target image data based on whether the similarity is greater than the new similarity threshold. The accuracy of the new target image data is improved.

[0141] In the new round of data mining tasks, the cloud can instruct the vehicle-side to enable the algorithm name of the small image encoder model based on business requirements. Alternatively, the cloud can use a prompting tuning method to efficiently fine-tune the parameters, adding one or more tuning parameters without changing the architecture of the large text encoder and image encoder models. These tuning parameters are then sent to the vehicle-side along with the task configuration file for use in this target data mining task.

[0142] The edge-cloud collaborative data mining method of this application embodiment is applied to the shadow mode of autonomous driving. It can realize the data mining of text with arbitrary interest targets using a large model, without the need to design and develop detection rules for each type of interest target.

[0143] The edge-cloud collaborative data mining method of this application embodiment can continuously update and optimize the image encoder on the vehicle side, thereby improving the model iteration efficiency.

[0144] The implementation of this application embodiment relies on the hardware and software system of an autonomous vehicle. The hardware system includes sensors, processors, etc., in the perception system. The software system includes the operating system, sensor drivers, and processing programs for each system.

[0145] Figure 7 A schematic diagram of an edge-cloud collaborative data mining system is provided for embodiments of this application. For example... Figure 7 As shown, the system includes: a demand processing device 71, located in the cloud, which determines the target text and task configuration file based on business requirements; encodes the target text using a text encoder to obtain text features; and sends the text features along with the task configuration file to the vehicle terminal. An image processing device 72, located in the vehicle terminal, encodes image data using a first image encoder to obtain image features; calculates the similarity between the text features and the image features; determines the target image based on the similarity value and the task configuration file; and uploads the target image to the cloud.

[0146] Both the demand processing device 71 and the image and text processing device 72 can be implemented in software or in hardware. For example, the implementation of the demand processing device 71 will be described below. Similarly, the implementation of the image and text processing device 72 can refer to the implementation of the demand processing device 71.

[0147] As an example of a software functional unit, the demand processing device 71 may include code running on a computing instance. The computing instance may be at least one of a physical host (computing device), a virtual machine, a container, or other computing devices. Further, the aforementioned computing device may be one or more. For example, the demand processing device 71 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the application may be distributed in the same region or in different regions. The multiple hosts / virtual machines / containers used to run the code may be distributed in the same Availability Zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0148] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a single region. Communication between two VPCs within the same region, and between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0149] As an example of a hardware functional unit, the demand processing device 71 may include at least one computing device, such as a server. Alternatively, the demand processing device 71 may also be a device implemented using an ASIC or a PLD. The PLD may be implemented using a CPLD, FPGA, GAL, or any combination thereof.

[0150] The demand processing unit 71 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the demand processing unit 71 includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the demand processing unit 71 includes multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0151] Figure 8 This is a schematic diagram of an edge-cloud collaborative data mining device provided in an embodiment of this application. Figure 8 As shown, it includes:

[0152] The requirement processing module 81 is located in the cloud. The requirement processing module 81 determines the target text and task configuration file according to business requirements; encodes the target text to obtain text features through a text encoder; and sends the text features to the vehicle terminal along with the task configuration file.

[0153] The image processing module 82 is installed on the vehicle end. The image processing module 82 encodes the image data through the first image encoder to obtain image features; calculates the similarity value between the text features and the image features; determines the target image based on the similarity value and the task configuration file; and uploads the target image to the cloud.

[0154] The edge-cloud collaborative data mining device of this application also includes: a compression module 83, which is set in the cloud. The compression training module is used to compress the large model by extracting the supervision information of the large model of the image encoder to obtain the first image encoder.

[0155] The parameter update module 84 inputs the labeled samples into the image-text multimodal large model for prompting training. When the image-text alignment loss between the second image encoder and the text encoder converges, the fine-tuning parameters are obtained. The fine-tuning parameters include visual fine-tuning parameters and / or text fine-tuning parameters.

[0156] Model update module 85: The cloud updates the text encoder based on the text fine-tuning parameters to generate new text features; the vehicle updates the first image encoder based on the visual fine-tuning parameters to generate new image features.

[0157] Among them, the demand processing module 81, the image and text processing module 82, the compression module 83, the parameter update module 84, and the model update module 85 can all be implemented by software or by hardware.

[0158] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0159] like Figure 9 As shown, the computing device cluster includes at least one computing device 100. The computing device 100 includes a bus 102, a processor 104, a memory 106, and a communication interface 108. The processor 104, memory 106, and communication interface 108 communicate with each other via the bus 102. The computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 100.

[0160] Bus 102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 102 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 102 may include a path for transmitting information between various components of the computing device 100 (e.g., memory 106, processor 104, communication interface 108).

[0161] The processor 104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0162] Memory 106 may include volatile memory, such as random access memory (RAM). Memory 106 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0163] The memory 106 stores executable program code, and the processor 104 executes the executable program code to implement the functions of the aforementioned processing module 81, compression module 83 and / or parameter update module 84, thereby realizing the edge-cloud collaborative data mining method.

[0164] Alternatively, the memory 106 stores executable code, and the processor 104 executes the executable code to implement the functions of the aforementioned image and text processing module 82 and model update module 85, thereby realizing the edge-cloud collaborative data mining method.

[0165] The communication interface 103 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 100 and other devices or communication networks.

[0166] The memory 106 of one or more computing devices 100 in a computing device cluster may contain the same instructions for executing edge-cloud collaborative data mining methods.

[0167] In some possible implementations, the memory 106 of one or more computing devices 100 in the computing device cluster may also store partial instructions for executing the edge-cloud collaborative data mining method. In other words, a combination of one or more computing devices 100 can jointly execute instructions for executing the edge-cloud collaborative data mining method.

[0168] It should be noted that the memory 106 in different computing devices 100 within the computing device cluster can store different instructions, which are used to execute parts of the functions of the demand processing device 71 or the image processing device 72, respectively. That is, the instructions stored in the memory 106 of different computing devices 100 can implement the functions of one or more modules among the demand processing module 81, compression module 83, parameter update module 84, or image processing module 82 and model update module 85.

[0169] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0170] Figure 10 One possible implementation is shown. For example... Figure 10 As shown, the two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this possible implementation, the memory 106 in computing device 100A stores instructions for executing the functions of the demand processing module 81. Simultaneously, the memory 106 in computing device 100B stores instructions for executing the functions of the image processing module 82.

[0171] Figure 10 The connection method between the computing device clusters shown can be as follows: considering that the edge-cloud collaborative data mining method provided in this application requires a large amount of data storage and model optimization and fine-tuning, the functions implemented by the compression module 83 and the parameter update module 84 are to be executed by the computing device 100A, and the model update module 85 is to be executed by the computing device 100B.

[0172] It should be understood that Figure 10 The functions of the computing device 100A shown can also be performed by multiple computing devices 100. Similarly, the functions of the computing device 100B can also be performed by multiple computing devices 100.

[0173] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute an edge-cloud collaborative data mining method.

[0174] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to execute an edge-cloud collaborative data mining method.

[0175] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cloud-edge collaborative data mining method, characterized in that, The method includes: A large-scale image-text multimodal model is deployed, comprising a text encoder, a first image encoder, and a second image encoder; wherein the text encoder and the second image encoder are configured in the cloud; the first image encoder is configured on the vehicle end; the first image encoder is obtained by compressing and optimizing the second image encoder; the large-scale model compression includes distillation, quantization, and / or operator optimization. The cloud determines the target text and task configuration file based on business needs; the target text is encoded by a text encoder to obtain text features; and the second image encoder is trained through prompting learning to obtain visual fine-tuning parameters. The text features and the visual fine-tuning parameters are placed together in the task configuration file and sent to the vehicle terminal; the task configuration file includes the algorithm name; The vehicle-mounted device starts the corresponding first image encoder according to the algorithm name; updates the first image encoder according to the visual fine-tuning parameters; encodes image data through the first image encoder to obtain image features; calculates the similarity value between the text features and the image features; determines the target image according to the similarity value and the task configuration file; and uploads the target image to the cloud. The method also includes the step of fine-tuning the parameters of the large multimodal model: The labeled samples are input into the image-text multimodal large model for training. When the image-text alignment loss between the second image encoder and the text encoder converges, fine-tuning parameters are obtained. The fine-tuning parameters include visual fine-tuning parameters and / or text fine-tuning parameters.

2. The method according to claim 1, characterized in that, The method also includes the step of compressing the large multimodal image-text model: The cloud-based system extracts the supervision information from the second image encoder to perform large model compression, thereby obtaining the first image encoder.

3. The method according to claim 1, characterized in that, The cloud platform updates the text encoder based on the text fine-tuning parameters to generate new text features.

4. The method according to claim 1, characterized in that, The task configuration file includes a similarity threshold, and the step of determining the target image based on the similarity value and the task configuration file includes: If the similarity between the text features and the image features is greater than the similarity threshold, the corresponding image data is determined to be the target image.

5. The method according to claim 1, characterized in that, The cloud platform determines the target text and task configuration file based on business requirements, including: The cloud platform updates the task configuration file according to business needs, and the task configuration file includes an updated similarity threshold.

6. A cloud-edge collaborative data mining device, characterized in that, The device includes: The large image-text multimodal model includes a text encoder, a first image encoder, and a second image encoder; wherein the text encoder and the second image encoder are configured in the cloud; the first image encoder is configured on the vehicle side; the first image encoder is obtained by compressing and optimizing the second image encoder; the large model compression includes distillation, quantization, and / or operator optimization. The requirement processing module, located in the cloud, is used to determine the target text and task configuration file according to business requirements; to encode the target text using the text encoder to obtain text features; to train the second image encoder through prompting learning to obtain visual fine-tuning parameters; and to send the text features together with the task configuration file to the vehicle terminal; the task configuration file includes visual fine-tuning parameters and algorithm name. The image processing module, located on the vehicle end, is used to update the first image encoder according to the visual fine-tuning parameters; start the corresponding first image encoder according to the algorithm name; encode image data through the first image encoder to obtain image features; calculate the similarity value between the text features and the image features; determine the target image according to the similarity value and the task configuration file; and upload the target image to the cloud. The device also includes a step for fine-tuning the parameters of the large image-text multimodal model: The labeled samples are input into the image-text multimodal large model for training. When the image-text alignment loss between the second image encoder and the text encoder converges, fine-tuning parameters are obtained. The fine-tuning parameters include visual fine-tuning parameters and / or text fine-tuning parameters.

7. The apparatus according to claim 6, characterized in that, Also includes: A compression module, located in the cloud, is used to compress a large model by extracting the supervision information of the second image encoder to obtain the first image encoder.

8. The apparatus according to claim 6, characterized in that, Also includes: The model update module, in which the cloud updates the text encoder based on the text fine-tuning parameters to generate new text features; The vehicle-mounted device updates the first image encoder based on the visual fine-tuning parameters to generate new image features.

9. A cloud-edge collaborative data mining system, characterized in that, The system includes: The large image-text multimodal model includes a text encoder, a first image encoder, and a second image encoder; wherein the text encoder is configured in the cloud; the first image encoder is configured on the vehicle; the first image encoder is obtained by the cloud through large model compression and optimization by extracting the supervision information of the second image encoder; the large model compression includes distillation, quantization, and / or operator optimization. A demand processing device, located in the cloud, is used to determine target text and task configuration file according to business needs; encode the target text using the text encoder to obtain text features; train the second image encoder through prompting learning to obtain visual fine-tuning parameters; and send the text features together with the task configuration file to the vehicle terminal; the task configuration file includes visual fine-tuning parameters and algorithm name. An image processing device is installed at the vehicle end. The image processing device is used to update the first image encoder according to the visual fine-tuning parameters; start the corresponding first image encoder according to the algorithm name; encode image data through the first image encoder to obtain image features; calculate the similarity value between the text features and the image features; determine the target image according to the similarity value and the task configuration file; and upload the target image to the cloud. The system also includes a step for fine-tuning the parameters of the large multimodal graph model: The labeled samples are input into the image-text multimodal large model for training. When the image-text alignment loss between the second image encoder and the text encoder converges, fine-tuning parameters are obtained. The fine-tuning parameters include visual fine-tuning parameters and / or text fine-tuning parameters.

10. A computing device cluster, characterized in that, It includes at least one computing device, said at least one computing device including a processor and a memory; The memory of the at least one computing device is used to store instructions; The processor of the at least one computing device executes instructions stored in the memory of the at least one computing device to cause the at least one computing device to implement the method as described in any one of claims 1-5.

11. A computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1-5.

Images