Visual model upgrading method, electronic device, storage medium and program product

Abstract

The present invention discloses a visual model upgrading method, electronic equipment, storage medium and program product, which relate to the field of artificial intelligence technology. The method comprises generating training samples according to image samples and image detection results, updating a database using the training samples, identifying the update time of the visual model, training the visual model using the training samples in the database when the number of training samples is greater than a quantity threshold, and upgrading the visual model of the target terminal using the trained visual model. The number of training samples reaches the expected quantity threshold, and the visual model can be used to perform image analysis to replace the manual verification process, and the model training is automatically performed. The method solves the technical problems of low model accuracy caused by the small number of training samples in related technologies, and manual collection and annotation of images are required for each training and iterative upgrade, which consumes a lot of manpower and material resources. The method achieves the technical effects of improving model recognition accuracy, saving labor costs and improving training efficiency.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a visual model upgrading method, electronic equipment, storage medium and program product. Background Art

[0002] With the development of artificial intelligence (AI) technology, more and more venues are undergoing intelligent transformation and deploying smart surveillance systems. These systems cover application scenarios such as fire and smoke detection, crowd detection, and electronic fencing, aiming to improve safety and operational efficiency. AI (Artificial Intelligence) models, as the foundation of deep learning applications such as CNNs (Convolutional Neural Networks) and RNNs (Recurrent Neural Networks), are of undeniable importance.

[0003] However, due to differences in environments, data characteristics, and device performance across different locations, traditional single models struggle to maintain high accuracy and efficiency in all scenarios. Therefore, continuous model training and iterative upgrades have become essential features of edge management software platforms. Traditional AI vision model training requires manual collection and labeling of images from different scenes, followed by training and upgrades. Each iteration and upgrade consumes significant manpower and resources, and this iteration is continuous. Acquiring sufficient training samples is often difficult. In actual production environments, the initially deployed model often fails to meet application requirements due to insufficient samples, necessitating subsequent model training and upgrades. Summary of the Invention

[0004] The present invention provides a visual model upgrading method, electronic device, storage medium and program product, so as to at least solve the technical problems in the related art that each training and iterative upgrade requires manual collection and labeling of pictures, which consumes a lot of manpower and material resources, and the small number of training samples leads to low model accuracy.

[0005] The present invention provides a visual model upgrading method, in which the visual model is deployed on a target terminal, and the visual model outputs image detection results of image samples. The method is applied to a server, and the method includes: obtaining image samples and image detection results uploaded by the target terminal; generating training samples according to the image samples and the image detection results, updating a database according to the training samples, and identifying the update time of the visual model; obtaining the number of training samples newly added to the database since the update time, and when the number of training samples is greater than a quantity threshold, training the visual model using the training samples in the database, and upgrading the visual model of the target terminal using the trained visual model.

[0006] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any one of the above-mentioned visual model upgrading methods when executing the computer program.

[0007] The present invention also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned visual model upgrading methods are implemented.

[0008] The present invention also provides a computer program product, comprising a computer program, which implements the steps of any of the above-mentioned visual model upgrading methods when executed by a processor.

[0009] Through the present invention, since training samples can be generated based on image samples and image detection results, the database can be updated using the training samples, and the update time of the visual model can be identified, when the number of training samples is greater than the quantity threshold, the visual model is trained using the training samples in the database, and the visual model of the target terminal is upgraded using the trained visual model, so that the number of training samples reaches the desired quantity threshold, and the visual model can be used to perform image analysis to replace the manual verification process, and the model training is automatically performed. Therefore, it can solve the technical problems of low model accuracy caused by the small number of training samples in related technologies, and manual collection and labeling of images are required for each training and iterative upgrade, which consumes a lot of manpower and material resources, and achieve technical effects such as improving model recognition accuracy, saving labor costs and improving training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0011] Figure 1 A flowchart of a visual model upgrading method provided by an embodiment of the present invention;

[0012] Figure 2 A system flow chart provided for one embodiment of the present invention;

[0013] Figure 3 A schematic diagram of a multimodal pre-trained language model sample verification process provided by one embodiment of the present invention;

[0014] Figure 4 A schematic diagram of a multimodal pre-trained language model negative sample enhancement process according to one embodiment of the present invention;

[0015] Figure 5A block diagram of a visual model upgrading device provided by an embodiment of the present invention;

[0016] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0018] It should be noted that, in the description of the present invention, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. The terms "first," "second," etc., in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence.

[0019] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] Figure 1 A flowchart of a visual model upgrade method provided by an embodiment of the present invention is provided. The visual model is deployed on a target terminal. The visual model outputs the image detection result of the image sample. The method is applied to the server, such as Figure 1 As shown, the method specifically includes the following steps:

[0021] In step S101, the image samples and image detection results uploaded by the target terminal are obtained.

[0022] The image sample is a picture collected by the target terminal and containing the object to be identified; the image detection result is the result obtained after analyzing the image sample, including but not limited to object labels and coordinate frame information.

[0023] It can be understood that the embodiments of the present invention collect image samples with preliminary recognition information and their corresponding image detection results, such as object labels and location information, from target terminals deployed in various application scenarios as a basis for further processing and analysis in the following steps.

[0024] In step S102, training samples are generated based on the image samples and the image detection results, the database is updated based on the training samples, and the update time of the visual model is identified.

[0025] Among them, the visual model refers to an artificial intelligence model specifically used to process and analyze image data. It is usually built based on deep learning technologies such as CNN, and can extract features from images and perform various tasks, such as object recognition, classification, detection, segmentation, etc.; the update time of the recognition visual model refers to the time point when the visual model completes an iterative upgrade and is ready.

[0026] It can be understood that the embodiment of the present invention uses the target terminal to collect uploaded image samples and image detection results to generate training samples that meet the requirements, and these training samples are used to update the database. In this process, the exact time of the iterative upgrade of the visual model is recorded for subsequent tracking and management of the model version.

[0027] In an embodiment of the present invention, updating the database according to the training samples further includes: verifying the training samples and updating the database using the verification results.

[0028] It is understood that, in the embodiment of the present invention, before using training samples to update the database, it is necessary to first verify these samples and use the verification results to update the database to ensure their accuracy and reliability. Verification is the process of checking the accuracy of the generated training samples, which is specifically as follows:

[0029] In an embodiment of the present invention, verifying the training samples includes: obtaining image samples and image detection results in the training samples; extracting coordinate frame information in the image detection results, and using the coordinate frame information to crop the image samples to obtain new images; generating a model verification problem according to the type of visual model, and calling a pre-trained language model on the server according to the model verification problem and the new image; and verifying the training samples according to the feedback results of the pre-trained language model.

[0030] Among them, model verification questions are used to inquire about information such as the objects in the image and their locations, so as to verify the image samples; the pre-trained language model is a high-performance, large-scale multimodal machine learning model deployed on the server, which can understand the content of the image and answer questions about the image.

[0031] It can be understood that when verifying a training sample, the embodiment of the present invention first obtains the image sample and its detection results in the sample, extracts the coordinate frame information therefrom, uses the coordinate frame information to crop the original image sample, generates a new image, constructs corresponding model verification questions according to the specific type of the visual model, and uses these questions together with the newly generated image as input to call the pre-trained language model on the server, evaluates and corrects the errors in the training sample based on the feedback results returned by the pre-trained language model, ensures the accuracy and reliability of the training sample, thereby improving the quality of the final visual model, greatly improving data processing efficiency in an automated manner, and reducing the need for manual intervention.

[0032] In an embodiment of the present invention, a new image is obtained by cropping an image sample using coordinate frame information, including: determining a picture frame of the image sample according to the coordinate frame information; cropping a regional image within the picture frame, and placing the regional image on a picture template to obtain a new image, wherein the position of the regional image on the image sample and the picture template is the same.

[0033] The image template is a new background image with the same size as the original image, which is used to place the cropped area image at a specific position to generate a new composite image.

[0034] It can be understood that the embodiment of the present invention uses the coordinate frame information to crop the image sample to obtain a new image. First, the specific object position in the image sample is determined based on the coordinate frame information, that is, the picture frame of the image sample is determined. Then, the corresponding area image is cropped according to the coordinate frame, and the area image is placed at the corresponding position on a background image template of the same size as the original image, that is, the original coordinate frame position, so that the position of the cropped object in the new image is consistent with its position in the original image, creating a new image that focuses on the specific object in the original image, while keeping its spatial relationship in the original image unchanged.

[0035] In an embodiment of the present invention, the training samples are verified according to the feedback results of the target pre-trained language model, including: identifying whether the feedback result is empty; if the feedback result is empty, deleting the coordinate frame information of the corresponding image sample; if the feedback result is not empty, comparing the image detection result and the label information in the feedback result to see if they are consistent; if the label information is inconsistent, deleting the coordinate frame information of the corresponding image sample; if the label information is consistent, calculating the new coordinate frame information of the image sample based on the image detection result and the coordinate frame information in the feedback result.

[0036] It can be understood that the method of verifying the training sample according to the feedback result of the target pre-trained language model in the embodiment of the present invention is: first check whether the feedback result is empty. If the feedback result is empty, it indicates that the pre-trained language model failed to recognize any object. At this time, the coordinate frame information and label in the corresponding image sample should be deleted; if the feedback result is not empty, then further compare whether the original detection result of the image sample is consistent with the label information in the feedback result of the pre-trained language model. If the label information of the two is inconsistent, it means that there is a recognition error. At this time, the coordinate frame information in the image sample also needs to be deleted; conversely, if the label information is consistent, then the new coordinate frame position is calculated using the above-mentioned weighted formula in combination with the image detection result and the coordinate frame information provided by the pre-trained language model, thereby updating the coordinate frame information in the image sample, thereby ensuring the accuracy and reliability of the training sample and providing high-quality data support for subsequent model training.

[0037] In an embodiment of the present invention, new coordinate frame information of an image sample is calculated based on the coordinate frame information in the image detection result and the feedback result, including: obtaining the respective weights of the visual model and the pre-trained language model; and weightedly calculating the new coordinate frame information of the image sample based on the coordinate frame information in the image detection result, the coordinate frame information in the feedback result, and the respective weights of the visual model and the pre-trained language model.

[0038] Among them, the weight coefficient is a constant, which is set according to actual needs and is not specifically limited here; the new coordinate frame information of the image sample is calculated based on the coordinate frame information in the image detection result and the feedback result, and the formula is used accomplish, To calculate the new coordinates of the detection box, and is the weight coefficient, and both are constants. For example, , The coordinates returned by the pre-trained language model, The detection coordinates uploaded for the vision model.

[0039] It can be understood that in order to calculate the new coordinate frame information of the image sample, the embodiment of the present invention first needs to obtain the respective weights of the visual model and the pre-trained language model, and use the coordinate frame information in the image detection results, the coordinate frame information in the feedback results of the pre-trained language model, and the weights of the two models to obtain a more accurate new coordinate frame through a weighted calculation method. Specifically, this process fuses the coordinate frame detected by the visual model with the coordinate frame provided by the pre-trained language model according to a preset weight ratio to obtain an optimized new coordinate frame information, thereby combining the advantages of the two models, reducing the errors that may occur in a single model, and ultimately improving the accuracy of the new coordinate frame information.

[0040] In an embodiment of the present invention, updating the database using the verification results includes: using the image sample with the coordinate frame information deleted as a negative sample; using the image sample with the new coordinate frame information and label information as a positive sample; and adding the negative sample and the positive sample to the database.

[0041] It can be understood that when the embodiment of the present invention uses the verification results to update the database, it first marks the image samples whose coordinate frame information has been deleted due to recognition errors as negative samples. At the same time, the image samples that have been verified and carry updated coordinate frame information and label information are used as positive samples. These two types of samples are added to the database together, ensuring that the training samples in the database include both positive samples that can help the model learn the correct features and negative samples that help the model avoid false positives, thereby supporting more efficient and accurate model training and iterative upgrades, improving the performance of the final deployed visual model, and enabling it to maintain high accuracy and reliability in various complex environments.

[0042] In step S103, the number of training samples added to the database since the update time is obtained. When the number of training samples is greater than the threshold, the visual model is trained using the training samples in the database, and the visual model of the target terminal is upgraded using the trained visual model.

[0043] Among them, the quantity threshold is a preset value. When the number of new training samples exceeds this threshold, the training process of the visual model is triggered. It is set according to the actual situation and is not specifically limited here. The trained visual model upgrades the visual model of the target terminal by deploying this trained and optimized new model to the target terminal, replacing the original old version of the visual model.

[0044] It can be understood that the embodiment of the present invention obtains the number of new training samples since the last update of the database. Once it is found that the number of new training samples exceeds the preset threshold, the process of using these newly added training samples to train the visual model is started. After the training is completed, a visual model with better performance is obtained. This trained and optimized new model is deployed to each target terminal to replace the original old version of the visual model, realizing automatic iterative upgrade of the model, ensuring that the visual model on the edge device can continue to learn the latest data, thereby continuously improving the recognition accuracy and efficiency, adapting to the ever-changing needs of actual application scenarios, and greatly improving the efficiency of model maintenance and updating through automation.

[0045] In an embodiment of the present invention, the training samples include positive samples and negative samples, and the visual model is trained using the training samples in the database, including: selecting positive samples and negative samples from the database; generating a training data set based on the positive samples and negative samples; and training the visual model using the training data set.

[0046] It can be understood that the embodiment of the present invention uses training samples in the database to train the visual model. First, positive samples and negative samples are selected from the database. The positive samples provide correctly labeled target objects and location information to help the model learn to identify the correct objects; while the negative samples show non-target objects or scenes that are prone to misjudgment to enhance the model's ability to distinguish between background and target. A training data set is generated based on the selected positive and negative samples. The data set covers various situations that the model needs to learn. This comprehensive training data set is used to train the visual model, so that the model can continuously adjust its own parameters, thereby more accurately performing target recognition tasks in practical applications.

[0047] In an embodiment of the present invention, before using the training data set to train the visual model, the method further includes: increasing the number of negative samples in the training data set based on the training samples in the database.

[0048] It is understandable that before using the training dataset to train the visual model, the embodiment of the present invention needs to first increase the number of negative samples in the training dataset based on the training samples in the database, making the training dataset more comprehensive, helping the trained visual model to more effectively distinguish background and non-target objects, reducing the false alarm rate, and thus improving the overall performance and accuracy of the model. This process is specifically as follows:

[0049] In an embodiment of the present invention, enhancing the number of negative samples in a training data set based on training samples in a database includes: obtaining the number of negative samples in the training data set; if the number of negative samples is less than a target number, calculating the difference between the number of negative samples and the target number; randomly selecting training samples from the database based on the difference, and enhancing the number of negative samples in the training data set based on the training samples.

[0050] Among them, the target number is set according to actual needs and is not specifically limited here; the number of negative samples in the training data set based on the training samples will be described in detail below and will not be repeated here.

[0051] It is understandable that when preparing training data, the embodiment of the present invention first needs to check whether the number of negative samples has reached a preset target number. If it is found that the existing number of negative samples is insufficient, the required difference number is calculated. Based on this difference number, a certain number of training samples are randomly selected from the database, and these samples are used to enhance the number of negative samples in the training data set, thereby ensuring that the training data has sufficient diversity, which helps to improve the learning effect and recognition accuracy of the visual model, enabling it to more accurately distinguish non-target objects and reduce the false alarm rate.

[0052] In an embodiment of the present invention, randomly selecting training samples from a database based on the difference amount includes: if the difference amount is less than a preset amount, randomly selecting a preset amount of training samples from the database; if the difference amount is greater than or equal to the preset amount, randomly selecting the difference amount of training samples from the database.

[0053] The preset number is set according to the actual situation and is not specifically limited here.

[0054] It can be understood that when the embodiment of the present invention selects training samples from the database according to the number of differences to enhance negative samples, it first determines whether the number of differences is less than a preset number. If the number of differences is small, that is, the number of negative samples that need to be added is lower than the preset value, the number of differences is directly set to the preset number, and the preset number of training samples is randomly selected from the database; conversely, if the number of differences is greater than or equal to the preset number, it means that more negative samples need to be added, then training samples equal to the number of differences are randomly selected from the database for enhancement, ensuring that the amount of negative samples can be effectively increased to improve the quality of model training.

[0055] In an embodiment of the present invention, the number of negative samples in a training data set is enhanced based on training samples, including: extracting sample images from the training samples; inputting the sample images into a pre-trained language model of a server, and using the image generation function of the pre-trained language model to generate images without label information; and generating negative samples based on image description information and images without label information.

[0056] It can be understood that the embodiment of the present invention enhances the number of negative samples in the training data set. First, sample images are extracted from the selected training samples, and these sample images are input into the pre-trained language model on the server. Its image generation function is used to generate new images without any label information. Combined with specific image description information, the generated new images are used to create negative samples, thereby enriching the content of the training data set, improving the ability of the visual model to distinguish non-target objects, reducing the false alarm rate, and ultimately improving the accuracy and reliability of the model.

[0057] In an embodiment of the present invention, before generating negative samples based on the image description information and the image without label information, the method further includes: obtaining misjudgment data of the visual model; and generating image description information based on the misjudgment data.

[0058] Among them, the misjudgment data of the visual model refers to the current visual model mistakenly identifying positive samples as negative samples, or mistakenly marking negative samples as data containing target objects, which is obtained by comparing the model prediction results with the actual situation.

[0059] It is understandable that before generating negative samples using images that do not carry label information and corresponding description information, the embodiments of the present invention need to obtain the misjudgment data generated by the current visual model. This includes situations where the model mistakenly labels an image that does not actually contain a target object as a target object, or fails to correctly identify the presence of a target object. Detailed image description information is generated based on the collected misjudgment data. The description information will specifically point out the key background conditions or environmental factors that are likely to cause model misjudgment. The purpose of this is to generate new negative sample images in a targeted manner, thereby helping to train a more accurate visual model, reducing the possibility of similar misjudgments in the future, and helping to improve the overall performance and accuracy of the model.

[0060] In an embodiment of the present invention, after enhancing the number of negative samples in a training data set based on training samples, the method further includes: obtaining the total number of training samples in the training data set; setting an enhancement stop threshold according to the total number of training samples, wherein the enhancement stop threshold is less than the total number of training samples; if the number of negative samples is less than the enhancement stop threshold, then stopping enhancing the number of negative samples in the training data set; if the number of negative samples is greater than or equal to the enhancement stop threshold, then deleting some negative samples in the training data set.

[0061] Among them, the total number of training samples refers to the number of all training samples contained in the entire training dataset, including positive samples and negative samples; the enhancement stop threshold is a preset value used to determine when to stop adding new negative samples to the training dataset. It is set according to actual conditions and is not specifically limited here; the number of negative samples refers to the number of negative samples already in the current training dataset.

[0062] It can be understood that, after the embodiment of the present invention enhances the number of negative samples in the training data set based on the training samples, it is necessary to obtain the total number of all training samples in the training data set, and set an enhancement stop threshold based on this total number. The threshold is less than the total number of training samples and is used to control the amount of increase in negative samples. The number of negative samples in the current training data set is checked. If it is found that the number of negative samples is less than the set enhancement stop threshold, the enhancement of the number of negative samples in the training data set is stopped, and the negative sample enhancement is ended; conversely, if the number of negative samples has reached or exceeded the enhancement stop threshold, the addition of new negative samples is stopped, and some negative samples need to be randomly deleted from the existing training data set to ensure that the number of negative samples does not exceed the set threshold, so as to maintain the balance between positive and negative samples in the training data set, avoid affecting the effect of model training due to too many samples of a certain type, and thus ensure that the visual model finally trained has good generalization ability and recognition accuracy.

[0063] According to the visual model upgrading method proposed in an embodiment of the present invention, training samples can be generated based on image samples and image detection results, and the database can be updated using the training samples to identify the update time of the visual model. When the number of training samples is greater than a quantity threshold, the visual model can be trained using the training samples in the database, and the visual model of the target terminal can be upgraded using the trained visual model so that the number of training samples reaches the desired quantity threshold. The visual model can also be used to perform image analysis to replace the manual verification process, and model training can be automatically performed to achieve technical effects such as improving model recognition accuracy, saving labor costs, and improving training efficiency.

[0064] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0065] The visual model upgrading method is further described below through a specific embodiment.

[0066] This embodiment proposes a method for automatic training and upgrading of AI models in an edge distributed system. This method requires the edge end and the central end (cloud) to work together to achieve cloud-edge collaboration.

[0067] This method is a process of model training, upgrading and optimizing. First, when the function is running, the edge and center software have been deployed, debugged and graduated, and the business functions are running normally. Only the model accuracy needs to be continuously upgraded and iterated.

[0068] The following uses the iterative training optimization of the fireworks detection model as an example to illustrate.

[0069] First, when the fireworks model deployed on each terminal is working, the model will detect the video images transmitted by the edge scene camera frame by frame. When the model detects that fireworks have occurred, the platform software saves the alarm picture and issues an alarm message, and at the same time sends the alarm information (pictures and detection information, etc.) to the central platform.

[0070] When the center receives an edge alarm, it adds the alarm image to the corresponding sample image library based on the alarm type, such as the fireworks detection sample image library. It also generates a corresponding txt file (a plain text file) and adds the corresponding detection label and coordinate frame information. The center performs the same processing on each alarm image received from an edge terminal.

[0071] At the same time, the training module at the center uses the image understanding function of the multimodal pre-trained language model to verify the images and detection information added to the sample image library, replacing the manual verification process. First, obtain a fireworks detection image and related detection information, such as labels and detection frame coordinates, from the sample image library. Then, based on the coordinate frame information, the image area within the image frame is cropped and the cropped part is placed in the original coordinate frame position of the white background image of the same size as the original image to form a new image. Then generate the pre-trained language model question:

[0072] Question 1 = "Are there any fireworks in the picture? The format of output should be like {"yes", "no"}", is used to detect whether a specific object, such as fireworks, exists in an image. It requires the pre-trained language model to simply answer whether the specified object is detected in the image. The output format is a simple binary form: "yes" or "no", which helps to quickly determine whether the image contains the target object.

[0073] Question2 = "What object is in the picture? The format of output should be like { "label": the name of this object}", which aims to identify and return the name of the main object in the picture. Through Question2, you can obtain the object label in the picture (for example, "fireworks", "helmet", etc.) and confirm the main content of the picture.

[0074] Question3 = "Detect all objects in the image and return their locations in the form of coordinates. The format of output should be like{"bbox": [x, y, w, h], "label": the name of this object}", which is used to comprehensively detect all objects in the image and requires the pre-trained language model to provide the location information (bounding box coordinates) of each detected object and its corresponding label. The detailed output can help accurately understand the location and category of each object in the image, and can provide more detailed information than the first two questions, supporting further data analysis and processing.

[0075] In Question 1, fireworks is replaced according to different models. For example, the helmet detection is changed to helmet.

[0076] Finally, the newly generated image and the generated pre-trained language model question are used as parameters to call the locally deployed pre-trained language model interface, and the relevant detection objects and coordinate information returned by the pre-trained language model are analyzed.

[0077] Based on the returned result information, if the returned result is not empty and the detection result returned by the pre-trained language model is the same as the detection result uploaded by the visual model, the new coordinates are calculated according to the following formula (1):

[0078] , (1)

[0079] in, To calculate the new coordinates of the detection box, and is the weight coefficient; The coordinates returned for the pre-trained language model; The detection coordinates uploaded for the vision model.

[0080] After the calculation is complete, the new coordinate information is used to update the corresponding image's txt file information. If the returned result is not empty and the target type is inconsistent, the corresponding image's txt coordinate frame information and detection category are deleted. If the pre-trained language model returns an empty result, the corresponding image's txt coordinate frame information and detection category are also deleted. Finally, after completing the verification of all detection frames in an image, if the txt file does not contain any coordinate frame information, the image is added to the negative sample library.

[0081] When the number of samples in the training sample library increases to meet the threshold requirement, the next step, sample preparation, begins. The number of negative samples in the negative sample library is checked. If the number of negative samples is less than the set value, such as 1 / 5 of the total number of samples, negative sample enhancement is performed. Images that exceed the set value are randomly extracted from the sample library and reconstructed using the image generation function of the pre-trained language model. Existing target objects are removed from the images, retaining only the background image information as negative sample material. Furthermore, text descriptions are generated for scenes that the model is most likely to misidentify (e.g., fireworks recognition), such as those under daylight or headlights at night. Negative sample images are generated using the image generation function of the pre-trained language model.

[0082] Once the samples are ready, the training script is called to train the model. After model training is complete, a preliminary verification is performed. After verification, the model is converted and then distributed in batches with one click to complete this round of iterative upgrades.

[0083] The system flow chart of this method is as follows Figure 2 shown.

[0084] Specifically, the main steps of sample image uploading, collection, verification, training, and conversion are as follows:

[0085] Step 1: Deploy each edge algorithm module and run the algorithm model for target recognition and detection (such as fireworks recognition).

[0086] Step 2: The edge model identifies the target warning image sample (such as fireworks) and sends the detection information to the center.

[0087] Step 3: After receiving the alarm image information, the center performs sample image information extraction and preprocessing to obtain the image, label and coordinate frame information, and generates a txt file.

[0088] Step 4: The image and the corresponding detection information are added to the sample image library.

[0089] Step 5: Add the image and corresponding detection information to the pre-trained language model verification list.

[0090] Step 6: Obtain image samples and detection information one by one from the pre-trained language model verification list for verification.

[0091] Step 7: Based on the verification results, update the training sample image library and the corresponding txt file information.

[0092] Step 8: Check the number of newly added training samples. If the number of newly added training samples is greater than the set threshold, go to step 9 to prepare the training samples. Otherwise, continue with this step.

[0093] Step 9: Prepare the training samples, that is, perform negative sample enhancement to meet the training requirements.

[0094] Step 10: Call the model training script to perform model training.

[0095] Step 11: After verification is complete, convert the model into a model format suitable for the edge accelerator card.

[0096] Step 12: Send the model with one click and send the latest model to each edge end in batches.

[0097] Step 13: The edge receives the model and verifies it.

[0098] Step 14: If the verification is qualified, the model is replaced, and a round of iterative upgrade of this model is completed.

[0099] like Figure 3 As shown in the figure, the main steps of multimodal pre-trained language model verification are as follows:

[0100] S201. Obtain the current image and current detection information (label and coordinate frame information) from the pre-trained language model verification list.

[0101] S202. Crop the image of the area within the picture frame according to the coordinate frame information and place the cropped part at the original coordinate frame position of the white background picture of the same size as the original picture to generate a new picture.

[0102] S203. Generate model verification questions: Question 1, Question 2, and Question 3.

[0103] S204. Call the multimodal pre-trained language model interface with the newly generated image and question.

[0104] S205. Analyze the feedback results of the pre-trained language model.

[0105] S206. If the feedback target result is empty, the coordinate information of the corresponding txt file is deleted. If the feedback target result is not empty, go to S207.

[0106] Step 207 extracts the target type and coordinate information and compares it with the uploaded result of the visual model. If the target type is consistent, go to step 210. If the type is inconsistent, go to step 208 and delete the coordinate information of the corresponding txt file.

[0107] S208. Delete the coordinate information of the corresponding txt file.

[0108] S209. Determine whether all the check boxes in the image have been verified. If so, proceed to step S213. If not, proceed to S202.

[0109] S210. Calculate the final coordinate frame. At this time, there are two sets of coordinate frame information. Take the visual model coordinate frame weight as 0.25 and the pre-trained language model coordinate frame information weight as 0.75, and calculate the final detection target coordinate frame by weight.

[0110] S212. Determine whether all the detection boxes in the image have been verified. If so, proceed to step S213. If not, proceed to S202.

[0111] S213. If the txt file corresponding to this image does not have any coordinate information, then the image is added to the negative sample library.

[0112] like Figure 4 As shown in Figure 2, the main steps of negative sample enhancement for modality pre-trained language models are as follows:

[0113] S301. Obtain the difference between the number of negative samples and the set value.

[0114] S302. Determine whether the difference is less than 1 / 3 of the set value. If so, go to S304; otherwise, go to S303.

[0115] S303. Randomly copy the sample images of the above-mentioned difference quantity from the sample library.

[0116] S304. The difference amount is set to 1 / 3 of the set total.

[0117] S305. Using the copied image as the background, call the image generation function of the pre-trained language model to generate an image without the recognition target.

[0118] S306. Import key environmental background information that is most likely to be misjudged by the current model.

[0119] S307. Use the imported current model to misjudge the key background information, generate image description information, and use the image description information to generate negative sample images with a difference of 1 / 2 the number using the pre-trained language model.

[0120] S308. Determine whether the number of negative samples is less than 1 / 5 of the total training samples. If so, end the negative sample enhancement; otherwise, go to S309.

[0121] S309. Randomly delete some negative sample images so that their total number is less than or equal to 1 / 5 of the total training samples.

[0122] The above solution is applied to the cluster version of the Edge Intelligence Management Platform to realize automatic training and upgrading of AI vision models, including automatic sample push, sample classification, sample verification, negative sample enhancement, model training, model conversion, batch model distribution and upgrading, etc., to meet the integrated training and pushing needs of the edge management platform.

[0123] The central / cluster version of the Yuanzhi management platform is generally deployed on the server in the central computer room, while the stand-alone version is deployed on device terminals scattered throughout the site. When the algorithm model deployed on the edge starts running, the target samples detected in real time will be pushed to the cluster version, i.e. the central end. A series of operations such as image sample classification, sample verification, negative sample enhancement, model training, and model conversion will be completed at the central end. Finally, it will be re-distributed to each edge end to realize the iterative upgrade of the model. This process does not require human intervention, thus solving the problem of manual collection, labeling and verification of sample images, greatly improving development efficiency and saving manpower and material resources.

[0124] This process uses automatic target detection samples from the algorithms configured on each edge to collect samples, thus eliminating the need for manual sample collection. The more devices deployed on the edge, the more samples collected, achieving sampling diversity and significantly improving the model's generalization capabilities.

[0125] In traditional model training, sample labeling and verification are the most time-consuming and labor-intensive. This embodiment achieves preliminary sample labeling through the first step of sample collection. At the same time, for the incorrectly labeled parts, it rationally constructs question and answer styles and combines the image understanding capabilities of the multimodal pre-trained language model to verify the sample images. It then makes modifications to the initial verification errors to replace the manual verification process.

[0126] To address the lack of negative samples, we use some sample images as background colors and call the image generation function of the pre-trained language model to generate images without recognition targets. At the same time, we use the imported key background information that is most likely to be misjudged in the current model to generate image descriptions, and use the pre-trained language model to generate negative sample images that meet the background requirements, thereby enriching the number of negative sample training and greatly improving the accuracy of the final model.

[0127] Through automatic sample collection and labeling, automatic verification, automatic negative sample enhancement, and then calling the training script to realize model training and conversion, and finally one-click distribution, the purpose of automatic iteration and upgrading is achieved.

[0128] The embodiment of the present invention also provides a visual model upgrading device, such as Figure 5 As shown, the visual model upgrading device 10 includes: an acquisition module 401 , a generation module 402 and an upgrading module 403 .

[0129] Among them, the acquisition module 401 is used to obtain image samples and image detection results uploaded by the target terminal; the generation module 402 is used to generate training samples based on the image samples and image detection results, update the database based on the training samples, and identify the update time of the visual model; the upgrade module 403 obtains the number of new training samples added to the database from the update time. When the number of training samples is greater than the number threshold, the visual model is trained using the training samples in the database, and the trained visual model is used to upgrade the visual model of the target terminal.

[0130] In an embodiment of the present invention, the generation module 402 is further used to: obtain image samples and image detection results in the training samples; extract coordinate frame information in the image detection results, and use the coordinate frame information to crop the image samples to obtain new images; generate a model verification problem according to the type of visual model, and call the pre-trained language model on the server according to the model verification problem and the new image; verify the training samples according to the feedback results of the pre-trained language model; and update the database using the verification results.

[0131] In an embodiment of the present invention, the generation module 402 is further used to: determine the picture frame of the picture sample according to the coordinate frame information; crop the regional image in the picture frame, and place the regional image on the picture template to obtain a new picture, wherein the position of the regional image on the picture sample and the picture template is the same.

[0132] In an embodiment of the present invention, the generation module 402 is further used to: identify whether the feedback result is empty; if the feedback result is empty, delete the coordinate frame information of the corresponding image sample; if the feedback result is not empty, compare whether the label information in the image detection result and the feedback result are consistent; if the label information is inconsistent, delete the coordinate frame information of the corresponding image sample; if the label information is consistent, calculate the new coordinate frame information of the image sample based on the image detection result and the coordinate frame information in the feedback result.

[0133] In an embodiment of the present invention, the generation module 402 is further used to: obtain the respective weights of the visual model and the pre-trained language model; and weightedly calculate the new coordinate frame information of the image sample based on the coordinate frame information in the image detection result, the coordinate frame information in the feedback result, and the respective weights of the visual model and the pre-trained language model.

[0134] In the embodiment of the present invention, the generation module 402 is further configured to: use the image sample with the coordinate frame information deleted as a negative sample; use the image sample with the new coordinate frame information and label information as a positive sample; and add the negative sample and the positive sample to the database.

[0135] In an embodiment of the present invention, the upgrading module 403 is further used to: select positive samples and negative samples from the database; generate a training data set based on the positive samples and negative samples; and train the visual model using the training data set.

[0136] In an embodiment of the present invention, an enhancement module is further included, and the enhancement is further used to: before using the training data set to train the visual model, it also includes: enhancing the number of negative samples in the training data set based on the training samples in the database.

[0137] In an embodiment of the present invention, the enhancement module is further used to: obtain the number of negative samples in the training data set; if the number of negative samples is less than the target number, calculate the difference between the number of negative samples and the target number; randomly select training samples from the database based on the difference, and enhance the number of negative samples in the training data set based on the training samples.

[0138] In an embodiment of the present invention, the enhancement module is further configured to: if the difference amount is less than a preset amount, randomly select training samples of the difference amount from the database; if the difference amount is greater than or equal to the preset amount, randomly select training samples of the preset amount from the database.

[0139] In an embodiment of the present invention, the enhancement module is further used to: extract sample images from the training samples; input the sample images into the pre-trained language model of the server, and use the image generation function of the pre-trained language model to generate images without label information; generate negative samples based on the image description information and the images without label information.

[0140] In an embodiment of the present invention, the enhancement module is further used to: obtain misjudgment data of the visual model; and generate image description information based on the misjudgment data.

[0141] In an embodiment of the present invention, the enhancement module is further used to: obtain the total number of training samples in the training data set after enhancing the number of negative samples in the training data set based on the training samples; set an enhancement stop threshold according to the total number of training samples, wherein the enhancement stop threshold is less than the total number of training samples; if the number of negative samples is less than the enhancement stop threshold, stop enhancing the number of negative samples in the training data set; if the number of negative samples is greater than or equal to the enhancement stop threshold, delete some negative samples in the training data set.

[0142] For the description of the features in the embodiment corresponding to the visual model upgrading device, please refer to the relevant description of the embodiment corresponding to the visual model upgrading method, which will not be repeated here.

[0143] An embodiment of the present invention further provides an electronic device, such as Figure 6 As shown, it includes a memory 501 and a processor 502, the memory 501 stores a computer program, and the processor 502 is configured to run the computer program to execute the steps in the above-mentioned visual model upgrading method embodiment.

[0144] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned visual model upgrading method embodiments when running.

[0145] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0146] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned visual model upgrading method embodiments are implemented.

[0147] An embodiment of the present invention also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned visual model upgrading method embodiments.

[0148] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0149] The above is a detailed introduction to a visual model upgrading method, electronic device, storage medium and program product provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A visual model upgrading method, characterized in that: The visual model is deployed on a target terminal, and the visual model outputs a picture detection result of a picture sample. The method is applied to a server, wherein the method includes: Obtaining the image samples and image detection results uploaded by the target terminal; Generate a training sample based on the image sample and the image detection result, and update the database based on the training sample, including: obtaining the image sample and the image detection result in the training sample; extracting the coordinate frame information in the image detection result, and using the coordinate frame information to crop the image sample to obtain a new image; generate a model verification question based on the type of the visual model, and call the pre-trained language model on the server based on the model verification question and the new image; verify the training sample based on the feedback result of the pre-trained language model; update the database using the verification result, and identify the update time of the visual model, wherein the pre-trained language model is a multimodal machine learning model deployed on the server, which can understand the content of the image and answer questions about the image; The number of training samples newly added to the database since the update time is obtained. When the number of training samples is greater than a quantity threshold, the visual model is trained using the training samples in the database, and the visual model of the target terminal is upgraded using the trained visual model.

2. The visual model upgrading method according to claim 1, characterized in that: The step of cropping the image sample using the coordinate frame information to obtain a new image includes: Determining a picture frame of the picture sample according to the coordinate frame information; The regional image in the picture frame is cropped, and the regional image is placed on a picture template to obtain a new picture, wherein the regional image is located at the same position on the picture sample and the picture template.

3. The visual model upgrading method according to claim 1, characterized in that: The verifying the training sample according to the feedback result of the target pre-trained language model includes: Identify whether the feedback result is empty; If the feedback result is empty, the coordinate frame information of the corresponding image sample is deleted; If the feedback result is not empty, compare the image detection result and the label information in the feedback result to see if they are consistent. If the label information is inconsistent, delete the coordinate frame information of the corresponding image sample. If the label information is consistent, calculate the new coordinate frame information of the image sample based on the image detection result and the coordinate frame information in the feedback result.

4. The visual model upgrading method according to claim 3, characterized in that: The calculating new coordinate frame information of the image sample according to the image detection result and the coordinate frame information in the feedback result includes: Obtaining respective weights of the visual model and the pre-trained language model; New coordinate frame information of the image sample is weightedly calculated according to the coordinate frame information in the image detection result, the coordinate frame information in the feedback result, and the respective weights of the visual model and the pre-trained language model.

5. The visual model upgrading method according to claim 1, characterized in that: The updating of the database using the verification result includes: The image samples with the coordinate frame information removed are used as negative samples; The image sample carrying the new coordinate frame information and label information is used as a positive sample; The negative samples and the positive samples are added to the database.

6. The visual model upgrading method according to claim 1, characterized in that: The training samples include positive samples and negative samples, and the training of the visual model using the training samples in the database includes: Selecting positive samples and negative samples from the database; Generate a training data set based on the positive samples and negative samples; The visual model is trained using the training data set.

7. The visual model upgrading method according to claim 6, characterized in that: Before using the training data set to train the visual model, the method further includes: The number of negative samples in the training data set is increased based on the training samples in the database.

8. The visual model upgrading method according to claim 7, characterized in that: The step of increasing the number of negative samples in the training data set based on the training samples in the database includes: Obtaining the number of negative samples in the training data set; If the number of negative samples is less than the target number, the difference between the number of negative samples and the target number is calculated; Training samples are randomly selected from the database based on the number of differences, and the number of negative samples in the training data set is increased based on the training samples.

9. The visual model upgrading method according to claim 8, characterized in that: The randomly selecting training samples from the database based on the difference amount includes: If the difference amount is less than a preset amount, randomly selecting the preset amount of training samples from the database; If the difference amount is greater than or equal to the preset amount, a training sample of the difference amount is randomly selected from the database.

10. The visual model upgrading method according to claim 8, characterized in that: The step of increasing the number of negative samples in the training data set based on the training samples includes: Extracting sample images from the training samples; Inputting the sample image into the pre-trained language model of the server, and using the image generation function of the pre-trained language model to generate an image without label information; Generate negative samples based on the image description information and the image without label information.

11. The visual model upgrading method according to claim 10, characterized in that: Before generating negative samples based on the image description information and the image without label information, the method further includes: Obtaining misjudgment data of the visual model; The image description information is generated according to the misjudgment data.

12. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the visual model upgrading method according to any one of claims 1 to 11 when executing the computer program.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the visual model upgrading method according to any one of claims 1 to 11 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the visual model upgrading method according to any one of claims 1 to 11 are implemented.

Images