Small model automatic training method based on multi-mode large model cognition intelligent driving

Abstract

The invention relates to the technical field of model automatic training, and discloses a small model automatic training method based on multi-modal large model cognition intelligent driving, and the method comprises the steps: obtaining each training sample generation task received in a target time period and a task execution state of each model driving end; judging whether the model driving group meets a deadline requirement for executing an exclusive sample labeling strategy or not, and selecting the exclusive sample labeling strategy or a shared sample labeling strategy to execute distribution and automatic labeling actions for all multi-modal training data in each training sample generation task, and sending the training sample set obtained by automatic labeling to a model application end to carry out small model training aiming at the corresponding detection scene. Therefore, by performing selective annotation image proportion sharing on the training sample generation task and establishing a dual-behavior judgment system of the model application end and the model driving end, the task processing amount is reduced, and meanwhile, the influence on the annotation data quality and the training model recognition accuracy is reduced as much as possible.

Description

Technical Field

[0001] The present invention relates to the technical field of model automated training, and in particular to a small model automated training method based on multimodal large model cognitive intelligence drive. Background Art

[0002] At present, there are a large number of demand for the application of behavior recognition models in various model application ends, such as security monitoring, intelligent transportation, industrial manufacturing, etc. These fields can improve supervision and control in detection scenarios by deploying artificial intelligence-based recognition models on the model application end.

[0003] Due to the large number of model deployment requirements in different application scenarios, how to improve the training and deployment efficiency when building small models on the application side has become a major difficulty. In traditional model training solutions, it is usually necessary to manually label the target categories and behavior categories of massive scene images, and then use the labeled images to train the model in the corresponding detection scenario. This process is time-consuming and labor-intensive. Manual labeling may take several months, and the subjectivity of manual labeling affects the labeling quality.

[0004] Based on the defects in the above-mentioned traditional model training solutions, some platforms have proposed using large models to automatically annotate data sets and use the annotated training sets to implement small model training solutions in corresponding scenarios. Users only need to provide basic multimodal data and model application requirements to obtain annotated training data samples, which can quickly implement small model training and deployment in local corresponding scenarios. However, with the continuous growth of model training needs, the amount of data annotation tasks performed by the platform is gradually approaching the load limit of the hardware equipment. Especially during the peak period of model training needs, the platform often needs to face short-term and overloaded tasks. At this time, the platform cannot reasonably allocate and process the annotation tasks, which will lead to inaccurate annotation data feedback to users, insufficient training data feedback, or overloaded operation of hardware equipment for deploying large models.

[0005] Therefore, how to achieve automated training of small models driven by cognitive intelligence of multimodal large models, to ensure the accuracy of user labeled data and the amount of training data feedback to a certain extent, while avoiding overload of hardware equipment deployed on large models, so as to balance the training quality and continuity during the peak period of model training demand, is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The present invention provides a small model automatic training method based on multimodal large model cognitive intelligence driving, aiming to solve at least one of the above-mentioned technical problems.

[0007] To achieve the above object, the present invention provides a small model automatic training method based on multimodal large model cognitive intelligent driving, which is used in a model automatic training system. The model automatic training system includes a model driving group having several model driving terminals, a model application group having several model application terminals, and a labeling task allocation server. The method includes the following steps: S1: The labeling task allocation server receives the training sample generation task sent by each model application end in the target period and the task execution status sent by each model driver end; wherein the training sample generation task includes the labeling task deadline association information and several detection scene features; S2: The labeling task allocation server determines whether the model-driven group meets the deadline requirements for executing the exclusive sample labeling strategy based on the task execution status and the labeling task deadline association information; S3: If yes, the labeling task allocation server distributes the training sample generation task to the corresponding model driver according to the exclusive sample labeling strategy, and uses the large model deployed by the model driver to perform the automatic labeling action of the multimodal training data; S4: If not, the labeling task allocation server divides the training sample generation task into several general level categories based on several detection scene features, generates a shared sample labeling strategy for the training sample generation task for the associated general level based on the general level of each training sample generation task, and distributes the training sample generation task to the corresponding model driver, and uses the large model deployed by the model driver to perform the automatic labeling action of the multimodal training data; S5: Each model driver sends the training sample set obtained after performing the automatic labeling action of the multimodal training data to the model application end corresponding to the training sample generation task, driving the model application end to use the training sample set to train and deploy the terminal category recognition model and the terminal behavior recognition model; S6: Each model application terminal uses the deployed terminal category recognition model and terminal behavior recognition model to perform scene target category recognition and scene target behavior recognition of the target scene image in the corresponding detection scene, and determines whether the behavior of the scene target is a prohibited behavior; S7: If yes, the target scene image is sent to the verification model driver of the model driver group, and the prohibited behavior judgment verification is performed using the large model deployed by the verification model driver.

[0008] Optionally, step S1 specifically includes: S11: Each model application end collects multimodal training data and scene detection description information of the target detection scene, extracts the labeling task deadline association information and several scene detection features in the scene detection description information, and constructs them as task labels together with the multimodal training data to generate training sample tasks, and sends them to the labeling task allocation server; S12: Each model driver calls the training sample generation task list stored in itself, extracts the estimated execution period of each training sample generation task to be executed in the training sample generation task list, and sends it to the labeling task allocation server as the task execution status of each model driver; S13: The labeling task allocation server receives the training sample generation task sent by each model application end in the target period and the task execution status sent by each model driver end.

[0009] Optionally, in step S11, the labeling task deadline association information and several scene detection features in the scene detection description information are extracted as task labels and constructed together with the multimodal training data as a training sample generation task, specifically including: Parsing the scene detection description information, extracting the sample generation period and several scene detection features in the scene detection description information, and generating the training data requirement for each training sample generation task based on the several scene detection features; The training data demand and sample generation deadline are summarized as labeling task deadline association information, and the labeling task deadline association information and several scene detection features are used as task labels together with multimodal training data to form a training sample generation task, which is sent to the labeling task allocation server.

[0010] Optionally, based on a number of scene detection features, the training data requirement for each training sample generation task is generated, specifically including: Extracting the number of recognized object types of recognized object features and the recognized scene categories determined by recognized scene features from a plurality of scene detection features; Call the predefined comparison table of the relationship between the number of different types of objects to be recognized and the corresponding number of training images required under the recognition scene category to match the training data requirement for each training sample generation task.

[0011] Optionally, step S2 specifically includes: S21: The labeling task allocation server extracts the estimated execution period of each to-be-executed training sample generation task of each model driver from the task execution status, and extracts the training data demand and sample generation deadline of each training sample generation task from the labeling task deadline association information; S22: performing ergodic allocation on all multimodal training data in each training sample generation task received within the target period, and obtaining a plurality of exclusive sample labeling schemes for allocating the multimodal training data in the training sample generation task to the corresponding model driver end; S23: according to the estimated execution period of each training sample generation task to be executed, the training data requirement and sample generation period of each training sample generation task, according to the standard annotation speed of each model driver and the total training data requirement of each training sample generation task, calculate the completion time after each model driver executes the annotation action of the allocated multimodal training data; S24: Determine whether there is an exclusive sample labeling scheme among several exclusive sample labeling schemes, in which the completion time of all model drivers executing the training sample generation task within the target period meets the sample generation deadline.

[0012] Optionally, step S3 specifically includes: S31: If yes, the labeling task allocation server extracts the target exclusive sample labeling scheme whose last completion time is the farthest from the sample generation deadline in the exclusive sample labeling strategy; S32: Based on the strategy of allocating all multimodal training data for each training sample generation task to the model driving end in the target exclusive sample labeling scheme, the corresponding multimodal training data in each training sample generation task is distributed to the corresponding model driving end, and the large model deployed by the model driving end is used to perform automatic labeling of the multimodal training data.

[0013] Optionally, step S4 specifically includes: S41: If not, the labeling task allocation server calculates the detection scene similarity between any two training sample generation tasks based on a number of detection scene features of each training sample generation task; S42: sorting the detection scene similarities between each training sample generation task and other training sample generation tasks from high to low, and judging whether the detection scene similarity with the highest ranking after sorting exceeds a preset similarity threshold; if so, defining the training sample generation task as a general-purpose scene task; if not, defining the training sample generation task as a non-general-purpose scene task; S43: according to the highest ranking detection scene similarity in the general-purpose scene task, all general-purpose scene tasks are divided into several general-purpose level categories, based on the training data demand and sample generation period of each training sample generation task, considering the estimated execution period of each to-be-executed training sample generation task of each model driver and the general-purpose level of each general-purpose scene task, ergodic allocation is performed on the multimodal training data in the training sample generation task received within the target period, and according to the principle of minimum shared general-purpose quantization value, a shared sample labeling strategy for the training sample generation task for the associated general-purpose level is generated; The minimum shared universal quantization value principle is specifically: First, all multimodal training data of non-general-purpose scenario tasks in the training sample generation task are ergodicly distributed to obtain several exclusive sample labeling schemes in which the multimodal training data in the non-general-purpose scenario tasks are cut in different proportions and distributed to the model driver end; Then, all multimodal training data of each general-purpose scenario task in the training sample generation task are ergodicly allocated in each exclusive sample annotation scheme according to the corresponding shared sample annotation ratio, so that the completion time of all model driving ends executing the training sample generation task within the target period meets the sample generation deadline, and the shared general-purpose quantization value of all general-purpose scenario tasks is minimized, and the shared general-purpose quantization value is configured as: the product and accumulation of the shared sample annotation ratio of each general-purpose scenario task and the training data sharing amount determined by the training data demand amount and the general level of the general-purpose scenario task; S44: According to the shared sample labeling strategy, the training sample generation task is distributed to the corresponding model driving end, and the large model deployed by the model driving end is used to perform automatic labeling of the multimodal training data. From the training image data set of the corresponding general-purpose scenario task with the highest detection scene similarity for each general-purpose scenario task, the training image data of a corresponding number of shared sample labeling ratios of the general-purpose scenario task are extracted and put into the training image data set of the general-purpose scenario task.

[0014] Optionally, the large model deployed on the model driver side can be used to perform automatic labeling of multimodal training data, including: Use the large model deployed on the model driver to perform data mode conversion and data enhancement processing on the multimodal training data to obtain a training image dataset for automatic annotation. The large model deployed on the model driver is used to perform target category detection and behavior detection for each target category on the training image dataset. The detection results are annotated in the training images and converted into a text file in YOLO format. A training sample set is constructed and divided into a training set and a validation set.

[0015] Optionally, in step S5, the model application end is driven to use the training sample set to train and deploy the terminal category recognition model and the terminal behavior recognition model, specifically: obtain the official pre-trained model yolo11n.pt and import it into the model application end, drive the model application end to use the training sample set of target category detection and the training sample set of behavior detection of each target category to train the pre-trained model yolo11n.pt respectively, and deploy the generated terminal category recognition model and the terminal behavior recognition model of each target category on the model application end.

[0016] Optionally, step S6 specifically includes: S61: Each model application terminal uses the deployed terminal category recognition model to perform scene target category recognition of the target scene image in the corresponding detection scene; S62: Perform target segmentation on the target scene image according to the scene target category recognition result, and use the terminal behavior recognition model of each target category deployed on each model application end to perform scene target behavior recognition on the category in each image obtained by segmentation, and determine whether the behavior of the scene target is a prohibited behavior.

[0017] The beneficial effects of the present invention are as follows: a small model automatic training method based on multimodal large model cognitive intelligent drive is proposed. By obtaining each training sample generation task received in the target period and the task execution status of each model driving end, it is judged whether the model driving group meets the deadline requirement for executing the exclusive sample labeling strategy. If so, the allocation and automatic labeling actions for all multimodal training data in each training sample generation task are executed according to the exclusive sample labeling strategy. If not, the allocation and automatic labeling actions for all multimodal training data in each training sample generation task are executed according to the shared sample labeling strategy, and the training sample set obtained by automatic labeling is sent to the model application end for small model training for the corresponding detection scenario. Therefore, when it is detected that the task processing capacity of the model-driven group does not meet the current task execution requirements, the task processing workload of the model-driven group is reduced by selectively sharing the proportion of annotated images for the training sample generation task. By establishing a dual behavior judgment system between the model application end and the model driver end, the impact of shared sample annotation on the quality of annotation data and the recognition accuracy of the training model is minimized. While ensuring the accuracy of user annotation data and the amount of training data feedback to a certain extent, overload operation of hardware equipment deployed with large models is avoided, thereby balancing the training quality and training continuity during the peak period of model training demand. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The present invention is a flowchart of an embodiment of a small model automatic training method driven by multimodal large model cognitive intelligence.

[0019] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0021] The embodiment of the present invention provides a small model automatic training method based on multimodal large model cognitive intelligence drive, referring to Figure 1 , Figure 1 The present invention is a flowchart of an embodiment of a small model automatic training method driven by multimodal large model cognitive intelligence.

[0022] In this embodiment, a small model automatic training method based on multimodal large model cognitive intelligent driving is used in a model automatic training system, wherein the model automatic training system includes a model driving group having a plurality of model driving terminals, a model application group having a plurality of model application terminals, and a labeling task allocation server, and the method includes the following steps: S1: The labeling task allocation server receives the training sample generation task sent by each model application end in the target period and the task execution status sent by each model driver end; wherein the training sample generation task includes the labeling task deadline association information and several detection scene features; S2: The labeling task allocation server determines whether the model-driven group meets the deadline requirements for executing the exclusive sample labeling strategy based on the task execution status and the labeling task deadline association information; S3: If yes, the labeling task allocation server distributes the training sample generation task to the corresponding model driver according to the exclusive sample labeling strategy, and uses the large model deployed by the model driver to perform the automatic labeling action of the multimodal training data; S4: If not, the labeling task allocation server divides the training sample generation task into several general level categories based on several detection scene features, generates a shared sample labeling strategy for the training sample generation task for the associated general level based on the general level of each training sample generation task, and distributes the training sample generation task to the corresponding model driver, and uses the large model deployed by the model driver to perform the automatic labeling action of the multimodal training data; S5: Each model driver sends the training sample set obtained after performing the automatic labeling action of the multimodal training data to the model application end corresponding to the training sample generation task, driving the model application end to use the training sample set to train and deploy the terminal category recognition model and the terminal behavior recognition model; S6: Each model application terminal uses the deployed terminal category recognition model and terminal behavior recognition model to perform scene target category recognition and scene target behavior recognition of the target scene image in the corresponding detection scene, and determines whether the behavior of the scene target is a prohibited behavior; S7: If yes, the target scene image is sent to the verification model driver of the model driver group, and the prohibited behavior judgment verification is performed using the large model deployed by the verification model driver.

[0023] It should be noted that with the continuous growth of model training demand, the amount of data labeling tasks performed by the platform is gradually approaching the load limit of the hardware equipment, especially during the peak period of model training demand. The platform often needs to face short-term and overloaded tasks. At this time, the platform cannot reasonably allocate and process the labeling tasks, which will lead to inaccurate labeling data fed back to users, insufficient training data feedback, or overload of hardware equipment for deploying large models.

[0024] In order to solve the above problems, this embodiment obtains the task execution status of each training sample generation task and each model driver end received in the target period, determines whether the model driver group meets the deadline requirements for executing the exclusive sample annotation strategy, selects the exclusive sample annotation strategy or the shared sample annotation strategy to perform the allocation and automatic annotation actions for all multimodal training data in each training sample generation task, and sends the training sample set obtained by automatic annotation to the model application end for small model training for the corresponding detection scene. Therefore, by selectively sharing the proportion of annotated images for the training sample generation task and establishing a dual behavior judgment system for the model application end and the model driver end, the task processing volume can be reduced while minimizing the impact on the quality of the annotated data and the recognition accuracy of the training model.

[0025] In a preferred embodiment, step S1 specifically includes: S11: Each model application end collects multimodal training data and scene detection description information of the target detection scene, extracts the labeling task deadline association information and several scene detection features in the scene detection description information, and constructs them as task labels together with the multimodal training data to generate training sample tasks, and sends them to the labeling task allocation server; S12: Each model driver calls the training sample generation task list stored in itself, extracts the estimated execution period of each training sample generation task to be executed in the training sample generation task list, and sends it to the labeling task allocation server as the task execution status of each model driver; S13: The labeling task allocation server receives the training sample generation task sent by each model application end in the target period and the task execution status sent by each model driver end.

[0026] Furthermore, in step S11, the labeling task deadline association information and several scene detection features in the scene detection description information are extracted as task labels and are jointly constructed with the multimodal training data as a training sample generation task, specifically including: parsing the scene detection description information, extracting the sample generation deadline and several scene detection features in the scene detection description information, and generating the training data requirement for each training sample generation task based on the several scene detection features; summarizing the training data requirement and the sample generation deadline as labeling task deadline association information, and jointly constructing the labeling task deadline association information and several scene detection features as task labels and the multimodal training data as a training sample generation task, and sending them to the labeling task allocation server.

[0027] Furthermore, based on several scene detection features, the training data requirement for each training sample generation task is generated, specifically including: extracting the number of recognized object types of the recognized object features and the recognized scene categories determined by the recognized scene features from several scene detection features; calling a predefined relationship comparison table between the number of different recognized object types and the corresponding number of training images required under the recognized scene categories to match the training data requirement for each training sample generation task.

[0028] In this embodiment, by driving each model application end and each model driving end to send the training sample generation task and the estimated execution period of each training sample generation task to be executed to the labeling task allocation server respectively, and using the scene detection description information recorded in the training sample generation task and the task execution status determined by the estimated execution period, the model training requirements and task execution capabilities of the model application end are summarized to the labeling task allocation server, providing data support for the subsequent labeling task allocation server to judge the task completion deadline and select the sample labeling strategy.

[0029] In a preferred embodiment, step S2 specifically includes: S21: The labeling task allocation server extracts the estimated execution period of each to-be-executed training sample generation task of each model driver from the task execution status, and extracts the training data demand and sample generation deadline of each training sample generation task from the labeling task deadline association information; S22: performing ergodic allocation on all multimodal training data in each training sample generation task received within the target period, and obtaining a plurality of exclusive sample labeling schemes for allocating the multimodal training data in the training sample generation task to the corresponding model driver end; S23: according to the estimated execution period of each training sample generation task to be executed, the training data requirement and sample generation period of each training sample generation task, according to the standard annotation speed of each model driver and the total training data requirement of each training sample generation task, calculate the completion time after each model driver executes the annotation action of the allocated multimodal training data; S24: Determine whether there is an exclusive sample labeling scheme among several exclusive sample labeling schemes, in which the completion time of all model drivers executing the training sample generation task within the target period meets the sample generation deadline.

[0030] In this embodiment, firstly, ergodic allocation is performed on all multimodal training data in each training sample generation task received within the target time period to obtain several exclusive sample labeling schemes (i.e., all multimodal training data in each training sample generation task must perform labeling after being allocated to the model driver). By calculating the completion time after each model driver performs the labeling action on the allocated multimodal training data and comparing it with the sample generation deadline corresponding to the training sample generation task, it is determined whether the exclusive sample labeling scheme can be used to execute the training sample generation task allocation within the target time period.

[0031] In a preferred embodiment, step S3 specifically includes: S31: If yes, the labeling task allocation server extracts the target exclusive sample labeling scheme whose last completion time is the farthest from the sample generation deadline in the exclusive sample labeling strategy; S32: Based on the strategy of allocating all multimodal training data for each training sample generation task to the model driving end in the target exclusive sample labeling scheme, the corresponding multimodal training data in each training sample generation task is distributed to the corresponding model driving end, and the large model deployed by the model driving end is used to perform automatic labeling of the multimodal training data.

[0032] In this embodiment, when the completion time after each model driver performs the labeling action on the assigned multimodal training data can meet the sample generation deadline corresponding to the training sample generation task, the exclusive sample labeling scheme can be used to control all multimodal training data in each training sample generation task to perform the labeling action after being assigned to the model driver, and the multimodal training data after the labeling action is performed is used as the training image data set for the model application end to perform small model training for the detection scene.

[0033] In a preferred embodiment, step S4 specifically includes: S41: If not, the labeling task allocation server calculates the detection scene similarity between any two training sample generation tasks based on a number of detection scene features of each training sample generation task; S42: sorting the detection scene similarities between each training sample generation task and other training sample generation tasks from high to low, and judging whether the detection scene similarity with the highest ranking after sorting exceeds a preset similarity threshold; if so, defining the training sample generation task as a general-purpose scene task; if not, defining the training sample generation task as a non-general-purpose scene task; S43: according to the highest ranking detection scene similarity in the general-purpose scene task, all general-purpose scene tasks are divided into several general-purpose level categories, based on the training data demand and sample generation period of each training sample generation task, considering the estimated execution period of each to-be-executed training sample generation task of each model driver and the general-purpose level of each general-purpose scene task, ergodic allocation is performed on the multimodal training data in the training sample generation task received within the target period, and according to the principle of minimum shared general-purpose quantization value, a shared sample labeling strategy for the training sample generation task for the associated general-purpose level is generated; The minimum shared universal quantization value principle is specifically: First, all multimodal training data of non-general-purpose scenario tasks in the training sample generation task are ergodicly distributed to obtain several exclusive sample labeling schemes in which the multimodal training data in the non-general-purpose scenario tasks are cut in different proportions and distributed to the model driver end; Then, all multimodal training data of each general-purpose scenario task in the training sample generation task are ergodicly allocated in each exclusive sample annotation scheme according to the corresponding shared sample annotation ratio, so that the completion time of all model driving ends executing the training sample generation task within the target period meets the sample generation deadline, and the shared general-purpose quantization value of all general-purpose scenario tasks is minimized, and the shared general-purpose quantization value is configured as: the product and accumulation of the shared sample annotation ratio of each general-purpose scenario task and the training data sharing amount determined by the training data demand amount and the general level of the general-purpose scenario task; S44: According to the shared sample labeling strategy, the training sample generation task is distributed to the corresponding model driving end, and the large model deployed by the model driving end is used to perform automatic labeling of the multimodal training data. From the training image data set of the corresponding general-purpose scenario task with the highest detection scene similarity for each general-purpose scenario task, the training image data of a corresponding number of shared sample labeling ratios of the general-purpose scenario task are extracted and put into the training image data set of the general-purpose scenario task.

[0034] In this embodiment, by obtaining the task execution status of each training sample generation task and each model driver end received in the target time period, it is determined whether the model driver group meets the deadline requirement for executing the exclusive sample annotation strategy. If not, the allocation and automatic annotation actions for all multimodal training data in each training sample generation task are performed according to the shared sample annotation strategy (that is, all multimodal training data in each training sample generation task are reduced by the shared sample annotation ratio and then allocated to the model driver end to perform the annotation action. The multimodal training data lacking in the shared sample annotation ratio is extracted from the training image data set of the corresponding general scene task with the highest detection scene similarity for supplementation. It should be noted that the shared sample annotation ratio usually has a limiting threshold in actual applications. On the one hand, in order to ensure the quality of the output training image data set, on the other hand, it is also necessary to ensure that the missing multimodal training data set is not too much, so as to avoid the situation where the amount of data in the supplementary training image data set is insufficient), and the training sample set obtained by automatic annotation is sent to the model application end for small model training for the corresponding detection scene. Therefore, when it is detected that the task processing capacity of the model-driven group does not meet the current task execution requirements, the task processing workload of the model-driven group can be reduced by selectively sharing the labeled image ratio for the training sample generation task. This ensures the accuracy of user labeled data and the amount of training data feedback to a certain extent, while avoiding overload of hardware equipment for deploying large models.

[0035] In a preferred embodiment, the action of automatically labeling multimodal training data using the large model deployed on the model driver end includes: using the large model deployed on the model driver end to perform data mode conversion and data enhancement processing on the multimodal training data to obtain a training image data set for automatic labeling; using the large model deployed on the model driver end to perform target category detection and behavior detection for each target category on the training image data set, annotating the detection results in the training image and converting them into a text file in YOLO format, constructing a training sample set and dividing it into a training set and a validation set.

[0036] On this basis, in step S5, the model application end is driven to use the training sample set to train and deploy the terminal category recognition model and the terminal behavior recognition model, specifically: obtain the official pre-trained model yolo11n.pt and import it into the model application end, drive the model application end to use the training sample set of target category detection and the training sample set of behavior detection of each target category to train the pre-trained model yolo11n.pt respectively, and deploy the generated terminal category recognition model and the terminal behavior recognition model of each target category on the model application end.

[0037] In a preferred embodiment, step S6 specifically includes: S61: Each model application terminal uses the deployed terminal category recognition model to perform scene target category recognition of the target scene image in the corresponding detection scene; S62: Perform target segmentation on the target scene image according to the scene target category recognition result, and use the terminal behavior recognition model of each target category deployed on each model application end to perform scene target behavior recognition on the category in each image obtained by segmentation, and determine whether the behavior of the scene target is a prohibited behavior.

[0038] In this embodiment, by establishing a dual behavior judgment system of the model application end and the model driving end, the impact of shared sample annotation on the quality of annotation data and the recognition accuracy of the training model is minimized, thereby balancing the training quality and training continuity during the peak period of model training demand.

[0039] In addition, the present invention also proposes a small model automatic training device based on multimodal large model cognitive intelligence driving, and the small model automatic training device based on multimodal large model cognitive intelligence driving includes: a memory, a processor, and a small model automatic training program based on multimodal large model cognitive intelligence driving stored in the memory and executable on the processor. When the small model automatic training program based on multimodal large model cognitive intelligence driving is executed by the processor, the steps of the small model automatic training method based on multimodal large model cognitive intelligence driving as described above are implemented.

[0040] The specific implementation of the small model automatic training device based on multimodal large model cognitive intelligence driving in the present application is basically the same as the above-mentioned embodiments of the small model automatic training method based on multimodal large model cognitive intelligence driving, and will not be repeated here.

[0041] It is understood that, in the description of this specification, the description with reference to the terms "one embodiment", "another embodiment", "other embodiments", or "first to Nth embodiments" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0042] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0043] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A small model automated training method based on multimodal large model cognitive intelligence drive, characterized in that: For a model automation training system, the model automation training system includes a model driving group having a plurality of model driving terminals, a model application group having a plurality of model application terminals, and a labeling task allocation server, the method includes the following steps: S1: The labeling task allocation server receives the training sample generation task sent by each model application end in the target period and the task execution status sent by each model driver end; wherein the training sample generation task includes the labeling task deadline association information and several detection scene features; S2: The labeling task allocation server determines whether the model-driven group meets the deadline requirements for executing the exclusive sample labeling strategy based on the task execution status and the labeling task deadline association information; S3: If yes, the labeling task allocation server distributes the training sample generation task to the corresponding model driver according to the exclusive sample labeling strategy, and uses the large model deployed by the model driver to perform the automatic labeling action of the multimodal training data; S4: If not, the labeling task allocation server divides the training sample generation task into several general level categories based on several detection scene features, generates a shared sample labeling strategy for the training sample generation task for the associated general level based on the general level of each training sample generation task, and distributes the training sample generation task to the corresponding model driver, and uses the large model deployed by the model driver to perform the automatic labeling action of the multimodal training data; S5: Each model driver sends the training sample set obtained after performing the automatic labeling action of the multimodal training data to the model application end corresponding to the training sample generation task, driving the model application end to use the training sample set to train and deploy the terminal category recognition model and the terminal behavior recognition model; S6: Each model application terminal uses the deployed terminal category recognition model and terminal behavior recognition model to perform scene target category recognition and scene target behavior recognition of the target scene image in the corresponding detection scene, and determines whether the behavior of the scene target is a prohibited behavior; S7: If yes, the target scene image is sent to the verification model driver of the model driver group, and the prohibited behavior judgment verification is performed using the large model deployed by the verification model driver.

2. The small model automatic training method based on multimodal large model cognitive intelligence drive as claimed in claim 1 is characterized in that: Step S1 specifically includes: S11: Each model application end collects multimodal training data and scene detection description information of the target detection scene, extracts the labeling task deadline association information and several scene detection features in the scene detection description information, and constructs them as task labels together with the multimodal training data to generate training sample tasks, and sends them to the labeling task allocation server; S12: Each model driver calls the training sample generation task list stored in itself, extracts the estimated execution period of each training sample generation task to be executed in the training sample generation task list, and sends it to the labeling task allocation server as the task execution status of each model driver; S13: The labeling task allocation server receives the training sample generation task sent by each model application end in the target period and the task execution status sent by each model driver end.

3. The small model automatic training method based on multimodal large model cognitive intelligence drive as claimed in claim 2 is characterized in that: In step S11, the labeling task deadline association information and several scene detection features in the scene detection description information are extracted as task labels and constructed together with the multimodal training data to form a training sample generation task, which specifically includes: Parsing the scene detection description information, extracting the sample generation period and several scene detection features in the scene detection description information, and generating the training data requirement for each training sample generation task based on the several scene detection features; The training data demand and sample generation deadline are summarized as labeling task deadline association information, and the labeling task deadline association information and several scene detection features are used as task labels together with multimodal training data to form a training sample generation task, which is sent to the labeling task allocation server.

4. The small model automated training method based on multimodal large model cognitive intelligence drive as claimed in claim 3, characterized in that: Based on several scene detection features, the training data requirements for each training sample generation task are generated, including: Extracting the number of recognized object types of recognized object features and the recognized scene categories determined by recognized scene features from a plurality of scene detection features; Call the predefined comparison table of the relationship between the number of different types of objects to be recognized and the corresponding number of training images required under the recognition scene category to match the training data requirement for each training sample generation task.

5. The small model automatic training method based on multimodal large model cognitive intelligence drive as claimed in claim 1, characterized in that: Step S2 specifically includes: S21: The labeling task allocation server extracts the estimated execution period of each to-be-executed training sample generation task of each model driver from the task execution status, and extracts the training data demand and sample generation deadline of each training sample generation task from the labeling task deadline association information; S22: performing ergodic allocation on all multimodal training data in each training sample generation task received within the target period, and obtaining a plurality of exclusive sample labeling schemes for allocating the multimodal training data in the training sample generation task to the corresponding model driver end; S23: according to the estimated execution period of each training sample generation task to be executed, the training data requirement and sample generation period of each training sample generation task, according to the standard annotation speed of each model driver and the total training data requirement of each training sample generation task, calculate the completion time after each model driver executes the annotation action of the allocated multimodal training data; S24: Determine whether there is an exclusive sample labeling scheme among several exclusive sample labeling schemes, in which the completion time of all model drivers executing the training sample generation task within the target period meets the sample generation deadline.

6. The small model automatic training method based on multimodal large model cognitive intelligence drive as claimed in claim 5, characterized in that: Step S3 specifically includes: S31: If yes, the labeling task allocation server extracts the target exclusive sample labeling scheme whose last completion time is the farthest from the sample generation deadline in the exclusive sample labeling strategy; S32: Based on the strategy of allocating all multimodal training data for each training sample generation task to the model driving end in the target exclusive sample labeling scheme, the corresponding multimodal training data in each training sample generation task is distributed to the corresponding model driving end, and the large model deployed by the model driving end is used to perform automatic labeling of the multimodal training data.

7. The small model automatic training method based on multimodal large model cognitive intelligence drive as claimed in claim 1, characterized in that: Step S4 specifically includes: S41: If not, the labeling task allocation server calculates the detection scene similarity between any two training sample generation tasks based on a number of detection scene features of each training sample generation task; S42: sorting the detection scene similarities between each training sample generation task and other training sample generation tasks from high to low, and judging whether the detection scene similarity with the highest ranking after sorting exceeds a preset similarity threshold; if so, defining the training sample generation task as a general-purpose scene task; if not, defining the training sample generation task as a non-general-purpose scene task; S43: according to the highest ranking detection scene similarity in the general-purpose scene task, all general-purpose scene tasks are divided into several general-purpose level categories, based on the training data demand and sample generation period of each training sample generation task, considering the estimated execution period of each to-be-executed training sample generation task of each model driver and the general-purpose level of each general-purpose scene task, ergodic allocation is performed on the multimodal training data in the training sample generation task received within the target period, and according to the principle of minimum shared general-purpose quantization value, a shared sample labeling strategy for the training sample generation task for the associated general-purpose level is generated; The minimum shared universal quantization value principle is specifically: First, all multimodal training data of non-general-purpose scenario tasks in the training sample generation task are ergodicly distributed to obtain several exclusive sample labeling schemes in which the multimodal training data in the non-general-purpose scenario tasks are cut in different proportions and distributed to the model driver end; Then, all multimodal training data of each general-purpose scenario task in the training sample generation task are ergodicly allocated in each exclusive sample annotation scheme according to the corresponding shared sample annotation ratio, so that the completion time of all model driving ends executing the training sample generation task within the target period meets the sample generation deadline, and the shared general-purpose quantization value of all general-purpose scenario tasks is minimized, and the shared general-purpose quantization value is configured as: the product and accumulation of the shared sample annotation ratio of each general-purpose scenario task and the training data sharing amount determined by the training data demand amount and the general level of the general-purpose scenario task; S44: According to the shared sample labeling strategy, the training sample generation task is distributed to the corresponding model driving end, and the large model deployed by the model driving end is used to perform automatic labeling of the multimodal training data. From the training image data set of the corresponding general-purpose scenario task with the highest detection scene similarity for each general-purpose scenario task, the training image data of a corresponding number of shared sample labeling ratios of the general-purpose scenario task are extracted and put into the training image data set of the general-purpose scenario task.

8. The small model automatic training method based on multimodal large model cognitive intelligence drive as described in claim 6 or 7, characterized in that: The automatic labeling actions of multimodal training data using the large model deployed on the model driver include: Use the large model deployed on the model driver to perform data mode conversion and data enhancement processing on the multimodal training data to obtain a training image dataset for automatic annotation. The large model deployed on the model driver is used to perform target category detection and behavior detection for each target category on the training image dataset. The detection results are annotated in the training images and converted into a text file in YOLO format. A training sample set is constructed and divided into a training set and a validation set.

9. The small model automated training method based on multimodal large model cognitive intelligence drive as claimed in claim 8, characterized in that: In step S5, the model application end is driven to use the training sample set to train and deploy the terminal category recognition model and the terminal behavior recognition model, specifically: obtain the official pre-trained model yolo11n.pt and import it into the model application end, drive the model application end to use the training sample set of target category detection and the training sample set of behavior detection of each target category to train the pre-trained model yolo11n.pt respectively, and deploy the generated terminal category recognition model and the terminal behavior recognition model of each target category on the model application end.

10. The small model automatic training method based on multimodal large model cognitive intelligence drive according to claim 9, characterized in that: Step S6 specifically includes: S61: Each model application terminal uses the deployed terminal category recognition model to perform scene target category recognition of the target scene image in the corresponding detection scene; S62: Perform target segmentation on the target scene image according to the scene target category recognition result, and use the terminal behavior recognition model of each target category deployed on each model application end to perform scene target behavior recognition on the category in each image obtained by segmentation, and determine whether the behavior of the scene target is a prohibited behavior.

Images