Model training method and device and readable storage medium

Abstract

The invention provides a model training method and device and a readable storage medium. The method comprises the following steps: controlling camera equipment to collect a first image in the current round of model training; the image recognition model recognizes the first image according to the structured cue word to obtain a first recognition result; adjusting the structured cue word according to the first feedback result to obtain an updated structured cue word; when the recognition accuracy of the first recognition result is smaller than an accuracy threshold, controlling the camera device to collect a second image; the image recognition model recognizes a second image according to the updated structured cue word to obtain a second recognition result; adjusting the updated structured cue word according to the second recognition result and the second feedback result to obtain a structured cue word for the next round of model training; and when the model training state meets a preset condition, obtaining a trained image recognition model. The structured cue word is optimized according to the recognition result and the feedback result, the semantic consistency between the structured cue word and the content of the image is improved, and the training effect is improved.

Description

Technical Field

[0001] This application belongs to the field of image recognition technology, and in particular relates to a model training method, apparatus and readable storage medium. Background Technology

[0002] Existing methods for training large language models include collecting samples via camera, filtering samples to obtain a static sample library, manually setting structured prompts, and training the large language model using the static sample library and structured prompts until the model meets the requirements of the scenario.

[0003] However, training a large language model using a static sample library and manually set structured prompts may result in a mismatch between the structured prompts and the actual static sample library, leading to poor model training performance. Summary of the Invention

[0004] This application provides a model training method, apparatus, and readable storage medium, which can solve the problem of poor model training effect caused by the mismatch between structured prompt words and the actual static sample library.

[0005] In a first aspect, embodiments of this application provide a model training method, including:

[0006] In this round of model training, the camera device is controlled to capture the first image; The first image is identified using an image recognition model based on structured prompts to obtain a first recognition result. The structured prompts are the prompts obtained after the previous round of model training. The structured prompts are adjusted based on the first feedback result to obtain updated structured prompts. The first feedback result is the correct recognition result of the first image. If the recognition accuracy of the first recognition result is less than the accuracy threshold, the camera device is controlled to acquire a second image, and the image content of the second image matches the semantics of the updated structured prompt words. The image recognition model is used to recognize the second image based on the updated structured prompts to obtain a second recognition result; The updated structured prompt words are adjusted according to the second recognition result and the second feedback result to obtain structured prompt words for the next round of model training, and then proceed to the following steps: In this round of model training, the camera device is controlled to acquire the first image, and the second feedback result is the correct recognition result of the second image; If the training state of the image recognition model meets the preset training conditions, then the trained image recognition model is obtained.

[0007] In some embodiments, controlling the camera device to acquire a second image includes: Determine the interference features in the first image, wherein the interference features are image features identified by the interference model; Based on the interference features, the first feedback result, and the updated structured prompt words, an image acquisition strategy is determined; According to the image acquisition strategy, the pose parameters and / or imaging parameters of the camera device are adjusted so that the camera device can acquire a second image.

[0008] In some embodiments, controlling the camera device to acquire a second image includes: Control the camera device to acquire new images; Calculate the similarity between the image features of the new image and the updated structured prompts; If the similarity is greater than or equal to the similarity threshold, then the second image is obtained; If the similarity is less than the similarity threshold, the camera device continues to acquire new images until a new image with a similarity greater than or equal to the similarity threshold is acquired, and the currently acquired new image is used as the second image.

[0009] In some embodiments, adjusting the updated structured prompts based on the second recognition result and the second feedback result to obtain structured prompts for the next round of model training includes: Determine the semantic deviation information between the second feedback result and the updated structured prompt words; Based on the second recognition result and the semantic deviation information, the structure of the updated structured prompt is adjusted. The structure of the structured prompt includes role information, template information, and reasoning constraint information.

[0010] In some embodiments, the role information includes task type, industry background and professional attributes; the template information includes the object to be detected, scene description, attribute features, recognition conditions and output format; and the inference constraint information includes viewpoint constraints, time constraints, lighting constraints, confidence constraints and output format constraints.

[0011] In some embodiments, controlling the camera device to acquire a first image includes: Control the camera device to acquire the initial image; The initial image and the initial structured prompts are semantically matched and analyzed to obtain a first semantic matching degree. The initial structured prompts are generated based on the natural language description input by the user. If the first semantic matching degree is greater than or equal to the matching degree threshold, then the initial image is deduplicated and extracted to obtain the first image.

[0012] In some embodiments, if the recognition accuracy of the first recognition result is less than an accuracy threshold, the method further includes the following steps before controlling the camera device to acquire the second image: The first identification result is semantically matched with the structured prompt words to obtain a second semantic matching degree; The recognition accuracy is determined based on the second semantic matching degree.

[0013] Secondly, embodiments of this application provide a model training apparatus, including: The image acquisition module is used to control the camera device to acquire the first image during this round of model training; It is also used to control the camera device to acquire a second image if the recognition accuracy of the first recognition result is less than the accuracy threshold, wherein the image content of the second image matches the semantics of the updated structured prompt word; The model training module is used to use an image recognition model to recognize the first image based on structured prompts and obtain a first recognition result. The structured prompts are the prompts obtained after the previous round of model training. It is also used to use the image recognition model to recognize the second image based on the updated structured prompts, and obtain a second recognition result; It is also used to obtain a trained image recognition model if the training state of the image recognition model meets the preset training conditions. The structured prompt word optimization module is used to adjust the structured prompt words according to the first feedback result to obtain the updated structured prompt words, wherein the first feedback result is the correct recognition of the first image; It is also used to adjust the updated structured prompt words according to the second recognition result and the second feedback result to obtain structured prompt words for the next round of model training, and proceed to the step: in this round of model training, control the camera device to acquire the first image, and the second feedback result is the correct recognition result of the second image.

[0014] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any of the first aspects above.

[0015] The beneficial effects of the embodiments in this application compared with the prior art are: In this embodiment, the structured prompts are adjusted based on the first feedback result in each round of model training to obtain updated structured prompts. The first feedback result is the correct recognition result of the first image. If the recognition accuracy of the first recognition result is less than the accuracy threshold, the camera device is controlled to acquire a second image. The image content of the second image matches the semantics of the updated structured prompts. The prompts are optimized using the feedback result and the images matched with the optimized prompts are acquired to improve the semantic consistency between the semantics of the structured prompts and the content of the images. The system utilizes an image recognition model to identify the second image based on the updated structured prompts, obtaining a second recognition result. Based on the second recognition result and the second feedback result, the updated structured prompts are adjusted to obtain structured prompts for the next round of model training. The prompts are further optimized based on the feedback and recognition results to improve the semantic consistency between the semantics of the structured prompts and the content of the image, thereby improving the training effect.

[0016] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the first flowchart of the model training method provided in the embodiments of this application; Figure 2 This is a schematic diagram of the second process of the model training method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of the model training device provided in the embodiments of this application. Detailed Implementation

[0019] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0020] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0021] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0022] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0023] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0025] In some embodiments, Figure 1 This is a schematic diagram of the first flowchart of the model training method provided in this application embodiment. The method is applied to a cloud server in the model training system.

[0026] In some implementations, the method can be specifically executed by a model training device, which can be implemented in software and / or hardware and can be configured on the aforementioned cloud server.

[0027] Figure 1 The method shown includes steps S11 to S16, which will be described in detail below.

[0028] S11: In this round of model training, control the camera device to acquire the first image.

[0029] The cloud server communicates with the camera equipment. During each round of model training (i.e., entering the current training cycle), the cloud server controls the camera equipment to capture images of the current environment in real time according to the current camera parameters. The camera equipment uploads the image data stream to the cloud server, which processes the image data stream to obtain the first image.

[0030] S12: Use an image recognition model to recognize the first image based on structured prompts to obtain the first recognition result.

[0031] Among them, the structured prompts are the prompts obtained after the previous round of model training.

[0032] In the first round of model training, the cloud server receives the user's intent, which is composed of natural language. The cloud server performs semantic analysis on the user's intent based on a pre-defined industry knowledge base, generating structured prompts. The structured prompts input to the image recognition model are those generated based on the user's intent. In each subsequent round of model training, the structured prompts input to the image recognition model are those obtained after the previous round of model training, i.e., the optimized prompts from the previous round of training.

[0033] In each round of model training, structured prompts and the first image are jointly input into the image recognition model. The structured prompts guide the image recognition model to focus on the image features in the first image and understand its content. These image features represent specific semantic features, such as frontal or profile views, or images of a trash can.

[0034] The image recognition model performs joint reasoning based on the semantics of structured prompts and the image content of the first image, and outputs the first recognition result.

[0035] In some embodiments, the image recognition model outputs the image category in an image recognition task.

[0036] In some embodiments, the image recognition model outputs the type and location of the target in a target detection task.

[0037] In some embodiments, the image recognition model outputs a state result in a state analysis task.

[0038] In some implementations, the structured prompts and the first image can be concatenated and then input into the image recognition model, thus enabling the structured prompts and the first image to be jointly input into the image recognition model.

[0039] S13: Adjust the structured prompts based on the first feedback result to obtain the updated structured prompts.

[0040] The first feedback result is the correct recognition result of the first image.

[0041] In some implementations, semantic analysis can be performed on the first identification result to determine whether the first identification result is correct, and then the first identification result can be confirmed or corrected to obtain the first feedback result.

[0042] In some implementations, the first feedback result can be obtained after the first identification result is confirmed or corrected manually.

[0043] The cloud server encodes the initial feedback result and the structured prompts into two vectors. These two vectors are compared to determine the semantic axis and keywords where semantic deviations occur, thus obtaining semantic deviation information. This information is then analyzed to identify the causes of the semantic deviations. Based on these causes, the structured prompts are semantically corrected and / or structurally updated to obtain the updated structured prompts.

[0044] S14: If the recognition accuracy of the first recognition result is less than the accuracy threshold, then control the camera device to acquire the second image.

[0045] The cloud server calculates the recognition accuracy of the first recognition result under the semantic expression of the structured prompts. When the recognition accuracy of the first recognition result is less than the accuracy threshold, it indicates that the semantics of the first recognition result is inconsistent with the structured prompts, and the first image is not the sample required for current training. It is necessary to drive the camera device to collect key samples.

[0046] A cloud server drives a camera device to capture images that semantically match the updated structured prompts. The image data stream is then uploaded to the cloud server, which processes the stream to obtain a second image. The content of this second image matches the semantics of the updated structured prompts, and this second image is used for joint retraining of the structured prompts and the model.

[0047] S15: Use an image recognition model to recognize the second image based on the updated structured prompts to obtain the second recognition result.

[0048] The updated structured prompts and the second image are jointly input into the image recognition model. The updated structured prompts guide the image recognition model to focus on the image features in the second image and understand the image content of the second image.

[0049] The image recognition model performs joint reasoning based on the semantics of the updated structured cue words and the image content of the second image, and outputs a second recognition result.

[0050] S16: Adjust the updated structured prompt words according to the second recognition result and the second feedback result to obtain structured prompt words for the next round of model training, and proceed to step S11: In this round of model training, control the camera device to acquire the first image.

[0051] The second image is one that semantically matches the updated structured prompt. In this case, the second recognition result of the second image better reflects the reasoning ability of the image recognition model. The updated structured prompt can be optimized using the second recognition result and the second feedback result to make the further optimized structured prompt more consistent with the semantics that the image recognition model can understand, and to enhance the image recognition model's ability to understand user intent.

[0052] In some implementations, semantic analysis can be performed on the second identification result to determine whether the second identification result is correct, and then the second identification result can be confirmed or corrected to obtain a second feedback result.

[0053] In some implementations, a second feedback result can be obtained after the second identification result is manually confirmed or corrected.

[0054] In some implementations, step S16 includes: S161: Determine the semantic deviation information between the second feedback result and the updated structured prompt words.

[0055] The cloud server performs text encoding on the second feedback result and the updated structured prompts, obtaining two vectors. The two vectors are compared to determine the semantic axis and keywords where the semantic deviation occurs, thus obtaining semantic deviation information.

[0056] S162: Based on the second recognition result and semantic bias information, adjust the structural content of the updated structured prompt words to obtain structured prompt words for the next round of model training.

[0057] The updated structured prompts include role information, template information, and reasoning constraint information.

[0058] Based on the second recognition result, the causes of poor recognition are determined, and semantic deviation information is analyzed to identify the root causes of semantic deviations. Based on the semantic deviations and the causes of poor recognition, specific structural information is adjusted. If the cause is inaccurate role positioning, the role information is adjusted. If the cause is inaccurate output template information, the output template information is adjusted. If the cause is inaccurate determination of inference constraint information, the inference constraint information is adjusted.

[0059] In some implementations, the second recognition result is semantically matched with the updated structured prompts. Based on the semantic analysis results, the image features of the second image that caused the poor recognition are determined, thereby identifying the cause of the poor recognition.

[0060] For example, when analyzing the image features of the second image that reveal most of the face, resulting in poor recognition, the cause of the poor recognition is found to be the revelation of most of the face.

[0061] After adjusting the updated structured prompts based on the second recognition result and the second feedback result, the updated structured prompts are optimized to obtain structured prompts for the next round of model training. Then, a new round of model training begins, and the cloud server re-executes steps S11 to S16.

[0062] Understandably, the adaptive, two-way closed loop of acquiring new images and optimizing structured prompts enables semantic consistency analysis and dynamic adaptation between the acquired images and structured prompts. This enhances the image recognition model's ability to understand user intent, ensuring the model's accuracy and generalization ability even in complex scenarios. Simultaneously, it reduces manual intervention in structured prompt optimization and image acquisition, improving the method's applicability and deployment efficiency.

[0063] In some embodiments, the cloud server can record generation version information and optimization version information of the structured prompt words. Version information includes metadata such as generation time, semantic structure, training accuracy, usage scenario, and recognition target. Users can compare versions or roll back by viewing historical training results. For example, historical training results include recognition accuracy, training convergence speed, and false recognition rate.

[0064] In some implementations, role information includes task type, industry background, and professional attributes.

[0065] Role information is used to define the professional role that the image recognition model plays in the current task, and the semantic role that matches the current task, so that the image recognition model can reason in a specific semantic space and reduce semantic divergence.

[0066] For example, the user intent is to detect personnel in a factory who are not wearing safety helmets. Your role information is "Industrial Safety Inspection Expert."

[0067] In some implementations, the template information includes the object to be detected, scene description, attribute features, recognition conditions, and output format.

[0068] Template information is used to standardize the structural expression of structured prompts, making them more consistent with the input format of the model and the semantic structure of the task type, thereby improving the trainability of structured prompts in the model.

[0069] For example, the output format is: Detect whether the [object to be detected] possesses the [attribute] in the [scene], with a confidence level > [threshold]. The user intent is whether anyone is not wearing a helmet, which is converted into detecting whether [personnel] lack [head protection equipment] in the [workshop / factory] according to the template information, with a confidence level threshold > 0.7.

[0070] In some implementations, inference constraint information includes viewpoint constraints, time constraints, lighting constraints, confidence constraints, and output format constraints.

[0071] Inference constraints are used to limit the context of model inference, reduce the model's identification of non-target features or misidentification of target features, and improve the consistency and accuracy of identification.

[0072] For example, the user intent is to identify people not wearing safety helmets during the day, from a clear frontal view. The corresponding constraints include viewpoint constraints, time constraints, and lighting constraints, specifically identifying facial features of people only during daytime, from a frontal view, and under sufficient lighting conditions.

[0073] S17: If the training state of the image recognition model meets the preset training conditions, then the trained image recognition model is obtained.

[0074] In some implementations, the image recognition model is a large, untrained model. The conditions that trigger model training include: the recognition accuracy is less than an accuracy threshold; the number of corrections exceeds a preset proportion; new image features appear in the image; images that are difficult to recognize appear; and the semantics of the structured prompts are inconsistent with the recognition results. The conditions that stop model training are: the recognition accuracy is greater than or equal to the accuracy threshold; the number of corrections is less than a preset proportion; no new image features appear in the image; no images that are difficult to recognize appear; and the semantics of the structured prompts are consistent with the recognition results.

[0075] In some implementations, the image recognition model is a large model that has been pre-trained and supports fine-tuning. The conditions that trigger model fine-tuning include the appearance of images with new recognition results, newly acquired structured prompts, new requirements for the semantic matching degree between the structured prompts and the recognition results, and the collection of new image features. The conditions for stopping fine-tuning are: incremental training is completed when images with new recognition results and optimized structured prompts appear, new image features are collected, and the semantic matching degree between the structured prompts and the recognition results meets new requirements.

[0076] In this embodiment, the structured prompts are adjusted based on the first feedback result in each round of model training to obtain updated structured prompts. The first feedback result is the correct recognition result of the first image. If the recognition accuracy of the first recognition result is less than the accuracy threshold, the camera device is controlled to acquire a second image. The image content of the second image matches the semantics of the updated structured prompts. The prompts are optimized using the feedback result and the images matched with the optimized prompts are acquired to improve the semantic consistency between the semantics of the structured prompts and the content of the images. The system utilizes an image recognition model to identify the second image based on the updated structured prompts, obtaining a second recognition result. Based on the second recognition result and the second feedback result, the updated structured prompts are adjusted to obtain structured prompts for the next round of model training. The prompts are further optimized based on the feedback and recognition results to improve the semantic consistency between the semantics of the structured prompts and the content of the image, thereby improving the training effect.

[0077] In some embodiments, image samples with high recognition value are retained to improve training effectiveness.

[0078] Correspondingly, step S11 includes: S111: Control the camera equipment to acquire the initial image.

[0079] S112: Perform semantic matching analysis between the initial image and the initial structured prompts to obtain the first semantic matching degree.

[0080] The initial structured prompts are generated based on the natural language description input by the user.

[0081] After receiving the natural language description input by the user, semantic analysis is performed on the natural language description, and corresponding information is generated according to the structure of preset prompt words to generate initial structured prompt words. Among them, the natural language description represents the user's intent.

[0082] S113: If the first semantic matching degree is greater than or equal to the matching degree threshold, then the initial image is deduplicated and extracted to obtain the first image.

[0083] During the construction of training samples, after the cloud server obtains the image data stream uploaded by the camera device, it performs semantic matching analysis between the initial image and the initial structured prompt words.

[0084] If the initial semantic match between the initial image and the initial structured prompt is greater than or equal to the match threshold, the initial image is determined to contain image features related to the initial structured prompt. This indicates that the image content of the initial image matches the semantics of the initial structured prompt, making it a high-value image sample. The high-value image sample (i.e., the first image) is retained, and deduplication, keyframe extraction, and feature extraction are performed on it.

[0085] For example, image samples whose image content matches the semantics of structured prompts contain targets with complete features, high resolution, and few interfering factors, making the recognition results verifiable and serving as high-quality training data.

[0086] In some implementations, the image features of the initial image and the features of the initial structured prompt words can be mapped to the same semantic space to obtain two vectors. The similarity between the two vectors is then calculated to obtain the first semantic matching degree.

[0087] In some implementations, if the first semantic matching degree is less than the matching degree threshold, it is determined that the image does not contain image features related to the initial structured prompt words. In this case, the initial image is discarded locally and does not participate in training or compression processing.

[0088] This application embodiment obtains a first image by performing deduplication and extraction processing on the initial image if the first semantic matching degree is greater than the matching degree threshold. It retains image samples with high recognition value while not retaining irrelevant images and images with low recognition value, thereby providing high-quality training data for model training and structured prompt word optimization, saving image acquisition bandwidth and storage resources, and improving resource utilization.

[0089] In some embodiments, prior to step S13, the method further includes: S21: Perform semantic matching analysis between the first recognition result and the structured prompt words to obtain the second semantic matching degree.

[0090] S22: Determine the recognition accuracy based on the second semantic matching degree.

[0091] In some implementations, the features of the first recognition result and the features of the structured prompt words can be mapped to the same semantic space to obtain two vectors. The similarity between the two vectors is then calculated to obtain the second semantic matching degree.

[0092] In practice, image recognition models focus on the image features of a first image based on the semantics of structured prompts. The second semantic matching degree characterizes the recognition accuracy of the image recognition model. The recognition accuracy of the first image is determined based on the second semantic matching degree. When the first image does not contain the features described by the structured prompts, the second semantic matching degree is low, and the corresponding recognition accuracy is low. When the first image contains the features described by the structured prompts, the second semantic matching degree is high, and the corresponding recognition accuracy is high.

[0093] In some implementations, the second semantic matching degree can be directly used as the recognition accuracy.

[0094] In some implementations, the second semantic matching degree can be weighted to obtain the recognition accuracy.

[0095] For example, the structured prompt is "detect trash can". The first recognition result is semantically matched with the structured prompt to determine whether the image contains the target image feature: trash can, and whether it contains irrelevant image features: people, animals, etc. When the image only contains trash cans, the recognition accuracy is high. When the image only contains people, animals, etc., the recognition accuracy is low.

[0096] This application embodiment obtains a second semantic matching degree by performing semantic matching analysis between the first recognition result and the structured prompt words. Based on the second semantic matching degree, the recognition accuracy is determined. By performing semantic matching analysis between the recognition result and the structured prompt words, the semantic binding between the structured prompt words and the image acquisition is realized, and the recognition accuracy of the image is accurately judged.

[0097] In some embodiments, Figure 2 This is a schematic diagram of the second flowchart of the model training method provided in the embodiments of this application. For example... Figure 2 As shown, when the settings allow adjustment of the camera device's physical parameters, the specific methods for driving the camera device to acquire images include: S141: Determine the interference features in the first image.

[0098] The cloud server analyzes the image features of the first image to determine the image features of the target in the first image, obtaining interference features. Interference features are the image features recognized by the interference model. Interference features affect the inference of the image recognition model, leading to poor image recognition performance and thus outputting inaccurate recognition results.

[0099] S142: Determine the image acquisition strategy based on the interference characteristics, the first feedback result, and the updated structured prompts.

[0100] The cloud server performs semantic analysis on the first feedback result to determine the first target feature for improving recognition accuracy. It also performs semantic matching between the updated structured prompts and the second recognition result, and then determines the second target feature for improving recognition accuracy based on the semantic matching result. Based on the interference features, the first target feature, and the second target feature, the cloud server adjusts the current image acquisition parameters of the camera device to determine the corresponding image acquisition strategy.

[0101] As an example, if the user's intent is to detect workers not wearing safety helmets, the corresponding model identifies the target as a person. The facial features in the first image are analyzed to determine if the current facial feature is a profile or a back view, thus obtaining interfering features. Furthermore, the analysis of the first feedback result shows that images of frontal faces can improve recognition accuracy, obtaining the first target feature as a frontal face. Semantic matching analysis is performed between the updated structured prompts and the first recognition result, determining that unobstructed faces can improve recognition accuracy, obtaining the second target feature as an unobstructed face. The cloud server then determines the image strategy to collect images of unobstructed, frontal faces, excluding images of profiles or back views.

[0102] S143: Adjust the pose parameters and / or imaging parameters of the camera device according to the image acquisition strategy so that the camera device can acquire a second image.

[0103] The cloud server generates corresponding adjustment instructions based on the image acquisition strategy. These instructions include at least one of pose parameters and imaging parameters. The cloud server sends these adjustment instructions to the camera device to drive the camera device to adaptively acquire image samples that do not contain interfering features and match the semantics of the updated structured prompts, thereby obtaining the second image.

[0104] In some implementations, pose parameters include angle parameters, and imaging parameters include resolution parameters, frame rate parameters, and illumination parameters. Correspondingly, adjustment commands include at least one of the following: angle parameters, resolution parameters, frame rate parameters, and illumination parameters.

[0105] In some implementations, during the process of acquiring new images, the camera device performs deduplication, keyframe extraction, and image feature extraction on the image data stream to preprocess the image and obtain the required image samples.

[0106] This application embodiment determines interference features in the first image, and determines an image acquisition strategy based on the interference features, the first feedback result, and the updated structured prompts. According to the image acquisition strategy, the pose parameters and / or imaging parameters of the camera device are adjusted so that the camera device can acquire the second image, drive the camera device to acquire key samples, enhance the acquisition of image samples that match the semantics of the updated structured prompts, reduce the interference of images with poor recognition performance on the model training process, and improve training efficiency.

[0107] To better understand the methods described in the embodiments of this application, an example is provided for illustration. The user inputs their intent: Is anyone not wearing a helmet? Based on the user intent, a structured prompt is generated: Detect people in the park who are not wearing safety helmets. The image recognition model performs image recognition, but the accuracy of the output recognition result is lower than the accuracy threshold. The cloud server identifies interfering features: side profile or back view; the first target feature is a frontal face; and the second target feature is an unobstructed face. The cloud server then determines the image strategy to collect images of unobstructed, frontal faces, and not to collect images of side profiles or back views. The cloud server adjusts the instructions to control the camera device to move its angle and collect a second image containing a frontal face, increasing the frequency of unobstructed, frontal shots. After uploading the second image, the image recognition model performs image recognition. Based on the second feedback result and the second recognition result, the updated structured prompt is adjusted to "Only recognize people whose faces are unobstructed and whose heads are not wearing protective equipment from a frontal view." The adjusted structured prompt is used to retrain the image recognition model to improve recognition accuracy.

[0108] In some embodiments, when adjusting the physical parameters of the camera device is not allowed, the specific methods for driving the camera device to acquire images include: S141`: Controls the camera equipment to acquire new images.

[0109] S142: Calculate the similarity between the image features of the new image and the updated structured cue words.

[0110] The cloud server drives the camera device to acquire new images, performs image feature analysis on the new images, and calculates the similarity between the image features of the new images and the updated structured prompts.

[0111] In some implementations, the image features of the new image and the features of the updated structured prompts are mapped to the same semantic space to obtain two vectors. The similarity between the two vectors is then calculated to obtain the similarity between the image features of the new image and the updated structured prompts.

[0112] S143: If the similarity is greater than or equal to the similarity threshold, then the second image is obtained.

[0113] When the similarity is greater than or equal to the similarity threshold, it means that image features related to the updated structured cue words have been collected. The image features of the new image match the semantics of the updated structured cue words, making it a high-value image sample for identification.

[0114] S144`: If the similarity is less than the similarity threshold, continue to control the camera device to acquire new images until a new image with a similarity greater than or equal to the similarity threshold is acquired, and use the currently acquired new image as the second image.

[0115] When the similarity is less than the similarity threshold, it indicates that image features unrelated to the updated structured prompts have been collected. The image features of the new image do not semantically match the updated structured prompts, and to identify low-value image samples, new images need to be collected. The camera device is then controlled to continue collecting new images. After a new image is collected, the similarity between the currently collected new image and the updated structured prompts is calculated. If the similarity between the currently collected new image and the updated structured prompts is less than the similarity threshold, new images are collected again. The similarity between the currently collected new image and the updated structured prompts is then calculated again. If the similarity between the currently collected new image and the updated structured prompts is greater than or equal to the similarity threshold, the currently collected new image is used as the second image, and the collection of the second image ends. If the similarity between the currently collected new image and the updated structured prompts is still less than the similarity threshold, new images are collected again, and this process is repeated until a new image with a similarity greater than or equal to the similarity threshold is collected.

[0116] In some embodiments, low-value image samples are identified as either discarded locally or compressed and uploaded.

[0117] This application embodiment calculates the similarity between the image features of a new image and the updated structured prompts. If the similarity is greater than or equal to a similarity threshold, a second image is obtained. If the similarity is less than the similarity threshold, the camera device continues to acquire new images until a new image with a similarity greater than or equal to the similarity threshold is acquired. The currently acquired new image is then used as the second image. This achieves image selection based on semantic understanding, enhances the acquisition of image samples that semantically match the updated structured prompts, prevents the acquisition of low-value image samples without mechanical capture, reduces the interference of irrelevant images on the model training process, and improves training efficiency.

[0118] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Furthermore, the data collection in the above embodiments is compliant, and its use or implementation does not involve any infringement upon public interests.

[0119] For ease of explanation, only the parts related to the embodiments of this application are shown in the methods corresponding to the above embodiments.

[0120] In some embodiments, Figure 3 This is a schematic diagram of the model training device provided in an embodiment of this application. The device includes: Image acquisition module 10 is used to control the camera device to acquire the first image during this round of model training; It is also used to control the camera device to acquire a second image if the recognition accuracy of the first recognition result is less than the accuracy threshold, and the image content of the second image matches the semantics of the updated structured prompt words.

[0121] The model training module 11 is used to use the image recognition model to recognize the first image based on the structured prompts and obtain the first recognition result. The structured prompts are the prompts obtained after the previous round of model training. It is also used to identify a second image based on the updated structured prompts using an image recognition model, and to obtain a second recognition result; It is also used to obtain a trained image recognition model if the training state of the image recognition model meets the preset training conditions.

[0122] The structured prompt word optimization module 12 is used to adjust the structured prompt words according to the first feedback result to obtain the updated structured prompt words. The first feedback result is the correct recognition result of the first image. It is also used to adjust the updated structured prompt words based on the second recognition result and the second feedback result, to obtain structured prompt words for the next round of model training, and then proceed to the step: In this round of model training, the camera device is controlled to acquire the first image, and the second feedback result is the correct recognition result of the second image.

[0123] In some embodiments, the image acquisition module includes a strategy generation unit and an image acquisition unit.

[0124] The strategy generation unit is used to determine the interference features in the first image, where the interference features are image features identified by the interference model. It is also used to determine image acquisition strategies based on interference features, first feedback results, and updated structured prompts.

[0125] The image acquisition unit is used to adjust the pose parameters and / or imaging parameters of the camera device according to the image acquisition strategy, so that the camera device can acquire a second image.

[0126] In some embodiments, the image acquisition unit is further configured to control the camera device to acquire new images; calculate the similarity between the image features of the new images and the updated structured prompts; if the similarity is greater than or equal to a similarity threshold, then obtain a second image; if the similarity is less than the similarity threshold, then continue to control the camera device to acquire new images until a new image with a similarity greater than or equal to the similarity threshold is acquired, and use the currently acquired new image as the second image.

[0127] In some embodiments, the structured prompt word optimization module is specifically used to determine semantic deviation information between the second feedback result and the updated structured prompt word; Specifically, it is used to adjust the structure of the updated structured prompts based on the second recognition result and semantic deviation information to obtain structured prompts for the next round of model training. The structure of the updated structured prompts includes role information, template information, and inference constraint information.

[0128] In some embodiments, the apparatus further includes a data preprocessing module and a semantic matching module.

[0129] The image acquisition module is also used to control the camera device to acquire the initial image.

[0130] The semantic matching module is used to perform semantic matching analysis between the initial image and the initial structured prompts to obtain the first semantic matching degree. The initial structured prompts are generated based on the natural language description input by the user.

[0131] The data preprocessing module is used to perform deduplication and extraction processing on the initial image to obtain the first image if the first semantic matching degree is greater than or equal to the matching degree threshold.

[0132] In some embodiments, the semantic matching module is further configured to perform semantic matching analysis between the first recognition result and the structured prompt words to obtain a second semantic matching degree; It is also used to determine the recognition accuracy based on the second semantic matching degree.

[0133] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0134] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0135] This application also provides a model training system, including a cloud server and a camera device, wherein the cloud server executes the steps described in the above method embodiments.

[0136] The camera device adjusts the image acquisition parameters according to the image acquisition strategy of the cloud server in order to acquire a second image.

[0137] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method embodiments.

[0138] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some cases, the computer-readable medium cannot be an electrical carrier signal or a telecommunication signal.

[0140] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0141] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0142] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0143] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0144] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A model training method, characterized in that, include: In this round of model training, the camera device is controlled to capture the first image; The first image is identified using an image recognition model based on structured prompts to obtain a first recognition result. The structured prompts are the prompts obtained after the previous round of model training. The structured prompts are adjusted based on the first feedback result to obtain updated structured prompts. The first feedback result is the correct recognition result of the first image. If the recognition accuracy of the first recognition result is less than the accuracy threshold, the camera device is controlled to acquire a second image, and the image content of the second image matches the semantics of the updated structured prompt words. The image recognition model is used to recognize the second image based on the updated structured prompts to obtain a second recognition result; The updated structured prompt words are adjusted according to the second recognition result and the second feedback result to obtain structured prompt words for the next round of model training, and then proceed to the following steps: In this round of model training, the camera device is controlled to acquire the first image, and the second feedback result is the correct recognition result of the second image; If the training state of the image recognition model meets the preset training conditions, then the trained image recognition model is obtained.

2. The method according to claim 1, characterized in that, The control of the camera device to acquire the second image includes: Determine the interference features in the first image, wherein the interference features are image features identified by the interference model; Based on the interference features, the first feedback result, and the updated structured prompt words, an image acquisition strategy is determined; According to the image acquisition strategy, the pose parameters and / or imaging parameters of the camera device are adjusted so that the camera device can acquire a second image.

3. The method according to claim 1 or 2, characterized in that, The control camera device acquires the second image, including: Control the camera device to acquire new images; Calculate the similarity between the image features of the new image and the updated structured prompts; If the similarity is greater than or equal to the similarity threshold, then the second image is obtained; If the similarity is less than the similarity threshold, the camera device continues to acquire new images until a new image with a similarity greater than or equal to the similarity threshold is acquired, and the currently acquired new image is used as the second image.

4. The method according to claim 3, characterized in that, The step of adjusting the updated structured prompts based on the second recognition result and the second feedback result to obtain structured prompts for the next round of model training includes: Determine the semantic deviation information between the second feedback result and the updated structured prompt words; Based on the second recognition result and the semantic deviation information, the structure of the updated structured prompt is adjusted to obtain the structured prompt for the next round of model training. The structure of the updated structured prompt includes role information, template information and inference constraint information.

5. The method according to claim 4, characterized in that, The role information includes task type, industry background and professional attributes; the template information includes the object to be detected, scene description, attribute features, recognition conditions and output format; and the inference constraint information includes viewpoint constraints, time constraints, lighting constraints, confidence constraints and output format constraints.

6. The method according to claim 5, characterized in that, Controlling the camera device to acquire the first image includes: Control the camera device to acquire the initial image; The initial image and the initial structured prompts are semantically matched and analyzed to obtain a first semantic matching degree. The initial structured prompts are generated based on the natural language description input by the user. If the first semantic matching degree is greater than or equal to the matching degree threshold, then the initial image is deduplicated and extracted to obtain the first image.

7. The method according to claim 6, characterized in that, If the recognition accuracy of the first recognition result is less than the accuracy threshold, then before controlling the camera device to acquire the second image, the following steps are also included: The first identification result is semantically matched with the structured prompt words to obtain a second semantic matching degree; The recognition accuracy is determined based on the second semantic matching degree.

8. A model training device, characterized in that, include: The image acquisition module is used to control the camera device to acquire the first image during this round of model training; It is also used to control the camera device to acquire a second image if the recognition accuracy of the first recognition result is less than the accuracy threshold, and the image content of the second image matches the semantics of the updated structured prompt words; The model training module is used to use an image recognition model to recognize the first image based on structured prompts and obtain a first recognition result. The structured prompts are the prompts obtained after the previous round of model training. It is also used to use the image recognition model to recognize the second image based on the updated structured prompts, and obtain a second recognition result; It is also used to obtain a trained image recognition model if the training state of the image recognition model meets the preset training conditions. The structured prompt word optimization module is used to adjust the structured prompt words according to the first feedback result to obtain the updated structured prompt words, wherein the first feedback result is the correct recognition of the first image; It is also used to adjust the updated structured prompt words according to the second recognition result and the second feedback result to obtain structured prompt words for the next round of model training, and proceed to the step: in this round of model training, control the camera device to acquire the first image, and the second feedback result is the correct recognition result of the second image.

9. A model training system, characterized in that, It includes a cloud server and a camera device, wherein the cloud server, when executed, implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as claimed in any one of claims 1 to 7.

Images