Behavior recognition model training methods, devices, equipment, storage media and products

Abstract

This application provides a method, apparatus, device, storage medium, and product for training a behavior recognition model. The method involves acquiring multiple category labels from an original behavior recognition dataset and combining each category label with predefined text to form input text. The predefined text describes the scenario in which the behavior occurs. Each input text is then input into a text-to-text model for text enhancement, generating text-to-image prompt text. This text-to-image prompt text is then concatenated with the corresponding input text to obtain an enhanced text set. Each text-to-image prompt text is input into the text-to-image model to generate an image, which is then fused with the original image from the original behavior recognition dataset to obtain an enhanced image set. The enhanced text set and the enhanced image set are used to train the behavior recognition model, resulting in a trained behavior recognition model that reduces overfitting and improves generalization performance.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a method, apparatus, device, storage medium and product for training behavior recognition models. Background Technology

[0002] Human behavior recognition aims to understand and determine the corresponding behavior category by analyzing human movements in images or video sequences. Human behavior recognition plays an irreplaceable role in various aspects of life. For example, in video communication, it can add real-time effects to users based on recognized actions; in the field of public safety, it can assist in the automated identification and early warning of specific behaviors, improving security efficiency.

[0003] Currently, human behavior recognition methods are mainly based on deep learning for predicting human behavior. For example, a convolutional neural network or Transformer model is trained on the source dataset, and the behavior of the human body is predicted by inputting a frame of the video during testing.

[0004] However, when training data is limited, the trained model may overfit, leading to a decrease in the model's generalization performance. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and product for training a behavior recognition model, which can reduce the overfitting degree of the model and improve the generalization performance of the model.

[0006] In a first aspect, embodiments of this application provide a method for training a behavior recognition model, the method comprising: Obtain multiple category labels from the original behavior recognition dataset, and combine each category label with predefined text to form the input text; the predefined text is used to describe the scenario in which the behavior occurs; Each input text is fed into the text-to-text model for text enhancement, generating text-to-image prompt text. The text-to-image prompt text and the corresponding input text are then concatenated to obtain the enhanced text set. Each text prompt in the text image is input into the text image large model to obtain the generated image. The generated image is then fused with the original image in the original behavior recognition dataset to obtain the enhanced image set. The behavior recognition model is trained using augmented text sets and augmented image sets to obtain the trained behavior recognition model.

[0007] In one feasible implementation, the method further includes: Obtain the bounding box coordinates used for cropping; the bounding box coordinates are determined based on random sampling. The cropped image is obtained by cropping the generated image using the bounding box coordinates; The enhanced image is obtained by replacing the region corresponding to the bounding box in the original image with the cropped image.

[0008] In one feasible implementation, the method further includes: The area of ​​the bounding box is calculated based on its coordinates. The blending ratio parameter is calculated based on the ratio of the bounding box area to the image area; where the image area is either the area of ​​the original image or the area of ​​the enhanced image. Based on the mixing ratio parameter, the enhanced text corresponding to the enhanced image and the category label corresponding to the original image are fused to obtain a soft label.

[0009] In one feasible implementation, the method further includes: Based on soft labels, calculate the cross-entropy loss of the behavior recognition model; The cross-entropy loss is used to determine whether the behavior recognition model meets the preset training stopping condition.

[0010] In one feasible implementation, the method further includes: Based on the text-to-image prompts, enhanced text and enhanced images are extracted from the enhanced text set and the enhanced image set, respectively, to obtain each pair of enhanced text and enhanced image; For each pair of enhanced text and enhanced image, perform the following steps: The enhanced text is input into the text feature extractor to obtain text features, and the enhanced image is input into the image feature extractor to obtain image features; Calculate the dot product of text features and image features to obtain the predicted value of human behavior; Based on the predicted human behavior values, determine the loss function value of the behavior recognition model; Determine whether the loss function value of the behavior recognition model meets the preset training stopping condition; If the conditions are not met, adjust the model parameters of the behavior recognition model and train the adjusted behavior recognition model using prompt text and enhanced images until the preset training stopping condition is met, thus obtaining the trained behavior recognition model.

[0011] In one feasible implementation, the method further includes: Acquire the image to be recognized; The behavior in the image to be identified is determined by inputting multiple category labels of the original behavior recognition dataset into the trained behavior recognition model.

[0012] Secondly, embodiments of this application provide a behavior recognition model training apparatus, the apparatus comprising: The acquisition module is used to acquire multiple category labels from the original behavior recognition dataset and combine each category label with predefined text to form the input text; the predefined text is used to describe the scenario in which the behavior occurs. The text enhancement module is used to input each input text into the text-to-text model, perform text enhancement, generate text-to-image prompt text, and concatenate the text-to-image prompt text with the corresponding input text to obtain an enhanced text set; The image enhancement module is used to input each text prompt text into the text prompt model to obtain the generated image, and then fuse the generated image with the original image in the original behavior recognition dataset to obtain the enhanced image set; The training module is used to train the behavior recognition model using augmented text sets and augmented image sets, resulting in the trained behavior recognition model.

[0013] Thirdly, embodiments of this application provide an electronic device, the device including: a processor, and a memory storing computer program instructions; A method for training a behavior recognition model by having the processor read and execute computer program instructions to achieve any one of the first aspects.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement a behavior recognition model training method as described in any of the first aspects.

[0015] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, cause the electronic device to perform a behavior recognition model training method as described in the first aspect.

[0016] The behavior recognition model training method, apparatus, device, storage medium, and product provided in this application can obtain multiple category labels from the original behavior recognition dataset, combine each category label with predefined text to form input text, and use a text-to-text model to perform text enhancement on the input text to obtain rich text-to-image prompt text, which can expand the diversity of the text semantic space. Then, the text-to-image prompt text is input into the text-to-image model to obtain generated images, which can expand the quantity and diversity of the original image dataset. Therefore, when the training data is limited, the amount of training data can be increased by performing text enhancement and image enhancement. Finally, when the behavior recognition model is trained using the enhanced text set and the enhanced image set, the overfitting phenomenon of the trained human behavior recognition model can be avoided, which leads to the problem of decreased generalization performance of the human behavior recognition model. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a behavior recognition model training method according to an embodiment of this application is shown; Figure 2 This illustration shows a schematic diagram of the text generation process for the text-based prompts provided in an embodiment of this application. Figure 3 A schematic diagram of the enhanced text generation process provided in an embodiment of this application is shown; Figure 4 A schematic diagram of the image generation process provided in an embodiment of this application is shown; Figure 5 A flowchart illustrating a behavior recognition model training method according to another embodiment of this application is shown; Figure 6 A schematic diagram of the enhanced image generation process provided in an embodiment of this application is shown; Figure 7 A flowchart illustrating a behavior recognition model training method according to another embodiment of this application is shown below; Figure 8 This illustration shows a schematic diagram of the behavior recognition model training process provided in an embodiment of this application; Figure 9 A schematic diagram of the structure of the behavior recognition model training device provided in an embodiment of this application is shown; Figure 10 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0020] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0021] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0022] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0023] First, let me explain the terms used in this application: Pre-trained large models: Pre-trained large models refer to various open-source models that have already been trained on datasets by other companies. Their parameters are obtained and then applied to one's own projects. Pre-trained large models typically contain a lot of semantic information, and different large models have different functions, such as text-to-text models, text-to-image models, and image recognition models.

[0024] Currently, existing technologies for human behavior recognition primarily involve training convolutional neural networks or Transformer models on source datasets and then using these trained models to predict human behavior in single or multiple frames of video sequences. While these deep learning methods offer improved accuracy compared to traditional methods, their performance largely depends on large-scale, high-quality labeled training datasets. However, in real-world applications, high-quality video data requires meticulous manual annotation, which is costly and time-consuming. Furthermore, certain behaviors (such as abnormal behavior or professional operations) occur infrequently, making it difficult to collect sufficient samples. Therefore, with limited training data, the trained model may overfit, leading to a decline in generalization performance.

[0025] To address the problems in existing technologies, embodiments of this application provide a method, apparatus, device, storage medium, and product for training a behavior recognition model. By expanding the diversity of the text semantic space using a text-to-text model, an enhanced text set is obtained. Similarly, by expanding the quantity and diversity of the original image dataset using a text-to-image model, an enhanced image set is obtained. This effectively avoids overfitting in the trained human behavior recognition model, even with limited training data, by increasing the number of training samples in the training set through expanding the number of text and images.

[0026] The following section first introduces a behavior recognition model training method provided in the embodiments of this application.

[0027] Figure 1 This illustration shows a flowchart of a behavior recognition model training method according to an embodiment of this application. Figure 1 As shown, the method may include the following steps: S101: Obtain multiple category labels from the original behavior recognition dataset and combine each category label with predefined text to form the input text.

[0028] In this embodiment, category labels define human behaviors, such as raising a hand or bending over. However, category labels are typically just keywords for a single behavior, which cannot provide sufficient contextual semantic information for the subsequent text-to-text model, causing the generated text and images to deviate from the actual application scenario. Therefore, by combining each category label with predefined text to form the input text, and by supplementing the environmental information of the behavior, the subsequently generated content is made more in line with actual needs.

[0029] In one example, the predefined text is a statement describing the scenario in which the behavior occurs, which may include information such as the identity of the subject of the behavior and the environment in which they are located.

[0030] In one example, the scene description text that matches each category label is selected as the predefined text. For example, for "raising hands," "students in a classroom" can be used as the predefined text, and for "bending over," "an elderly person in a park" can be used as the predefined text. Therefore, the resulting input texts are: "students raising hands in a classroom" and "an elderly person bending over in a park."

[0031] S102: Input each input text into the text-to-text model, perform text enhancement, generate text-to-image prompt text, and concatenate the text-to-image prompt text with the corresponding input text to obtain the enhanced text set.

[0032] In this embodiment, when augmenting the training set data, not only are images augmented, but each input text is also input into the text-to-text model for text augmentation, generating semantically richer text-to-image prompts. These text-to-image prompts are then concatenated with the corresponding input text to construct an augmented text set. This addresses the problem of scarce and semantically limited prompts in the field of human behavior recognition, providing rich and accurate text data support for subsequent text-to-image generation and behavior recognition model training.

[0033] In one example, the "Text-to-Text Big Model" is a pre-trained big model. It has been trained on massive amounts of text data, containing a wealth of semantic information and possessing powerful semantic expansion and logical extension capabilities. Therefore, by inputting the text into the "Text-to-Text Big Model," and leveraging its language understanding and generation capabilities, it can output text-based visual aids with the same semantics but richer expression and more specific context. The "Text-to-Text Big Model" can be a GPT-4 model.

[0034] In one example, when the input text is "A student raises his hand in the classroom," inputting the text into the Wensheng Wenda model results in the Wensheng graphic prompt text, which could be "A student raised one hand in a classroom full of people, and the teacher saw him." To expand the total amount of input text, the Wensheng graphic prompt text is concatenated with the input text to generate the enhanced text, "A student raises his hand in the classroom. A student raised one hand in a classroom full of people, and the teacher saw him." In another example, both the Wensheng graphic prompt text and the input text can be directly added to the enhanced text set simultaneously.

[0035] In one example, the same input text can be fed into the text-to-text model multiple times to obtain multiple text-to-image prompt texts. Among the multiple text-to-image prompt texts, poor quality or irrelevant descriptions can be removed, and finally multiple high-quality text-to-image prompt texts are obtained. Therefore, it not only expands the quantity of text data, but also improves the quality of text data, and can provide richer semantic information for subsequent image generation.

[0036] Figure 2 This illustration shows a schematic diagram of the text generation process for the graphic prompt provided in an embodiment of this application, such as... Figure 2 As shown, category label 21 and predefined text 22 are combined to obtain input text 23. Then, input text 23 is input into text-to-text model 24. Based on input text 23, text-to-text model 24 has powerful semantic expansion and logical extension capabilities. It can automatically output text-to-text prompt text 25 with the same semantics but richer expression and more specific scene, thus preparing for the subsequent generation of enhanced text and enhanced image.

[0037] Figure 3 A schematic diagram of the enhanced text generation process provided in an embodiment of this application is shown, such as... Figure 3 As shown, the input text 23 and the text-generated image prompt text 25 are concatenated to obtain the enhanced text 31. Then, the enhanced text is added to the enhanced text set 32, thereby expanding the total amount of input text and enriching the semantic diversity and scene coverage of the corpus during model training.

[0038] S103: Input each text prompt text into the text prompt model to obtain the generated image, and fuse the generated image with the original image in the original behavior recognition dataset to obtain the enhanced image set.

[0039] In this embodiment, the original behavior recognition dataset is limited in number due to privacy restrictions and other factors. Directly using it for model training can easily lead to overfitting. The text-generated image model can generate entirely new and realistic images based on the prompt text, supplementing the original image set. The generated images are then fused with the original images to enrich the semantic space of the resulting enhanced image.

[0040] In one example, the text-to-image large model is a pre-trained large model trained on massive amounts of text-image pairing data. It possesses accurate semantic-visual conversion capabilities and employs a diffusion model and encoder-decoder architecture to accurately capture key information such as behavioral cores, scene details, and character features in the text-to-image prompt text, thereby outputting an image that semantically matches the text-to-image prompt text. The text-to-image large model can be a DALLE 2.

[0041] In one example, the text prompt for the text image could be "A student raised one hand in a classroom full of people, and the teacher saw him." When this text prompt is input into the text image model, the model can generate an image that is exactly the same as the context of the sentence. This image does not exist in real life, but it is very realistic. Therefore, it can be added to the original dataset, which not only enriches the quantity of the original dataset, but also enriches the semantic space of the original dataset.

[0042] In one example, the same text prompt text can be input into the text-generated image model multiple times to obtain multiple generated images. Among the multiple generated images, samples with high image quality and accurate semantics can be retained, while blurry or inconsistent images can be removed, ultimately resulting in multiple high-quality generated images.

[0043] Figure 4 A schematic diagram of the image generation process provided in an embodiment of this application is shown, such as... Figure 4 As shown, the text prompt 25 is input into the text prompt model 41. Based on the text prompt model 41, an identical generated image 42 can be generated for the scene described by the text prompt 25, thereby expanding the number of images in the training set data.

[0044] S104: Train the behavior recognition model using the enhanced text set and the enhanced image set to obtain the trained behavior recognition model.

[0045] In this embodiment, the enhanced text set and the enhanced image set provide rich semantic and visual information, respectively. By using the enhanced text set and the enhanced image set to train the behavior recognition model, the phenomenon of overfitting of the behavior recognition model due to insufficient training dataset can be avoided.

[0046] In one example, after obtaining the trained behavior recognition model, the following is also included: Obtain the image to be recognized.

[0047] In this application embodiment, the image to be identified is the image for which the category of human behavior needs to be determined. In one example, a single frame image can be extracted from the video from which human behavior needs to be extracted for identification, such as: footage captured by a surveillance camera, screenshots of video calls, short video frames, etc.

[0048] The behavior in the image to be identified is determined by inputting multiple category labels of the original behavior recognition dataset into the trained behavior recognition model.

[0049] In this embodiment of the application, the image to be identified and multiple category labels are input into the trained behavior recognition model. The trained behavior recognition model calculates the matching degree between the image to be identified and each category label, and selects the category label with the highest matching degree as the recognition result, thereby determining the behavior in the image to be identified.

[0050] In this embodiment, the trained behavior recognition model is trained based on an enhanced text set and an enhanced image set. The enhanced text set and the enhanced image set contain rich semantic information and image visual information, which can improve the generalization ability of the trained behavior recognition model. Therefore, the trained behavior recognition model can accurately identify the human behavior category of the image to be recognized in various complex scenarios.

[0051] In this embodiment, multiple category labels from the original behavior recognition dataset are obtained, and each category label is combined with predefined text to form input text. The input text is then augmented using a text-to-text model to obtain rich text-to-image prompts, which expands the diversity of the text semantic space. The text-to-image prompts are then input into the text-to-image model to generate images, thus expanding the quantity and diversity of the original image dataset. Therefore, when training data is limited, text augmentation and image augmentation can increase the amount of training data. Finally, when training the behavior recognition model using augmented text and augmented image sets, overfitting of the trained human behavior recognition model can be avoided, which leads to a decrease in the generalization performance of the human behavior recognition model.

[0052] Figure 5 A flowchart illustrating a behavior recognition model training method according to another embodiment of this application is shown. Figure 5 As shown above, in the above Figure 1 Based on the illustrated embodiment, step S103 involves fusing the generated image with the original image from the original action recognition dataset to obtain an enhanced image, including: S501: Obtain the bounding box coordinates used for clipping. The bounding box coordinates are determined based on random sampling.

[0053] In this embodiment of the application, in order to enrich the features of the enhanced image and increase the diversity of the image dataset, the bounding box coordinates for cropping are determined by random sampling.

[0054] In one example, a uniform distribution can be used to determine the bounding box coordinates. The formula for determining the bounding box coordinates is: (1) (2) (3) (4) in, This indicates the starting x-coordinate of the top-left corner of the bounding box. This indicates the starting y-coordinate of the top-left corner of the bounding box. Indicates the height of the bounding box. Indicates the width of the bounding box. Indicates the width of the image. Indicates the height of the image. The mixing ratio parameter is obtained by sampling from a uniform distribution (0,1). It indicates a uniform distribution.

[0055] S502: Cropping is performed on the generated image using bounding box coordinates to obtain a cropped image.

[0056] In this embodiment, the bounding box coordinates obtained by random sampling are used to crop the image at the corresponding position in the generated image to obtain a cropped image, which provides the necessary image material for subsequent image fusion.

[0057] In one example, to further enhance the diversity of the augmented image set and improve the generalization ability of the action recognition model, multiple bounding boxes can be generated simultaneously. These bounding boxes are then used to crop the generated image, resulting in a cropped image. Specifically, based on multiple samplings with a uniform distribution, multiple bounding box coordinates with different locations and sizes, but all meeting the fusion ratio requirements, are generated. These bounding boxes are then applied to the same generated image, cropping it into multiple different image patches. Each image patch corresponds to a different local region in the generated image, thus providing richer options for subsequent image fusion. In another example, to avoid the randomly generated bounding box coordinates enclosing an area that is too small, resulting in insufficient effective information, or an area that is too large, rendering the cropping meaningless, the size range of the bounding box can be constrained. By setting minimum and maximum thresholds for the width and height of the bounding box, it can be ensured that the cropped image patch contains sufficient semantic information and effectively complements the original image.

[0058] S503: Replace the region corresponding to the bounding box in the original image with the cropped image to obtain the enhanced image.

[0059] In this embodiment, the region corresponding to the bounding box in the original image is replaced with a cropped image, thereby obtaining an enhanced image with richer features. The formula for calculating the enhanced image is as follows: (5) in, Indicates an enhanced image; This represents a binary mask whose size corresponds to the bounding box, indicating the locations to be deleted and filled from the two images; This represents a binary mask filled with 1s; This represents the Hadamard product, which is the product of corresponding elements. Represents the original image; This indicates the generation of an image.

[0060] In one example, to increase the diversity of the enhanced image, the bounding box region of the enhanced image can be replaced with the corresponding region in the original image.

[0061] In another example, to further increase the number of enhanced images, this application does not limit the combination of image fusion methods. Each original image can be fused with multiple different generated images to generate multiple enhanced images, maximizing the use of limited original data to increase the number of images when the amount of image data is too small; or, when the number of images is sufficient, original images can be selectively fused with semantically relevant generated images. For example, fusing an original image labeled "raising hands" with images generated from different textual prompts such as "students raising hands in the classroom" and "raising hands to speak at a meeting" can ensure that the enhanced images provide visual diversity while maintaining semantic consistency through a semantic association-based fusion strategy.

[0062] In another example, a fusion strategy can be selected based on the number of original images corresponding to a label. For example, if there are too few original images corresponding to a certain label, a one-to-many fusion method can be used first to fuse each original image in that category with multiple different generated images to maximize the enhancement effect. At the same time, the number of fusions for that category can be increased so that the model can be exposed to more enhanced samples of that category during training.

[0063] Secondly, since the generated image is generated from the text-based image prompt text, and the text-based image prompt text is generated from the input text, and the input text is generated from each category label, when each category label corresponds to a type of original image, the generated image can be fused with the corresponding original image.

[0064] Figure 6 A schematic diagram of the enhanced image generation process provided in an embodiment of this application is shown, such as... Figure 6 As shown, the generated image 42 and the original image 61 are input into the image enhancement model 62, which is the image enhancement step described above. The image enhancement model 62 is used to fuse the generated image 42 and the original image 61 to obtain the enhanced image 63. The enhanced image 63 is then added to the enhanced image set 64, thereby expanding the image data volume of the original image dataset and avoiding overfitting of the behavior recognition model caused by the scarcity of images during training.

[0065] In one example, after obtaining the enhanced image in step S503 above, the process further includes: The area of ​​the bounding box is calculated based on the bounding box coordinates.

[0066] In this embodiment, the bounding box area directly reflects the proportion of the cropped block of the generated image in the enhanced image, and is the basis for determining the degree of influence of the generated image features on the enhanced image. Therefore, it is necessary to determine the label based on the bounding box area.

[0067] The blending ratio parameter is calculated based on the ratio of the bounding box area to the image area; wherein the image area is the area of ​​the original image or the area of ​​the enhanced image.

[0068] In this embodiment, the mixing ratio parameter reflects the proportion of features from different sources in the enhanced image, so that the subsequently generated soft label can accurately match the feature distribution of the enhanced image. The mixing ratio parameter can be determined by calculating the proportion of the bounding box area to the area of ​​the original image or the area of ​​the enhanced image.

[0069] In another example, the blending ratio parameter can be determined directly. After determining the blending ratio parameter, the coordinates of the bounding box are determined according to the blending ratio parameter. The above formulas (3) and (4) of this application are the width and height of the bounding box determined after the blending ratio parameter is determined.

[0070] In one example, if a cropped image is used to replace the original image, the blending ratio parameter is the proportion of the area of ​​the cropped image to the area of ​​the original image.

[0071] Based on the mixing ratio parameter, the enhanced text corresponding to the enhanced image and the category label corresponding to the original image are fused to obtain a soft label.

[0072] In this embodiment, soft tags can fully combine the semantic information of the enhanced text and the category information of the original tags according to the mixing ratio parameter, which is more in line with the actual features of the enhanced image, thus improving the accuracy of model training. The calculation formula for soft tags is as follows: (6) in, Indicates a soft tag; Indicates the mixing ratio parameter; This indicates the category label corresponding to the original image; This represents the enhanced text corresponding to the enhanced image.

[0073] In this embodiment, a mixing ratio parameter is determined by the area of ​​the bounding box relative to the area of ​​the original image or the area of ​​the enhanced image. Based on the mixing ratio parameter, the enhanced text corresponding to the enhanced image and the category label corresponding to the original image are fused to obtain a soft label. This soft label can accurately reflect the weights of features from different sources in the enhanced image, allowing the model to be trained based on more realistic supervision information during the training process, thus effectively improving the model's recognition performance.

[0074] In this embodiment, compared to the traditional method of randomly deleting neurons or erasing random regions on the input to improve model generalization ability, which often results in the deleted regions being cleared or filled with random noise, reducing the proportion of effective information pixels in the training image, this embodiment does not simply delete or cover up image regions. Instead, it obtains the bounding box coordinates for cropping, uses these coordinates to crop the generated image, and replaces the regions corresponding to the bounding boxes in the original image with the cropped image to obtain the enhanced image. This enhanced image is then replaced with semantically relevant regions from the generated image. This not only preserves the pixel information of the original image but also enhances the semantic information contained in a single training image by introducing visual features from the generated image. Thus, it avoids the loss of effective information in the original image while preventing model overfitting.

[0075] Figure 7 The following is a flowchart illustrating a behavior recognition model training method according to another embodiment of this application. Figure 7 As shown above, in the above Figure 1 Based on the illustrated embodiment, one specific implementation of step S104 includes: S701: Based on the text-to-image prompt, extract the enhanced text and enhanced image from the enhanced text set and the enhanced image set respectively to obtain each pair of enhanced text and enhanced image.

[0076] In this embodiment, by using the text-generated image prompt text as an association identifier, enhanced text containing the prompt text is found in the enhanced text set, and enhanced images formed by fusing images generated from the prompt text are found in the enhanced image set, thereby forming multiple sets of paired training data. The semantics of each pair of enhanced text and enhanced image are consistent, avoiding the model encountering semantically contradictory data, which could lead to misunderstandings of behavioral features.

[0077] For each pair of enhanced text and enhanced image, perform the following steps: S702: Input the enhanced text into the text feature extractor to obtain text features, and input the enhanced image into the image feature extractor to obtain image features.

[0078] In this embodiment, by dividing the behavior recognition model into a text feature extractor and an image feature extractor, and extracting the text features corresponding to the enhanced text and the image features corresponding to the enhanced image respectively, more comprehensive behavior information can be obtained compared with single-modal feature extraction.

[0079] In one example, the text feature extractor can perform word segmentation and encoding on the enhanced text, outputting text features; the image feature extractor can perform convolution and pooling on the enhanced image, outputting image features.

[0080] S703: Calculate the dot product of text features and image features to obtain the predicted value of human behavior.

[0081] In this embodiment, the dot product operation measures the similarity between two feature vectors. By calculating the dot product of text features and image features (i.e., multiplying corresponding elements and then summing), the predicted human behavior value is obtained. The predicted human behavior value is a probability value that reflects the probability of matching behavior in the enhanced image with enhanced text. The higher the probability value, the greater the likelihood that the behavior in the enhanced image belongs to the enhanced text.

[0082] S704: Determine the loss function value of the behavior recognition model based on the predicted human behavior values.

[0083] In the embodiments of this application, the loss function value reflects the model prediction error. By determining the loss function value of the behavior recognition model through the human behavior prediction value, it can be determined whether training should stop.

[0084] In one example, when the enhanced image is obtained by fusing the generated image and the original image, the label of the enhanced image also becomes the fusion label, i.e., the soft label. Therefore, a specific implementation of step S704 is as follows: Based on the soft labels, the cross-entropy loss of the behavior recognition model is calculated.

[0085] In this embodiment, the soft label is an accurate supervisory signal corresponding to the enhanced image. By calculating the cross-entropy loss of the behavior recognition model using the soft label, the training error of the behavior recognition model can be accurately determined, providing a basis for adjusting the model parameters. The cross-entropy loss is an indicator used to measure the difference between the human behavior prediction value output by the behavior recognition model and the soft label. The smaller the loss value, the more accurate the model prediction; conversely, the larger the prediction error.

[0086] The behavior recognition model is determined based on the cross-entropy loss to determine whether it meets the preset training stopping condition.

[0087] In this embodiment of the application, the cross-entropy loss threshold can be set to determine whether the cross-entropy loss meets the threshold. When the cross-entropy loss meets the threshold, the training of the behavior recognition model can be stopped.

[0088] In another example, a maximum number of training iterations can be set simultaneously. If the cross-entropy loss does not meet the cross-entropy loss threshold, but the maximum number of training iterations is reached, the training of the behavior recognition model can also be stopped.

[0089] In this embodiment, when the enhanced image is obtained by fusing the generated image and the original image, in order to improve the accuracy of the behavior recognition model, the soft label corresponding to the enhanced image is used as supervision information. This can accurately reflect the actual contribution ratio of the original image and the generated image in the enhanced image, providing the model with supervision information that is completely corresponding to the training samples, thereby improving the accuracy of the behavior recognition model.

[0090] S705: Determine whether the loss function value of the behavior recognition model meets the preset training stopping condition.

[0091] In this embodiment, setting a training stopping condition can prevent overfitting due to model overtraining, while also preventing low recognition accuracy due to undertraining, ensuring that the model stops when the training effect is optimal, thus balancing model performance and training cost. The preset training stopping condition can be determined by setting a loss function threshold and the maximum number of training iterations.

[0092] In one example, hyperparameter optimization techniques can be used to dynamically adjust the loss function threshold or the maximum number of iterations based on the performance of the behavior recognition model on the validation set.

[0093] S706: If not satisfied, adjust the model parameters of the behavior recognition model, and train the adjusted behavior recognition model using prompt text and enhanced images until the preset training stop condition is met, thus obtaining the trained behavior recognition model.

[0094] In this embodiment, when the preset training stopping condition is not met, adjusting the parameters can continuously reduce the loss function value, allowing the model to gradually learn the behavioral features in the augmented data and improve recognition accuracy. In one example, the gradient of the parameters can be calculated based on the loss function value using the backpropagation algorithm. The model parameters of the behavior recognition model can be adjusted according to the gradient descent direction, and the behavior recognition model can continue to be trained after the adjustment is completed until the preset training stopping condition is met, thus obtaining the trained behavior recognition model.

[0095] In this embodiment, the behavior recognition model includes a text feature extractor and an image feature extractor, which can extract text features and image features simultaneously. By fusing bimodal semantic information, the behavior recognition model can learn more comprehensive behavior features, so that the finally trained behavior recognition model can have accurate recognition accuracy and generalization ability.

[0096] Figure 8The diagram illustrates the training process of the behavior recognition model provided in this application. The behavior recognition model provided in this application can be a SLIP (Self-supervision meets Language-Image Pre-training) model. This model forms a richer semantic space through the interaction of image and text modalities. Furthermore, by combining language supervision and image self-supervision, it can learn high-quality visual representations without category labels, providing a foundation for accurate classification of human behavior. The SLIP model is also a pre-trained large model, therefore only full fine-tuning on the training set data is needed, reducing computational resources and costs compared to retraining the model. Figure 8 As shown, the enhanced text 31 is input into the text feature extractor 81, and the text feature extractor 81 extracts features to obtain text features 82. The enhanced image 63 is input into the image feature extractor 83, and the image feature extractor 83 extracts features to obtain image features 84. Then, the image features 84 and text features 82 are multiplied by a dot product to obtain the human behavior prediction value 85. Finally, the cross-entropy loss function value 86 is calculated based on the human behavior prediction value 85, and the cross-entropy loss function value 86 is used to determine whether the behavior recognition model should stop training and adjust the parameters. Through the deep fusion of dual-modal features at the semantic level, the feature learning ability and prediction accuracy of the behavior recognition model are effectively improved.

[0097] Based on the behavior recognition model training method provided in the above embodiments, this application also provides a specific implementation of a behavior recognition model training device. Please refer to the following embodiments.

[0098] First see Figure 9 , Figure 9 The diagram shows a schematic of the structure of the behavior recognition model training device provided in the embodiment of this application. The behavior recognition model training device 900 provided in the embodiment of this application includes: an acquisition module 901, a text enhancement module 902, an image enhancement module 903, and a training module 904.

[0099] The acquisition module 901 is used to acquire multiple category labels from the original behavior recognition dataset and combine each category label with predefined text to form the input text; the predefined text is used to describe the scenario in which the behavior occurs. The text enhancement module 902 is used to input each input text into the text-to-text model, perform text enhancement, generate text-to-image prompt text, and concatenate the text-to-image prompt text with the corresponding input text to obtain an enhanced text set; The image enhancement module 903 is used to input each text prompt text into the text prompt model to obtain the generated image, and to fuse the generated image with the original image in the original behavior recognition dataset to obtain the enhanced image set; Training module 904 is used to train the behavior recognition model using an augmented text set and an augmented image set to obtain the trained behavior recognition model.

[0100] In one example, image enhancement module 903 includes: The `get` submodule is used to obtain the bounding box coordinates for cropping. The bounding box coordinates are determined based on random sampling. The cropping submodule is used to crop the generated image using bounding box coordinates to obtain a cropped image. The replacement submodule is used to replace the region corresponding to the bounding box in the original image with the cropped image to obtain the enhanced image.

[0101] In one example, the behavior recognition model training device 900 further includes: The first calculation module is used to calculate the area of ​​the bounding box based on the bounding box coordinates; The second calculation module is used to calculate the blending ratio parameter based on the ratio of the bounding box area to the image area; wherein the image area is the area of ​​the original image or the area of ​​the enhanced image. The fusion module is used to fuse the enhanced text corresponding to the enhanced image and the category label corresponding to the original image based on the mixing ratio parameter to obtain a soft label.

[0102] In one example, training module 904 includes: The computation submodule is used to calculate the cross-entropy loss of the behavior recognition model based on soft labels; The judgment submodule is used to determine whether the behavior recognition model meets the preset training stopping conditions based on cross-entropy loss.

[0103] In one example, training module 904 includes: The extraction submodule is used to extract enhanced text and enhanced images from the enhanced text set and enhanced image set respectively, based on the text-image prompt text, to obtain each pair of enhanced text and enhanced image; For each pair of enhanced text and enhanced image, perform the following steps: The feature extraction submodule is used to input enhanced text into the text feature extractor to obtain text features, and to input enhanced image into the image feature extractor to obtain image features; The calculation submodule is used to calculate the dot product of text features and image features to obtain the human behavior prediction value; The determination submodule is used to determine the loss function value of the behavior recognition model based on the predicted human behavior values; The judgment submodule is used to determine whether the loss function value of the behavior recognition model meets the preset training stopping condition; The adjustment submodule is used to adjust the model parameters of the behavior recognition model when the conditions are not met, and to train the adjusted behavior recognition model using prompt text and enhanced images until the preset training stop condition is met, thus obtaining the trained behavior recognition model.

[0104] In one example, the behavior recognition model training device 900 further includes: The acquisition module is used to acquire the image to be recognized; The input module is used to input the image to be recognized and multiple category labels of the original behavior recognition dataset into the trained behavior recognition model to determine the behavior in the image to be recognized.

[0105] The various modules of the behavior recognition model training device provided in this application embodiment can achieve Figure 1 The functions of each step in the provided behavior recognition model training method, and the corresponding technical effects they achieve, will not be elaborated here for the sake of brevity.

[0106] Figure 10 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0107] An electronic device may include a processor 1001 and a memory 802 storing computer program instructions.

[0108] Specifically, the processor 1001 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0109] Memory 1002 may include mass storage for data or instructions. For example, and not limitingly, memory 1002 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 1002 may include removable or non-removable (or fixed) media, or memory 1002 may be a non-volatile solid-state memory.

[0110] In one instance, memory 1002 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0111] Memory 1002 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0112] The processor 1001 reads and executes computer program instructions stored in the memory 1002 to implement a behavior recognition model training method in the above embodiment.

[0113] In one example, the electronic device may also include a communication interface 1003 and a bus 1004. For example, Figure 10 As shown, the processor 1001, memory 1002, and communication interface 1003 are connected through bus 1004 and complete communication with each other.

[0114] The communication interface 1003 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0115] Bus 1004 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1004 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0116] The behavior recognition model training method described in the above embodiments can be implemented using a computer storage medium. This computer storage medium stores computer program instructions; when these instructions are executed by a processor, they implement any of the behavior recognition model training methods described in the above embodiments.

[0117] This application also provides a computer program product, including a computer program, which, when executed, implements any of the behavior recognition model training methods described in the above embodiments.

[0118] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0119] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0120] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for training a behavior recognition model, characterized in that, include: Obtain multiple category labels from the original behavior recognition dataset, and combine each category label with predefined text to form the input text; The predefined text is used to describe the scenario in which the behavior occurs; Each input text is input into the text-to-text model for text enhancement, generating text-to-image prompt text. The text-to-image prompt text and the corresponding input text are then concatenated to obtain an enhanced text set. Each of the text-based image prompts is input into the text-based image large model to obtain a generated image. The generated image is then fused with the original image in the original behavior recognition dataset to obtain an enhanced image set. The step of fusing the generated image with the original image in the original behavior recognition dataset to obtain an enhanced image set includes: Obtain the bounding box coordinates used for cropping, the bounding box coordinates being determined based on random sampling; The generated image is cropped using the bounding box coordinates to obtain a cropped image. The region corresponding to the bounding box in the original image is replaced with the cropped image to obtain the enhanced image set; The area of ​​the bounding box is calculated based on the bounding box coordinates. The blending ratio parameter is calculated based on the ratio of the bounding box area to the image area; wherein the image area is the area of ​​the original image or the area of ​​the enhanced image. Based on the aforementioned mixing ratio parameter, the enhanced text corresponding to the enhanced image and the category label corresponding to the original image are fused to obtain a soft label; the calculation formula for the soft label is: in, Indicates a soft tag; Indicates the mixing ratio parameter; This indicates the category label corresponding to the original image; This refers to the enhanced text corresponding to the enhanced image; The behavior recognition model is trained using the enhanced text set and the enhanced image set to obtain the trained behavior recognition model; The step of training the behavior recognition model using the enhanced text set and the enhanced image set includes: Based on the soft labels, calculate the cross-entropy loss of the behavior recognition model; The behavior recognition model is determined based on the cross-entropy loss to determine whether it meets the preset training stopping condition.

2. The method according to claim 1, characterized in that, The behavior recognition model includes: a text feature extractor and an image feature extractor; The step of training the behavior recognition model using the enhanced text set and the enhanced image set includes: Based on the text-based image prompts, enhanced text and enhanced images are extracted from the enhanced text set and the enhanced image set, respectively, to obtain each pair of enhanced text and enhanced images; For each pair of enhanced text and enhanced image, perform the following steps: The enhanced text is input into the text feature extractor to obtain text features, and the enhanced image is input into the image feature extractor to obtain image features; Calculate the dot product of the text features and the image features to obtain the human behavior prediction value; Based on the predicted human behavior values, determine the loss function value of the behavior recognition model; Determine whether the loss function value of the behavior recognition model meets the preset training stopping condition; If the conditions are not met, the model parameters of the behavior recognition model are adjusted, and the adjusted behavior recognition model is trained using the prompt text and the enhanced image until the preset training stop condition is met, thus obtaining the trained behavior recognition model.

3. The method according to claim 1, characterized in that, After training the behavior recognition model using the enhanced text set and the enhanced image set, the method further includes: Acquire the image to be recognized; The image to be identified and multiple category labels of the original behavior recognition dataset are input into the trained behavior recognition model to determine the behavior in the image to be identified.

4. A behavior recognition model training device, characterized in that, The device includes: The acquisition module is used to acquire multiple category labels from the original behavior recognition dataset and combine each category label with predefined text to form input text; the predefined text is used to describe the scenario in which the behavior occurs. The text enhancement module is used to input each of the input texts into the text-to-text model, perform text enhancement, generate text-to-image prompt text, and concatenate the text-to-image prompt text with the corresponding input text to obtain an enhanced text set; The image enhancement module is used to input each of the text-based image prompts into the text-based image large model to obtain a generated image, and to fuse the generated image with the original image in the original behavior recognition dataset to obtain an enhanced image set; The image enhancement module is specifically used for: Obtain the bounding box coordinates used for cropping, the bounding box coordinates being determined based on random sampling; The generated image is cropped using the bounding box coordinates to obtain a cropped image. The region corresponding to the bounding box in the original image is replaced with the cropped image to obtain the enhanced image set; The first calculation module is used to calculate the area of ​​the bounding box based on the bounding box coordinates; The second calculation module is used to calculate the blending ratio parameter based on the ratio of the bounding box area to the image area; wherein the image area is the area of ​​the original image or the area of ​​the enhanced image; The fusion module is used to fuse the enhanced text corresponding to the enhanced image and the category label corresponding to the original image based on the fusion ratio parameter to obtain a soft label; the calculation formula for the soft label is: in, Indicates a soft tag; Indicates the mixing ratio parameter; This indicates the category label corresponding to the original image; The enhanced text represents the enhanced image; the training module is used to train the behavior recognition model using the enhanced text set and the enhanced image set to obtain the trained behavior recognition model. The training module is specifically used for: Based on the soft labels, calculate the cross-entropy loss of the behavior recognition model; The behavior recognition model is determined based on the cross-entropy loss to determine whether it meets the preset training stopping condition.

5. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement a behavior recognition model training method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement a behavior recognition model training method as described in any one of claims 1-3.

7. A computer program product, characterized in that, When the instructions in a computer program product are executed by the processor of an electronic device, the electronic device performs a behavior recognition model training method as described in any one of claims 1-3.

Images