Scene understanding information generation method and device, equipment and medium

Abstract

The invention discloses a scene understanding information generation method and device, equipment and a medium, and the method comprises the steps: inputting an image into a scene classification model, and obtaining a scene type corresponding to the image outputted by the scene classification model; identifying a target object in the image by using a target detection model to obtain target object information, the target object information including a target type and position information, and the target type including an object and a person; identifying target attribute information of personnel in the image based on a personnel attribute identification model; taking the scene type, the target object information and the target attribute information as context information, and determining model input information based on the context information and a user input text; and inputting the model input information into the natural language processing model to enable the natural language processing model to generate scene understanding information corresponding to the image based on the prompt of the context information. Therefore, the scene perception capability is improved, and more accurate scene understanding information is generated, so that the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a method, device, equipment and medium for generating scene understanding information. Background Art

[0002] With the development of artificial intelligence technology, extracting and analyzing information from images and then applying it to industry management and operations to improve operational efficiency and service quality has become an important technical means.

[0003] At present, it is difficult to obtain richer information from images by relying on image recognition or target detection models. The perception ability of ever-changing scenes is weak and it is impossible to provide users with better scene understanding information, thus affecting the user experience. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a scene understanding information generation method, device, equipment and medium, which can improve the perception of the scene, generate more accurate scene understanding information, and thus improve the user experience. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a method for generating scene understanding information, comprising:

[0006] Inputting an image into a scene classification model to obtain a scene type corresponding to the image output by the scene classification model;

[0007] Using a target detection model to identify a target object in the image, and obtaining target object information, wherein the target object information includes target type and location information, and the target type includes objects and people;

[0008] Identify target attribute information of a person in the image based on a person attribute recognition model;

[0009] Taking the scene type, the target object information, and the target attribute information as context information, and determining model input information based on the context information and user input text;

[0010] The model input information is input into a natural language processing model so that the natural language processing model generates scene understanding information corresponding to the image based on the prompt of the context information.

[0011] Optionally, the identifying target attribute information of a person in the image based on a person attribute recognition model includes:

[0012] Based on the target object information, extracting an image region containing a person from the image;

[0013] The target attribute information of the person in the image area is identified using a person attribute recognition model.

[0014] Optionally, the extracting an image region containing a person from the image based on the target object information includes:

[0015] Determine the interception range based on the location information of the personnel;

[0016] An image region containing a person is intercepted from the image based on the interception range, wherein the image region includes a person region and a partial background region.

[0017] Optionally, the identifying target attribute information of a person in the image region by using a person attribute recognition model includes:

[0018] Allocating a weighted weight to the image region based on the position information of the person, wherein the weighted weight of the person region is higher than the weighted weight of the partial background region;

[0019] weighting the image region using the weighted weights to obtain a weighted image;

[0020] The weighted image is input into the personnel attribute recognition model to obtain target attribute information of the personnel in the weighted image.

[0021] Optionally, the target detection model is trained based on an image training sample set, the label information of the image training sample set includes the target type and location information of the target object, and during the training process, the training loss is calculated by weighting the category loss based on the weighted value corresponding to the specified target object.

[0022] Optionally, the identifying target attribute information of a person in the image based on a person attribute recognition model includes:

[0023] Extracting image features in the image using a convolution module in the personnel attribute recognition model, and extracting key point features in the image using a key point detection module in the personnel attribute recognition model;

[0024] Target attribute information of the person in the image is determined based on the image features and the key point features.

[0025] Optionally, the identifying the target object in the image by using the target detection model to obtain target object information includes:

[0026] The target detection model is used to identify type information of the target object in the image, and the confidence corresponding to the type information is adjusted based on the scene type, and the target type of the target object is determined based on the confidence.

[0027] It can be seen from the above scheme that the present invention provides a method for generating scene understanding information, including: inputting an image into a scene classification model to obtain a scene type corresponding to the image output by the scene classification model; using a target detection model to identify a target object in the image to obtain target object information, wherein the target object information includes a target type and location information, and the target type includes objects and persons; identifying target attribute information of persons in the image based on a person attribute recognition model; using the scene type, the target object information, and the target attribute information as context information, and determining model input information based on the context information and user input text; inputting the model input information into a natural language processing model, so that the natural language processing model generates scene understanding information corresponding to the image based on the prompt of the context information.

[0028] It can be seen that the beneficial effects of the present application are: performing scene classification, target detection and person attribute recognition on images, obtaining scene type, target object information and person attribute information as context information. The context information contains rich image information, which can prompt the natural language processing model to understand the scene of the image more accurately, improve the ability to perceive the scene, generate more accurate scene understanding information, and thus improve the user experience.

[0029] Correspondingly, a scene understanding information generating device, equipment and readable storage medium provided by the present application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0031] Figure 1 A flow chart of a method for generating scene understanding information provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of the structure of a scene understanding information generating device provided in an embodiment of the present application;

[0033] Figure 3 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] First, the terms involved in the embodiments of the present application are explained:

[0036] Object detection model: refers to a model that automatically detects and locates objects in images or videos through computer vision technology.

[0037] YOLOv8 (You Only Look Once version 8) is one of the YOLO series of target detection algorithms. YOLO is a real-time target detection system that treats target detection as a regression problem and directly predicts the object bounding box and category from the full image without the need for an additional candidate box generation step. YOLOv8 has significant improvements in accuracy and speed compared to earlier versions.

[0038] ImageNet (Image Recognition Database) is a large visual recognition database containing more than 14 million manually annotated images covering more than 20,000 categories. ImageNet is often used to train and evaluate computer vision models, especially deep learning models.

[0039] Transformer is a model architecture for sequence-to-sequence learning. The Transformer model abandons the traditional sequence processing methods of recurrent neural networks and convolutional neural networks, and completely relies on the self-attention mechanism to process the input sequence, thereby greatly improving the training speed and parallelism. The Transformer architecture has become the cornerstone of natural language processing and computer vision.

[0040] ViT (Vision Transformer) is an image processing model based on the Transformer architecture. Unlike traditional convolutional neural networks, ViT divides the input image into fixed-size image blocks and treats them as sequence data, processing them through the self-attention mechanism. ViT has demonstrated excellent performance in a variety of image recognition tasks, especially for processing large-resolution images.

[0041] NLP (Natural Language Processing) is a branch of artificial intelligence that focuses on enabling computers to understand, interpret, and generate human language. NLP technology covers multiple fields such as speech recognition, text classification, sentiment analysis, machine translation, and question-answering systems.

[0042] BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model based on the Transformer architecture. Through bidirectional training, BERT can understand the contextual relationship of words in a sentence and generate deep semantic representations of words. This model performs well on a variety of NLP tasks, including question answering, named entity recognition, and text classification.

[0043] EfficientNet is an efficient and high-precision convolutional neural network architecture. EfficientNet uses compound scaling rules to simultaneously adjust the width, depth, and resolution of the network to achieve a good balance between computing resources and model performance. EfficientNet has achieved leading levels in multiple image recognition benchmarks while maintaining low computational complexity.

[0044] Clustering algorithm: It is a type of unsupervised learning method used to divide samples in a data set into different groups (or clusters). The goal of a clustering algorithm is to classify similar data points into the same group and dissimilar data points into different groups. Common clustering algorithms include K-Means (i.e., K-means algorithm), DBSCAN (i.e., Density-Based Spatial Clustering of Applications with Noise), hierarchical clustering, etc.

[0045] Anchor box: Anchor box is a mechanism used to predict the location of objects in target detection tasks. In target detection algorithms, such as YOLO, SSD (Single Shot MultiBox Detector), etc., multiple predefined rectangular boxes (i.e. anchor boxes) are placed at different locations in the image, and these anchor boxes are used to predict the location and category of objects. Anchor boxes are designed to cover objects of different sizes and shapes, so the size of the anchor box should match the detected object.

[0046] OpenPose (Open Source Pose Estimation): OpenPose is an open source human pose estimation algorithm that can detect key points of the human body in an image or video and estimate the human body's pose. OpenPose is commonly used in the field of computer vision and can estimate the pose of a single person or multiple people in real time.

[0047] See also Figure 1 As shown, the embodiment of the present application discloses a method for generating scene understanding information, including:

[0048] Step S11: inputting the image into a scene classification model to obtain the scene type corresponding to the image output by the scene classification model.

[0049] The scene understanding information generation method provided in this embodiment can be deployed on a back-end server or a terminal device. The back-end server can be a cloud server or a physical server, and the terminal device can include but is not limited to monitoring equipment, personal computers, tablet computers, smart phones, etc.

[0050] The scene type is the type of the background environment. For example, in the retail industry, the scene types may include the front hall, kitchen, shopping mall, supermarket, cashier counter, warehouse, street, etc.

[0051] Furthermore, the scene classification model can be understood as a deep network model for scene classification. In this embodiment, the preset model can be trained using an image training sample set to obtain a scene classification model. The image training sample set can include image training samples and scene type labels corresponding to the image training samples. In addition, the image training samples are images obtained by data enhancement of the original images. The data enhancement method may include one or more of random cropping, flipping, color jittering, etc. The original image may be an image collected for the corresponding industry. For example, in the retail industry, original images of scenes including the lobby, kitchen, shopping mall, supermarket, cashier, warehouse, street, etc. are collected. The preset model is a deep network model.

[0052] Furthermore, in an optional implementation, the initial classification model can be trained using a public image training set to obtain a preset model, and then the preset model can be trained using an image training sample set to obtain a better scene classification effect. The public image training set is a large image training set, that is, a training set with a sample number greater than a specified number. For example, the initial classification model can use ViT, pre-train on large datasets such as ImageNet, and then fine-tune on a retail scene dataset. In some other embodiments, the initial classification model can use other classification models.

[0053] It should be pointed out that this embodiment uses ViT as the basic framework, which has extremely high performance when processing image classification tasks, and is particularly suitable for training on large-scale data sets. ViT can capture global context information and can better understand complex scenes. Pre-training on large data sets such as ImageNet and then fine-tuning on retail scene data sets can obtain better scene classification effects. In addition, a variety of image enhancement techniques such as random cropping, flipping, and color jittering are applied to increase the generalization ability of the model. The scene classification model obtained by such training can recognize and classify specific types of scenes, such as lobby, kitchen, shopping mall, supermarket, cash register, warehouse, street, etc.

[0054] Step S12: using the target detection model to identify the target object in the image and obtain target object information, wherein the target object information includes target type and location information, and the target type includes objects and people.

[0055] Wherein, the target detection model is obtained by training based on an image training sample set, and the label information of the image training sample set includes the target type and location information of the target object. In this embodiment, the image training sample set can be used to train the initial detection model to obtain the target detection model. The image training samples in the image training sample set can be consistent or inconsistent with the scene classification model. For example, YOLOv8 can be used as the initial detection model, which can achieve a good balance between real-time and accuracy, and is very suitable for processing a large number of video streams in a retail environment. Of course, in some other embodiments, other detection models can also be selected as the initial detection model.

[0056] In this embodiment, the anchor frame can be customized, the size of the target object to be identified can be collected, clustering can be performed based on the size, and the anchor frame size can be determined based on the clustering result. For example, the annotation data of objects in the retail environment are collected, including the location and size of the objects. Then, these annotation data are statistically analyzed to find out the distribution pattern of the object size. Then, a clustering algorithm (such as K-means) is used to cluster the object size, and a representative size is obtained as the new anchor frame size for subsequent object detection tasks. In this way, according to the size distribution of retail items, the anchor frame is recalculated and set, which can improve the detection accuracy of small objects.

[0057] Furthermore, in this embodiment, during the training process, the training loss can be calculated by weighting the class loss based on the weighted value corresponding to the specified target object. The specified target object can be specified according to different environments. For example, in a retail environment, the weighted value of the target object commonly seen in the retail environment can be increased to make the model more sensitive to the specified target object.

[0058] In an optional implementation, the training loss includes a category loss and a bounding box loss, and the category loss may be a cross entropy loss. It is understood that the present embodiment may update the model parameters based on the training loss until the training end condition is met, for example, the training loss is less than a preset threshold, and the training end condition is met.

[0059] For example, you can set a training loss function: adjust the loss function weights so that the model is more sensitive to common commodity categories in retail environments. Training loss functions include cross entropy loss function and bounding box regression loss function. In order to make the model more sensitive to certain categories (such as related commodities in the retail industry), this can be achieved by adjusting the weights in the loss function. For common commodity categories in retail environments, the weights of these commodity categories in the loss function can be increased. During the training process, for each predicted object, the cross entropy loss between its category prediction and the true label is calculated, and weighted according to the category weight; at the same time, the regression loss between the predicted bounding box and the true bounding box is calculated. The weighted category loss and bounding box loss are combined into the total training loss. In summary, the total training loss is calculated as:

[0060] .

[0061] Among them, L represents the total training loss, C represents the total number of categories, i represents the i-th category, that is, one of the categories, and w i Represents the weight of category i, which is used to adjust the relative importance of this category in the loss function. i Represents the value of the i-th category in the true label vector, usually 0 or 1. i represents the value of the i-th category in the predicted probability vector. λ is a hyperparameter that balances the classification loss and the bounding box regression loss. b p,j denotes the coordinates of the predicted bounding box j (top, left, bottom, right). b g,j L1 represents the coordinates of the ground-truth bounding box j (top, left, bottom, right). smooth Represents a smooth L1 loss function that calculates the difference between the predicted bounding box and the true bounding box. Represents the set of all predicted rectangular boxes (top, left, bottom, right), where top, left, bottom, and right refer to pixel coordinates, that is, this embodiment calculates the loss based on the pixel coordinates of the upper left corner and the lower right corner of the predicted rectangular box, where the top and left constitute the pixel coordinates of the upper left corner, and the bottom and right constitute the pixel coordinates of the lower right corner. L is the loss of an object.

[0062] In order to make the model more sensitive to specific categories (such as common commodity categories in retail environments), this embodiment can adjust the weight w in the loss function iSpecifically, you can determine the sensitive categories: determine the categories that you want the model to be more sensitive to, such as retail categories such as beverages and food. Then set the weights: for sensitive categories ci, set a higher weight w i . For example, for beverage category, it can be set to 2, while other categories keep the default value of 1. Calculate classification loss: For each predicted target, calculate the cross entropy loss between its class prediction and the true label, and weight it according to the class weight.

[0063] .

[0064] Among them, L cls,i Denotes the cross entropy loss of the i-th category of any target. Increasing the weight makes the classification errors of sensitive categories have a greater impact on the total loss L, thus prompting the model to pay more attention to these categories during training.

[0065] The bounding box regression loss is calculated using a smooth L1 loss function for the difference between the predicted bounding box and the true bounding box.

[0066] .

[0067] The weighted classification loss and bounding box regression loss are combined into the total loss function L. When the weight w of the sensitive category i When λ increases, the contribution of sensitive categories in the total loss will also increase accordingly. This means that during training, the model will work harder to reduce the classification errors of these categories, thereby improving the recognition accuracy of these categories. The parameter λ is used to balance the relationship between the classification loss and the bounding box regression loss. By adjusting λ, it is possible to maintain good bounding box positioning accuracy while ensuring that the model is more sensitive to specific categories. By adjusting the weights in the loss function in this way, the model can be guided to pay more attention to specific product categories during training, thereby achieving better performance in object detection tasks in retail environments.

[0068] Furthermore, in an optional implementation, the target detection model can be used to identify the type information of the target object in the image, and the confidence corresponding to the type information can be adjusted based on the scene type, and the target type of the target object can be determined based on the confidence. For example, in a back kitchen scene, a certain object does not usually appear, so the confidence corresponding to the object is lowered, thereby filtering out the object. The confidence of objects with a high correlation with the scene type can also be increased. That is, the present embodiment can adjust the confidence based on the correlation with the scene type. The confidence can be understood as the degree of certainty of the model in its prediction results.

[0069] Step S13: Identify target attribute information of the person in the image based on the person attribute recognition model.

[0070] Among them, the personnel attribute recognition model can be understood as a deep learning model used to identify personnel attribute information. The target attribute information includes but is not limited to one or more of gender, age, clothing style, carried items, and posture. The personnel attribute recognition model can provide richer personnel attribute information.

[0071] Furthermore, in this embodiment, the convolution module in the personnel attribute recognition model can be used to extract image features in the image, and the key point detection module in the personnel attribute recognition model can be used to extract key point features in the image; and the target attribute information of the person in the image can be determined based on the image features and the key point features. That is, in this embodiment, a key point detection module is added and introduced to help the model locate human body parts more accurately, thereby improving the accuracy of attribute recognition.

[0072] In an optional implementation, an image region containing a person may be intercepted from the image based on the target object information; and the target attribute information of the person in the image region may be identified using a person attribute recognition model. That is, an image region containing a person may be intercepted from the image based on the position information of the person. In this way, intercepting the image region containing a person and identifying it can eliminate much background interference.

[0073] In a specific implementation, the interception range can be determined based on the location information of the person; based on the interception range, the image area containing the person is intercepted from the image, wherein the image area includes the person area and part of the background area. A weighted weight is assigned to the image area based on the location information of the person, wherein the weighted weight of the person area is higher than the weighted weight of the part of the background area; the image area is weighted using the weighted weight to obtain a weighted image; the weighted image is input into the person attribute recognition model to obtain the target attribute information of the person in the weighted image. For example, the target detection model detects the person and outputs the corresponding rectangular frame, and outputs the image portion containing only the person (the rectangular frame is expanded outward by 1.5 times and cut out from the original image) to the person attribute recognition model.

[0074] This embodiment can capture an image area including part of the background area, which can provide more abundant information for the personnel attribute recognition model, thereby improving the accuracy of personnel attribute recognition, such as the items carried.

[0075] Among them, the weighted weight is the weight calculated based on the attention mechanism, that is, the attention weight, and the target detection model is used to obtain multiple bounding boxes in the image {B i}, each bounding box can be represented by four coordinates: x1, y1, x2, y2, which represent the coordinates of the upper left corner and the lower right corner of the bounding box respectively, where i is the index of the person. In order to calculate the attention weight of each pixel, a weight function w(p) can be defined, where p=(x, y) represents the pixel coordinates in the image. The purpose of the weight function is to assign a weight value to each pixel in the image, so that the area where the person is located gets a higher weight, while the background area gets a lower weight. Attention weight calculation:

[0076] .

[0077] in, , c i represents the coordinates of the center of the i-th bounding box, , , σ is a constant used to control the concentration of attention, represents the Euclidean distance between pixel p and the center of the person's bounding box. In this way, the attention weight w(p) of each pixel can be calculated. This weight function assigns higher weights to pixels close to the center of the person and lower weights to pixels far from the center of the person.

[0078] Furthermore, the image region can be weighted by applying the attention weight w(p) to the original image (i.e., image region) I to obtain the weighted image Iw: Iw(p)=w(p)*I(p). The weighted image Iw is passed as input to the person attribute recognition model.

[0079] Among them, the personnel attribute recognition model can be obtained by training based on the training sample set. The image samples in the training sample set can be obtained by using the target detection model to perform target detection on the image training sample set and intercepting the image area containing the personnel. The label information can include target attribute information including but not limited to gender, age, dress style, etc. For example, the EfficientNet architecture can be used to train the personnel attribute recognition model based on the training sample set. The EfficientNet architecture performs well in processing image classification and recognition tasks and has high computational efficiency. Key point positioning and attribute label extension are introduced in the recognition algorithm to improve the recognition accuracy of the model. The personnel attribute recognition model uses the EfficientNet architecture to extract features from the image, and introduces a key point detection module (OpenPose, etc.) into the framework, so that in addition to general image features, human key point features are also obtained. According to the detected key points and feature information, the personnel attributes are predicted. The key point detection module is introduced to help the model locate human body parts more accurately, thereby improving the accuracy of attribute recognition. Attribute label extension means that in addition to basic attributes such as gender and age, labels such as dress style and carried items are added to provide richer personnel attribute information.

[0080] Step S14: taking the scene type, the target object information, and the target attribute information as context information, and determining model input information based on the context information and the user input text.

[0081] The user input text may be a user's question. In this embodiment, the context information and the user input text may be determined as model input information based on a preset text structure.

[0082] Step S15: inputting the model input information into a natural language processing model, so that the natural language processing model generates scene understanding information corresponding to the image based on the prompt of the context information.

[0083] In this embodiment, the scene classification model mentioned above is used to obtain the scene category of the current image, and then the target detection model is used to obtain the target object information in the scene, including the object category and location information. If the target includes a person, the above-mentioned person attribute recognition model is used to obtain the relevant attributes of the person. This information is used as context information and input into the natural language processing model together with the standard input of the model (such as the question text sequence). The natural language processing model will generate a more appropriate answer based on the input context information.

[0084] It is understandable that in this embodiment, the scene type, target object information, and target attribute information are input into the natural language processing model to assist the model in generating relevant content. By integrating the output information of multiple models such as scene classification, target detection, and personnel attribute recognition, the natural language processing model's ability to understand the scene is enhanced, thereby generating a more accurate and rich scene description. Through the fusion of multimodal information, the model can understand the scene more comprehensively, improve the accuracy and richness of the description, and the generated description is not limited to a simple list of objects, but can reflect the story or behavior that occurs in the scene. For example, the scene classification model recognizes that the input image scene is a store, the target detection detects that the image contains people, shelves, beverages, and their corresponding positions, and the personnel attribute recognition model obtains the attributes of the current person as male, standing, and middle-aged. At this time, the text entered by the user is: Please describe the current picture. Then the natural language processing model will output such a scene description: "A middle-aged man is standing in front of the shelf with a bottle of beverage. He should have just bought the beverage from the shelf."

[0085] Among them, the natural language processing model can be trained based on retail industry knowledge. Retail-related text data is added to the pre-training stage of the model to enable it to have professional knowledge in the retail field. Training process: First, prepare text data containing retail industry terms, product descriptions, customer reviews, etc., and clean the data to remove irrelevant or redundant information to ensure data quality. You can use a series of models such as BERT, initialize the model with public pre-trained weights, and use the above retail industry text data to fine-tune the model to enhance the model's understanding of the retail industry.

[0086] The following is an example of a conversation: Customer asks: What are the special features of this XYZ brand 55-inch 4K smart TV? The answer of the basic model: This TV has a variety of features, including high-definition display, smart connectivity, and built-in applications. The answer of the model after adding the retail knowledge base: This XYZ brand 55-inch 4K smart TV is equipped with HDR10+ and DolbyVision, supports Alexa voice assistant and Google Assistant, and also provides multiple HDMI ports and USB interfaces, as well as built-in Wi-Fi and Bluetooth connectivity. From the above conversation, it can be seen that in the answer of the basic model, although some common features are mentioned, no specific technical details or professional terms are mentioned. In the answer of the enhanced model, professional retail industry terms such as HDR10+, Dolby Vision, and specific connection options are mentioned more specifically, which shows that the enhanced model can better understand the knowledge of the retail industry. In order to achieve this difference, text data containing retail industry terms needs to be prepared. Below are some example data points that can help the model learn vocabulary and expressions unique to the retail industry. For example, product descriptions: "This TV supports HDR10+ and Dolby Vision for excellent picture quality.", "This refrigerator is equipped with a double-layer fresh-keeping room, which can effectively keep food fresh." Customer reviews: "I really like the colors of this TV. HDR10+ really makes the picture look more vivid.", "The battery life of this phone is amazing. I still have power left after a day." Training data example: FAQ: "Does this TV support HDR10+?", "Answer: Yes, this TV supports HDR10+ and Dolby Vision, which can provide richer colors and better contrast." Through such data training, the model can learn how to more accurately understand and generate conversations related to the retail industry.

[0087] It is understandable that in order to achieve effective processing and analysis of multimodal data, this embodiment integrates the above-mentioned trained scene classification model, target detection model, personnel attribute recognition model and natural language processing model to form a comprehensive intelligent system. The front-end data preprocessing can be performed first: first, the picture or video frame captured by the camera is input into the preprocessing module, and the image is standardized, cropped and scaled to meet the input requirements of different models. The video frame is sent to the scene classification model, and then the scene type and video frame are sent to the target detection model. After that, the image area is intercepted based on the detection result of the target detection model, and the personnel attribute recognition model is input, and each model performs its specific task respectively. The results output by each model are sent to the back-end fusion layer (i.e., the natural language processing model), which is responsible for integrating information from different models into a unified scene understanding.

[0088] This embodiment can adopt one or more of the following strategies in the training process of scene classification model, target detection model, and personnel attribute recognition model: Hyperparameter search strategy: Use grid search or Bayesian optimization and other methods to automatically find the best hyperparameter combination of the model, such as learning rate, batch size, regularization coefficient, etc., to achieve the best detection accuracy and operating efficiency. Model pruning strategy: Prune the model structure, remove redundant nodes and connections, reduce the complexity of the model, reduce the amount of calculation and memory usage, while maintaining high detection accuracy. Quantization and compression strategy: Convert model parameters from floating point numbers to integers and quantize them. At the same time, model compression techniques such as weight sharing or matrix decomposition may be applied to further reduce the model size and accelerate the reasoning process. Furthermore, this embodiment can also perform adaptive threshold adjustment: dynamically adjust the decision threshold of the model according to changes in the scene. For example, a lower confidence threshold may be required to improve the detection rate under low light conditions.

[0089] In addition, the collaborative optimization between models based on the attention mechanism in this embodiment introduces the attention mechanism between models, so that some models (such as target detection) can guide other models (such as personnel attribute recognition) to focus on the area of ​​interest, thereby improving the overall efficiency. In a multi-model system, the attention mechanism can be used to guide one model to pay attention to the key information provided by another model. The attention mechanism allows the model to focus on the most relevant part when processing the input, thereby improving the efficiency and accuracy of the model. First, the scene classification model classifies the input image and identifies that the image belongs to a specific category. According to the result of the scene classification, the target detection model adjusts the confidence of a specific category according to the result of the scene classification. The adjustment can be based on the weight adjustment. When performing the detection, not only the characteristics of the object are considered, but also the weight assigned by the attention mechanism is referred to increase the detection probability with high scene relevance and reduce the detection probability with low relevance. The object position information generated by the target detection model is used to assign different attention weights to different areas in the image. The personnel attribute recognition model receives the weighted image area as input, pays more attention to the area where the person is located, and ignores the background information, thereby improving the accuracy and efficiency of the recognition.

[0090] Furthermore, in the context of multi-model fusion, real-time and resource constraints have become particularly important considerations, especially in the retail industry, which requires processing of a large number of real-time video streams. In order to overcome these challenges, this embodiment adopts hardware acceleration and asynchronous processing to optimize the performance of the multi-model system. Among them, hardware acceleration adopts multi-GPU (Graphics Processing Unit) parallel computing to provide high-speed computing capabilities for model reasoning. Asynchronous processing adopts multi-threading and asynchronous I / O (Input / Output). Multi-threading technology on a multi-core processor can process multiple requests in parallel, while asynchronous I / O allows the system to perform other tasks while waiting for I / O operations to complete, thereby improving the responsiveness and throughput of the system. At the same time, the event-driven architecture can better handle concurrent requests, and each event triggers an independent task, which enables the system to effectively manage resources and respond to multiple users or data streams.

[0091] Furthermore, this embodiment may use cross-validation technology to evaluate the performance of the model on unseen data to ensure the generalization ability and robustness of the model.

[0092] Furthermore, this embodiment can deploy an online learning mechanism: set a data stream interface to achieve real-time data collection and model adaptive learning. In a rapidly changing retail environment, the adaptive learning ability of the model is crucial. This requires not only that the model can capture new product categories and customer behavior patterns in a timely manner, but also that it can quickly adjust itself to cope with these changes. The following are specific steps for designing an online learning mechanism, including self-learning strategies and the algorithm model used.

[0093] Design data flow interface: Build a real-time data pipeline to connect front-end data collection devices (cameras, user interaction records, etc.) with back-end data analysis systems. The data pipeline should have the ability to efficiently transmit and store data and support streaming processing. Data preprocessing: Before data enters the model, necessary preprocessing is performed, including cleaning, standardization, outlier detection, and missing value filling to ensure data quality. Feature engineering: Update feature sets in real time, adjust feature selection and construction according to business needs and model performance to improve the model's predictive ability.

[0094] Online learning: Adopting an incremental learning strategy, the model is allowed to gradually absorb the information of new samples without forgetting the existing knowledge. Active learning is implemented in the data stream, and the model can select the most valuable samples for annotation and learning, thereby making more effective use of limited human resources.

[0095] Regular model iteration: Periodic retraining, set the model retraining cycle, and regularly use the accumulated new data to retrain the model to maintain the timeliness and accuracy of the model.

[0096] That is, this embodiment can collect scene understanding information output by the natural language processing model, user input text, and images corresponding to the scene understanding information based on user feedback information, take positive user feedback information as positive samples and negative user feedback information as negative samples, and regularly train and adjust the target model based on the positive samples and / or negative samples. The target model includes one or more of a scene classification model, a target detection model, a personnel attribute recognition model, and a natural language processing model.

[0097] Version control and rollback: Implement model version control so that if the new model performs poorly, it can be quickly rolled back to the previous stable version.

[0098] Performance monitoring and alerts: Establish a model performance monitoring system. When the model accuracy, latency or other key indicators drop, an alert is triggered and timely intervention is carried out. That is, before releasing a new version, the new version needs to be tested. The scene understanding model is a composite model that integrates multiple sub-models, combining multiple tasks such as scene classification, object detection, personnel attribute recognition and natural language processing. In order to evaluate the accuracy of the performance of such a composite model, it is necessary to comprehensively consider the performance of each sub-model and the interaction between them. Single-task accuracy: For each individual task (such as scene classification, object detection, personnel attribute recognition, etc.), calculate its accuracy separately. Joint accuracy: Evaluate the comprehensive performance of the entire composite model on all tasks. That is, calculate the proportion of correct answers on all tasks at the same time. In addition to accuracy, other related indicators include latency, which can be used to evaluate the performance of industry large models (the calculation method is similar to accuracy, divided into single task and joint). Latency: Measures the time required for the model to receive input and generate output.

[0099] It can be understood that this embodiment builds an adaptive detection framework for scene perception and proposes an adaptive scene perception model based on a convolutional neural network and an attention mechanism. The model first uses a scene classification model to make a preliminary scene label prediction for the input image, and then uses a collaborative optimization method based on the attention mechanism to adjust the confidence of the target detection network according to the prediction results, thereby enhancing the detection accuracy of high-probability targets in specific scenarios. For example, in the back kitchen scene, the model will automatically enhance the recognition ability of kitchen supplies, while in the supermarket scene, it will increase the sensitivity to product barcodes. In addition, multimodal identity and behavior recognition are performed, and a set of identity and behavior recognition models based on the Transformer architecture are developed. The model can not only identify the gender and age of individuals, but also obtain target posture information through a posture detection algorithm (OpenPose), enriching the input features of the recognition model, and helping the model to infer implicit information such as shopping intentions and fatigue levels through posture and behavior analysis. By introducing the attention mechanism algorithm, the object position information generated by the target detection model is used to assign different attention weights to different areas in the image. The output of the identity and behavior recognition model is combined with the output of the target detection model to generate richer context descriptions and improve the overall scene understanding ability. In addition, scene description generation that integrates natural language processing, by introducing pre-trained natural language processing (NLP) models, such as BERT, enhances the natural language processing model's ability to understand the scene by integrating the output information of multiple models such as scene classification, target detection, and personnel attribute recognition, thereby generating more accurate and rich scene descriptions. Through the fusion of multimodal information, the model can understand the scene more comprehensively and improve the accuracy and richness of the description. The generated description is not limited to a simple list of objects, but can reflect the story or behavior that occurs in the scene.

[0100] Therefore, the final natural language processing model has stronger understanding ability, better scene perception ability, flexibility and scalability, and can add more visual models or attribute information as needed to further enhance the details and accuracy of the description. For example, "a young woman is picking fruits in a supermarket" improves the model's scene understanding and expression capabilities.

[0101] In addition, it can learn and optimize dynamically. We designed an online learning framework so that the model can learn new concepts from streaming data without supervision, such as the identification of new categories of goods or new customer behavior patterns. Combining user feedback and market data, the model can achieve self-iteration through the constructed learning strategy, maintaining its robustness and timeliness in a dynamic environment.

[0102] In the rapidly developing retail industry, intelligent understanding of the in-store environment and accurate analysis of customer behavior have become the key to improving operational efficiency and service quality. However, existing image recognition technologies and target detection methods face significant challenges in complex and changing retail environments, as shown in the following: Limited detection accuracy: Traditional image recognition algorithms often have difficulty in achieving high-precision target detection when dealing with retail scenarios, due to factors such as the wide variety of shelf goods, diverse customer behaviors, and changing light conditions, especially in densely placed areas. Lack of scene understanding capabilities: Most target detection models can only recognize single or multiple objects, but cannot understand the meaning of these objects in a specific scenario and their relationship with each other. For example, they can recognize "shopping cart" and "customer", but cannot determine whether the customer is using the shopping cart to shop. Limitations of natural language models in retail scenarios: Although natural language processing technology has made great progress in text understanding, in retail scenarios, scene understanding models that rely solely on text data generation often fail to capture the rich details contained in visual information, resulting in insufficient accuracy and completeness of understanding. At present, a single model is difficult to take into account: A single image recognition or target detection model is difficult to fully cover all elements in a retail scenario, such as goods, customers, employees, and their interactive behaviors. The model has weak generalization ability: Due to the high dynamics of the retail environment, a single model has poor adaptability in different scenarios, especially when factors such as lighting, angle, and occlusion change, the model's performance will drop significantly. Cross-modal information fusion is difficult: It is a technical challenge to effectively integrate data from multiple modalities such as images, videos, and text to form a unified scene understanding, which requires overcoming the semantic gap between data of different modalities. Therefore, there is an urgent need for a large model construction method that can integrate multi-source information, adapt to complex retail environments, and have a high degree of scene understanding capabilities, so as to break through the limitations of existing technologies and provide strong technical support for the intelligent upgrade of the retail industry. This application provides a large model construction method for scene understanding for the retail industry, which aims to build a comprehensive and accurate retail scene understanding system by integrating scene classification, human attribute recognition, target detection, and natural language processing technology, so as to effectively improve the level of intelligent management in the retail industry.

[0103] There are the following difficulties in scene understanding in the retail industry. Cross-domain adaptability: Retail scenes differ greatly from other scenes in terms of lighting conditions, viewing angles, backgrounds, crowd density, etc. Traditional target detection models are general models that cannot cope with the particularities of retail scenes. This may lead to poor performance of pre-trained models in retail scenarios. Multimodal fusion: Combining visual information with natural language descriptions involves the understanding and fusion of multimodal data, which requires the model to be able to process different types of information at the same time and maintain consistency. Robustness in dynamic environments: Product placement, customer behavior, and the environment change frequently, and the model needs to be highly robust to cope with changing scenes, which requires the model to have strong perception capabilities. Model iteration and update: Products and services in the retail industry change rapidly, and the model needs to obtain new data and retrain in a timely manner, but how to maintain stability during online learning and model updates is also a difficult problem.

[0104] For the retail industry, this embodiment can establish a retail data set and knowledge base. Among them, data set construction: by collecting a large number of images of different retail scenes, including but not limited to the front hall, back kitchen, shopping mall, supermarket, cashier, warehouse, street, etc., mark the target objects, character attributes and behaviors, and create a comprehensive retail industry-specific data set. Data screening and annotation: a professional team will screen the data and annotate the targets, scenes, and character attributes. Knowledge base construction: integrate industry specifications, product catalogs, consumer behavior research and other materials to build a retail industry knowledge graph, provide domain background information for the natural language processing model, and enhance the accuracy and richness of the scene description. Use labeled data and knowledge base to train scene classification models, target detection models, personnel attribute recognition models and natural language processing models. In order to ensure the high performance of the model in the retail environment, a deep learning framework is adopted, and customized adjustments are made to the specific needs of the retail industry, integrating different algorithm frameworks, and finally forming a large model of scene understanding exclusive to the retail industry. During the training process, pay attention to the collaborative work between models to ensure that the models can be seamlessly integrated and jointly improve the performance of the entire system. Through carefully designed training strategies and parameter adjustments, we can achieve the best balance between accuracy and real-time performance, providing strong intelligent support for the retail industry. In addition, in order to accelerate the training process and optimize resource allocation, we also use technical means such as mixed precision training and distributed training to ensure the best training effect under limited computing resources. Through these customization and optimization measures, we can better adapt to the complexity and diversity of the retail environment and provide more accurate services.

[0105] In addition, this embodiment can be deployed in the cloud or on a mobile terminal to detect image scene information. A data set is established for the retail industry. By integrating scene classification models, human attribute recognition models, target detection models, scene description generation of natural language processing, and dynamic learning and optimization, the accuracy of scene image detection and understanding in the retail industry is significantly improved, providing strong technical support for store management, customer flow analysis, etc. in the retail industry. When understanding scenes in the retail industry, it is possible to semantically understand pictures or videos in retail scenes to form scene semantic information. An intelligent, efficient, and flexible retail industry solution is provided to build a comprehensive and accurate retail scene understanding system, effectively improving the intelligent management level of the retail industry. Including but not limited to the following aspects: Refined operations: By deeply understanding customer behavior and preferences, retail companies can optimize product layout, provide personalized services, and improve customer satisfaction. Inventory management: Real-time monitoring of shelf vacancy status, automatic triggering of replenishment processes, and reducing out-of-stock risks. Security monitoring: Timely detection of abnormal behaviors, such as theft attempts, to improve store security.

[0106] See also Figure 2 As shown, this embodiment provides a scene understanding information generating device, including:

[0107] A scene classification module 11 is used to input an image into a scene classification model to obtain a scene type corresponding to the image output by the scene classification model;

[0108] A target detection module 12 is used to identify the target object in the image using a target detection model to obtain target object information, wherein the target object information includes target type and position information, and the target type includes objects and people;

[0109] A person identification module 13, used to identify target attribute information of a person in the image based on a person attribute identification model;

[0110] An input determination module 14, configured to use the scene type, the target object information, and the target attribute information as context information, and determine model input information based on the context information and user input text;

[0111] The information generation module 15 is used to input the model input information into a natural language processing model, so that the natural language processing model generates scene understanding information corresponding to the image based on the prompt of the context information.

[0112] It can be seen that the embodiment of the present application performs scene classification, target detection and personnel attribute recognition on the image to obtain the scene type, target object information and personnel attribute information as context information. The context information contains rich image information, which can prompt the natural language processing model to understand the scene of the image more accurately, improve the perception ability of the scene, generate more accurate scene understanding information, and thus improve the user experience.

[0113] The personnel identification module 13 may specifically include:

[0114] An image region interception submodule, used for intercepting an image region containing a person from the image based on the target object information;

[0115] The attribute information recognition submodule is used to recognize target attribute information of people in the image area by using a person attribute recognition model.

[0116] Among them, the image area capture submodule is specifically used to determine the capture range based on the location information of the person; based on the capture range, the image area containing the person is captured from the image, wherein the image area includes the person area and part of the background area.

[0117] Correspondingly, the attribute information recognition submodule is specifically used to assign weighted weights to the image area based on the location information of the personnel, wherein the weighted weight of the personnel area is higher than the weighted weight of the partial background area; weighting the image area using the weighted weights to obtain a weighted image; inputting the weighted image into the personnel attribute recognition model to obtain the target attribute information of the personnel in the weighted image.

[0118] In an optional embodiment, the target detection model is trained based on an image training sample set, the label information of the image training sample set includes the target type and location information of the target object, and, during the training process, the training loss is calculated by weighting the category loss based on the weighted value corresponding to the specified target object.

[0119] In an optional embodiment, the personnel identification module 13 is specifically used to extract image features in the image using the convolution module in the personnel attribute recognition model, and to extract key point features in the image using the key point detection module in the personnel attribute recognition model; and to determine the target attribute information of the person in the image based on the image features and the key point features.

[0120] In an optional embodiment, the target detection module 12 is specifically used to use a target detection model to identify type information of the target object in the image, and adjust the confidence corresponding to the type information based on the scene type, and determine the target type of the target object based on the confidence.

[0121] See also Figure 3 As shown, an embodiment of the present application discloses an electronic device 20, including a processor 21 and a memory 22; wherein the memory 22 is used to store a computer program; the processor 21 is used to execute the computer program, the scene understanding information generation method disclosed in the above embodiment.

[0122] For the specific process of the above-mentioned scene understanding information generation method, reference may be made to the corresponding content disclosed in the aforementioned embodiments, which will not be repeated here.

[0123] Furthermore, the memory 22 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0124] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26; wherein the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input / output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0125] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the scene understanding information generation method disclosed in the aforementioned embodiment.

[0126] For the specific process of the above-mentioned scene understanding information generation method, reference may be made to the corresponding content disclosed in the aforementioned embodiments, which will not be repeated here.

[0127] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0128] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0129] The above is a detailed introduction to a scene understanding information generation method, device, equipment and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for generating scene understanding information, characterized in that: include: Inputting an image into a scene classification model to obtain a scene type corresponding to the image output by the scene classification model; Using a target detection model to identify a target object in the image, and obtaining target object information, wherein the target object information includes target type and location information, and the target type includes objects and people; Identify target attribute information of a person in the image based on a person attribute recognition model; Taking the scene type, the target object information, and the target attribute information as context information, and determining model input information based on the context information and user input text; The model input information is input into a natural language processing model so that the natural language processing model generates scene understanding information corresponding to the image based on the prompt of the context information.

2. The scene understanding information generation method according to claim 1, characterized in that: The identifying target attribute information of a person in the image based on a person attribute recognition model includes: Based on the target object information, extracting an image region containing a person from the image; The target attribute information of the person in the image area is identified using a person attribute recognition model.

3. The scene understanding information generation method according to claim 2, characterized in that: The step of intercepting an image region containing a person from the image based on the target object information includes: Determine the interception range based on the location information of the personnel; An image region containing a person is intercepted from the image based on the interception range, wherein the image region includes a person region and a partial background region.

4. The scene understanding information generation method according to claim 3, characterized in that: The step of using a person attribute recognition model to recognize target attribute information of a person in the image region includes: Allocating a weighted weight to the image region based on the position information of the person, wherein the weighted weight of the person region is higher than the weighted weight of the partial background region; weighting the image region using the weighted weights to obtain a weighted image; The weighted image is input into the personnel attribute recognition model to obtain target attribute information of the personnel in the weighted image.

5. The scene understanding information generation method according to claim 1, characterized in that: The target detection model is trained based on an image training sample set, the label information of the image training sample set includes the target type and location information of the target object, and during the training process, the training loss is calculated by weighting the category loss based on the weighted value corresponding to the specified target object.

6. The scene understanding information generation method according to claim 1, characterized in that: The identifying target attribute information of a person in the image based on a person attribute recognition model includes: Extracting image features in the image using a convolution module in the personnel attribute recognition model, and extracting key point features in the image using a key point detection module in the personnel attribute recognition model; Target attribute information of the person in the image is determined based on the image features and the key point features.

7. The method for generating scene understanding information according to any one of claims 1 to 6, characterized in that: The step of using the target detection model to identify the target object in the image and obtain target object information includes: The target detection model is used to identify type information of the target object in the image, and the confidence corresponding to the type information is adjusted based on the scene type, and the target type of the target object is determined based on the confidence.

8. A scene understanding information generating device, characterized in that: include: A scene classification module, used to input an image into a scene classification model to obtain a scene type corresponding to the image output by the scene classification model; A target detection module, used to identify the target object in the image using a target detection model to obtain target object information, wherein the target object information includes target type and location information, and the target type includes objects and people; A person identification module, used to identify target attribute information of a person in the image based on a person attribute identification model; An input determination module, configured to use the scene type, the target object information, and the target attribute information as context information, and determine model input information based on the context information and user input text; An information generation module is used to input the model input information into a natural language processing model so that the natural language processing model generates scene understanding information corresponding to the image based on the prompt of the context information.

9. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store the computer program; The processor is used to execute the computer program to implement the scene understanding information generating method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the scene understanding information generating method according to any one of claims 1 to 7 is implemented.

Images