Implementation method for identifying handshake and driving dialogue based on visual model

Abstract

The invention discloses an implementation method for identifying handwrites and driving dialogues based on a visual model, and belongs to the technical field of artificial intelligence and intelligent interaction. According to the method, the technical problem that an existing handmade display scene lacks handmade identification and interaction functions is solved. According to the technical scheme, the method comprises the steps of training and deploying a visual identification model for handwritings; configuring a multi-stage identification model strategy; and executing a hand-made identification driving intelligent interaction process. Finally, the camera detects and recognizes the handshake, obtains a handshake brand IP image and a wearing detail feature tag set, and uses the tag set as a parameter to call a streaming end-to-end voice large model cloud service to drive immersive dialogue interaction conforming to the handshake style.

Description

Technical Field

[0001] This invention relates to a method for artificial intelligence recognition and dialogue for figurines, and more specifically, to a system implementation method for recognizing figurines based on visual models and driving intelligent dialogue interaction through recognition features. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent interactive systems have been widely applied in various fields. Traditional figurines are mainly for static viewing and lack the ability to interact with users. In recent years, technicians have begun to explore the application of intelligent recognition and dialogue technologies to the transformation of figurine display devices to improve the user experience. However, existing figurine recognition usually requires embedding RFID chips in the figurine and relying on a specific base. This method requires modification of the figurine itself, has weak compatibility, and cannot be applied to various existing figurine products on the market. Secondly, intelligent dialogue systems use a traditional three-stage processing method of speech recognition, large text model, and speech synthesis. This segmented processing results in significant interaction delays and an unsmooth user experience. Summary of the Invention

[0003] To address the problems in existing technologies where figurine recognition requires specific structural modifications and bases, and is difficult to integrate with existing figurines to drive intelligent dialogue interaction with specific figurine IP styles, this invention provides a method for recognizing figurines and driving dialogue based on a visual model. Using this method, figurine recognition and intelligent dialogue applications can be achieved by developing only a mobile app or by modifying figurine displays and cabinets using only embedded systems.

[0004] The method for recognizing figurines and driving dialogue based on visual models, which solves the technical problem in this invention, includes the following steps:

[0005] Step S1 involves training and deploying a visual recognition model for figurines, including selecting a basic visual recognition model, collecting training samples, classifying and labeling samples, training and validating the model, converting the model format, and deploying the system.

[0006] First, select a stable and suitable visual target recognition model for deployment in the system environment. For example, when modifying a collectible cabinet, you need to choose YOLOv8n, which is suitable for embedded systems, as the basic model. If you are developing a mobile APP, you also need to choose a small-sized visual basic model with compatible performance requirements.

[0007] Furthermore, a multi-angle, multi-light sampling method with the figurine as the origin was adopted. Multiple rotation radii were set to simulate cameras at different display distances to collect sample videos of figurines of various brand IPs and images.

[0008] Furthermore, after segmenting the sample videos into frames, multi-level classification labels are applied using image labeling tools. The samples are then divided into training, validation, and test sets according to the proportions. Two-level model training is performed using CPU or GPU based on the multi-level labels (the first-level model training uses brand IP classification, and the second-level model training uses brand IP sub-product image classification).

[0009] Furthermore, the model is converted into a format suitable for the actual operating system, and the software environment required for the model to run is installed.

[0010] Step S2: Configure the multi-level recognition model strategy, including a first-level classification model for recognizing brand IP (e.g., Brand A puppy); a second-level classification model (optional, configured as needed) for recognizing sub-product images under the brand IP category (e.g., chef puppy); and a third-level classification model (optional, configured as needed) that uses the original basic model to recognize the wear features of the figurine (e.g., spatula).

[0011] Step S3, the implementation of figurine recognition and intelligent interaction process, includes periodically calling the camera to photograph the area where the figurine is placed, submitting the image to the multi-level visual recognition model, outputting the result to form a structured figurine feature tag set, playing a welcome introduction voice based on the tag set result, and if there is a network connection, calling the end-to-end voice big model cloud service with the recognized feature tags as parameters through the WebSocket protocol to realize streaming intelligent dialogue interaction through the streaming end-to-end voice big model cloud service to perform intelligent dialogue interaction, and if there is no network connection, calling the local voice interaction model.

[0012] The beneficial effects of this invention are as follows: Compared with existing technologies, by introducing a visual model-based recognition technology, it achieves the recognition of figurines and their rich detailed features without modifying the design structure of the figurines themselves, thus improving the compatibility of figurine recognition; by adopting a multi-level recognition model strategy, combining the figurine brand IP, the image information of sub-products under a certain brand IP, and the recognition of general features based on the basic model, a rich set of figurine feature tags is constructed. By using the figurine feature tags as parameters for end-to-end large voice model invocation, a deeply immersive effect of the intelligent dialogue system is achieved. This method is applicable to various display devices such as figurine cabinets and display stands, and has strong versatility and practicality. The modified device can be applied to figurine shops, exhibition halls, and home figurine display scenarios. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating the implementation steps of the method of the present invention.

[0014] Figure 2 The diagram shows a structural block diagram of an example of modifying a figurine display cabinet based on the method of this invention. Detailed Implementation

[0015] The following is in conjunction with the appendix Figure 1 and appendix Figure 2 This application will be described in further detail so that those skilled in the art can better understand and implement the invention, but the embodiments described are not intended to limit the invention. Example

[0016] This implementation provides an example of using the method of the present invention to develop, deploy, and modify a figurine display cabinet based on an embedded system. The specific implementation steps are as follows:

[0017] Step S1 involves training and deploying a visual recognition model for figurines, including selecting a basic visual recognition model, collecting training samples, classifying and labeling samples, training and validating the model, converting the model format, and deploying the system.

[0018] First, in the basic model selection stage, YOLOv8n was chosen as the basic visual target recognition model. YOLOv8n has only 3.2M parameters, which balances accuracy and real-time performance and is suitable for deployment in embedded environments.

[0019] Furthermore, during the training sample collection phase, ten core brand IPs on the market were collected, along with five figurines of each IP's sub-product image. Three rotation radii (30cm, 15cm, and 10cm) were set around the figurines to simulate cameras at different display distances. Frontal and back views were captured horizontally, as well as overhead views at 30° and 60° angles, covering multiple viewing angles of the figurines. Bright and gentle lighting were used for each rotation angle. FFmpeg was used to segment the captured videos into frames to obtain sample images.

[0020] Furthermore, in the sample classification and labeling stage, the labelimg tool is used to label the sample images. The category content includes a two-level label classification of the brand IP of the figurine and the specific brand IP sub-product image.

[0021] Furthermore, during the model training and validation phases, the collected sample datasets are randomly shuffled in an 8:1:1 ratio for training, validation, and testing. The training parameters are as follows: model.train( data="toy.yaml", # Dataset configuration file epochs=100, # Number of training epochs imgsz=416, # Input image size batch=4, # Batch size, adjust according to video memory. device='cpu', # GPU 0 project="runs / train", # Directory for saving training results name="toy", # The name of this training session exist_ok=True, # Allow overwriting if the directory already exists. cache=True, # Caching data speeds up training save_period=1, # Save weights each round patience=20, # Number of early stops verbose=True, # Display training information and loss in real time lr0=0.01 # Initial learning rate ) Based on the two-layer labels, the model was trained using CPU to obtain one first-level model (10 categories) and ten second-level models (5 categories).

[0022] Furthermore, in the model format conversion and model embedded system deployment stage, the .pt format model is converted into the .onnx format model and deployed on a Raspberry Pi or NVIDIA Jetson hardware environment, and the basic software system for model operation is built, including the Linux operating system, Python, and Ultralytics environment.

[0023] Step S2: Configure a multi-level recognition model strategy, including a first-level classification model for recognizing brand IP, a second-level classification model for recognizing sub-product images under the brand IP category, and a third-level classification model that uses the original basic model to recognize the wear features of the figurine.

[0024] First, the first-level classification module prioritizes identifying the brand IP to which the figure belongs based on the classification model's confidence level (greater than 0.6). For example, if a figure belongs to a brand's dog IP, it is included in the IP tag. By adjusting the weights of lower classification categories, the module ensures that brand IP characteristics can be accurately identified, providing a foundation for subsequent segmentation and identification.

[0025] Furthermore, the second-level classification module identifies specific sub-product image information based on the first-level IP classification. It uses the first-level classification results as input to a model trained on sub-product samples of figurines belonging to this IP category, further extracting specific figurine image features from the same IP, such as chef dogs or pilot dogs, and incorporating them into the sub-product image tags. This hierarchical recognition method enables more accurate image identification based on a pre-defined IP.

[0026] Furthermore, the third-level classification module identifies specific features based on the second-level classification. Using a base model such as YOLOv8, it further identifies detailed clothing features of the figure, such as hats and spatulas, and incorporates these features into the detailed clothing feature labels. This level of recognition can capture the figure's subtle features, providing rich information support for subsequent personalized dialogue.

[0027] Step S3, the figurine recognition and intelligent interaction process is implemented. First, the main program calls the camera every 5 seconds to detect the target area, acquire the image of the target area, and convert the acquired image into a format supported by YOLOv8n with a size of 416x416 pixels.

[0028] Furthermore, the data is sent to a visual recognition model for identification.

[0029] Furthermore, the model outputs a structured set of figurine feature labels, including a main label, auxiliary labels, and their respective confidence scores. The recognition result is a JSON object containing five fields: {"brand": "a manufacturer", "ip": "puppy", "product": "chef puppy", "confidence": 0.96, "auxiliary_tags": ["chef's hat", "spatula"]}. The confidence field represents the confidence score of the main label, and the auxiliary_tags field contains auxiliary label information, providing rich contextual information for subsequent intelligent dialogue.

[0030] Furthermore, a welcome voice message is played: "Hello! I am a puppy from a certain manufacturer, welcome to my kitchen," generating a personalized welcome voice message based on the recognized figurine feature information.

[0031] Furthermore, using the feature tag set as parameters, an end-to-end large-scale voice model cloud service is invoked via the WebSocket protocol to achieve streaming intelligent dialogue interaction. Network connection status is detected through a Wi-Fi connection detection module. If a Wi-Fi signal is detected, a connection to a cloud service API, such as the Baidu Smart Cloud audio-mini-realtime-near model, is established via the WebSocket protocol. The identified feature tag set is then used to send a session.updated event for intelligent dialogue. The use of the WebSocket protocol ensures real-time performance and smoothness, making the dialogue interaction more natural. When there is no network connection, local voice with a tone similar to that of the figure's IP is invoked for interaction, ensuring basic interactive functionality is still provided offline.

[0032] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can be made to the technical solutions described in the embodiments, or equivalent substitutions can be made to some of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for recognizing figurines and driving dialogue based on a visual model, characterized in that... Includes the following steps: Step S1, training and deployment of the visual recognition model for figurines: including selection of basic visual recognition model, collection of training samples, classification and labeling of samples, model training and verification, model format conversion and system deployment; Step S2, configure a multi-level recognition model strategy: including a first-level classification model for recognizing brand IP, a second-level classification model (optional) for recognizing sub-product images under the brand IP category, and a third-level classification model (optional) for recognizing the wear features of the figurine using the original basic model; Step S3, the implementation of figurine recognition and intelligent interaction process: includes periodically calling the camera to take pictures of the figurine placement area, submitting the images to the multi-level visual recognition model to output the results to form a structured figurine feature tag set, playing a welcome introduction voice based on the tag set results, and calling the end-to-end voice big model cloud service or local voice for intelligent dialogue interaction based on the network connection status and the tag set as parameters.

2. The implementation method according to claim 1, characterized in that, The sample classification labeling in step 1 requires two layers of labels: brand IP label and sub-product image label under brand IP category; the model training and validation in step 1 requires two classification model trainings based on brand IP label and sub-product image label under brand IP category.

3. The implementation method according to claim 1, characterized in that, In step 3, the tag set is used as a parameter to call the end-to-end voice big data cloud service to realize streaming intelligent dialogue interaction through the WebSocket protocol.

Images