Image shooting interaction method and shooting terminal

Abstract

The invention relates to the technical field of information processing, in particular to an image shooting interaction method and a shooting terminal, and the method is applied to the shooting terminal. The shooting terminal provides a multi-entrance shooting interface; a first entry mode is set in the multi-entry shooting interface, the first entry mode has a vertical response model corresponding to the first entry mode, and the vertical response model is matched with the corresponding image category. The image shooting interaction method provided by the invention is applied to the shooting terminal providing a multi-entrance shooting interface, and dispersive multifunctional entrances are aggregated to a unified multi-entrance shooting interface, so that a photographer can realize diversified function requirements without cutting out the interface, and the split feeling of shooting operation is eliminated; besides, different function entry modes in the multi-entry shooting interface correspond to multiple background model programs respectively, actual function requirements of photographers can be distinguished, and the obtained images are shunted to the suitable background model programs to be processed, so that the accuracy and the reliability of response results are improved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and in particular to an image shooting interaction method and a shooting terminal. Background Art

[0002] With the rapid development of Internet technology, the interaction entrances between photographers and service systems are becoming increasingly diverse and complex.

[0003] However, in the related art, an interactive interface and a background model program are usually developed separately for each functional entry. When the photographer calls different background model programs based on the required functional requirements, he often needs to switch to different interactive interfaces or even applications, which leads to the fragmentation of each functional entry, seriously affecting the photographer's experience. In addition, the calling of the background model program is completely dependent on the photographer's understanding of the functional requirements, and there is a lack of a diversion mechanism, resulting in the called model not necessarily fully matching the functional requirements actually required by the photographer. Summary of the invention

[0004] In view of this, an object of the present invention is to provide an image shooting interaction method and a shooting terminal.

[0005] In a first aspect, an embodiment of the present invention provides an image shooting interaction method, which is applied to a shooting terminal; the shooting terminal provides a multi-entry shooting interface; a first entry mode is set in the multi-entry shooting interface, and the first entry mode has a vertical category response model corresponding thereto, and the vertical category response model matches the corresponding image category; The method includes: In response to a shooting instruction triggered by the first entry mode, acquiring the shot first target image, and inputting the first target image into a universal recognition model to determine a recognition result; In response to the recognition result matching the vertical category recognition model corresponding to the first entry mode, the recognition result is input into the corresponding vertical category recognition model to obtain a corresponding response result; In response to the recognition result not matching the vertical category recognition model corresponding to the first entry mode, inputting the recognition result into the corresponding diversion decision model to determine the corresponding vertical category recognition model; The recognition result is input into the determined vertical category recognition model to obtain the corresponding response result.

[0006] In combination with the first aspect, a second entry mode is further provided in the multi-entry shooting interface, and the method further includes: In response to a shooting instruction triggered by the second entry mode, acquiring the shot second target image, and inputting the second target image into the universal recognition model to determine a recognition result; The recognition result is input into the corresponding diversion decision model to determine the corresponding vertical classification recognition model; The recognition result is input into the determined vertical category recognition model to obtain the corresponding response result.

[0007] In combination with the first aspect, a third entry mode is further provided in the multi-entry shooting interface, and the method further includes: In response to an upload instruction triggered by the third entry mode, acquiring the uploaded third target image, and inputting the third target image into the universal recognition model to determine a recognition result; The recognition result is input into the corresponding diversion decision model to determine the corresponding vertical classification recognition model; The recognition result is input into the determined vertical category recognition model to obtain the corresponding response result.

[0008] In combination with the first aspect, the recognition result includes the recognized image features and the description text corresponding to the image features, and the response result is generated by the large language model.

[0009] In combination with the first aspect, a fourth entry mode is further provided in the multi-entry shooting interface, and the method further includes: In response to the session instruction triggered by the fourth entry mode, the corresponding session content is obtained, and the session content is used to input into the general recognition model to determine the recognition result, and / or, to input into the vertical recognition model to obtain the corresponding response result.

[0010] In combination with the first aspect, the first entry mode is a shooting and recognizing face mode, the vertical recognition model corresponding to the first entry mode is a face recognition model, and the corresponding image category is a face image; the recognition result includes facial index information and skin quality index information; The step of inputting the recognition result into the corresponding vertical category recognition model to obtain the corresponding response result includes: The facial index information and skin index information are input into the face recognition model, and a first response result is output; the first response result includes at least one of the photographer's physical health status, physical indexes, and improvement suggestions; wherein the face recognition model is trained based on a fatigue degree algorithm.

[0011] In combination with the first aspect, the first entry mode is a tongue coating recognition mode of photographing, the vertical category recognition model corresponding to the first entry mode is a tongue coating recognition model, and the corresponding image category is a tongue coating image; the recognition result includes tongue coating indicator information; The step of inputting the recognition result into the corresponding vertical category recognition model to obtain the corresponding response result includes: The tongue coating index information is input into the tongue coating recognition model, and a second response result is output; the second response result includes at least one of the physical index, body index, physical health status, and improvement suggestions of the photographer; wherein the tongue coating recognition model is trained based on the health level algorithm.

[0012] In combination with the first aspect, the first entry mode is a shooting and identifying food mode, the vertical category recognition model corresponding to the first entry mode is a food recognition model, and the corresponding image category is a food image; the recognition result includes at least food information; The step of inputting the recognition result into the corresponding vertical category recognition model to obtain the corresponding response result includes: In response to the category information of the ingredient information matching the subcategories in the preset ingredient name list, the ingredient information is input into an ingredient recognition model, and a third ingredient response result is output; wherein the ingredient recognition model is trained based on the g-Dino detection method and the ROI area extraction method.

[0013] In combination with the first aspect, the second entry mode is a shooting recognition diversion mode, and the second entry mode corresponds to a general recognition model and a diversion decision model; the diversion decision model includes: A multimodal feature extraction module is used to extract features from the image to be processed and output a feature vector; The diversion decision fully connected network is used to perform data processing on the input feature vector to determine the target vertical category response model; wherein, the diversion decision fully connected network includes an input layer, a fully connected layer, and an output layer, and a Softmax layer or a Sigmoid layer is also arranged between the fully connected layer and the output layer.

[0014] In a second aspect, the present application also provides a photographing terminal, which includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the photographing terminal to execute the above method.

[0015] The embodiments of the present invention bring the following beneficial effects: an image shooting interaction method and a shooting terminal provided in the present application are applied to the shooting terminal; the shooting terminal provides a multi-entry shooting interface; a first entry mode is set in the multi-entry shooting interface, and the first entry mode has a vertical category response model corresponding to it, and the vertical category response model matches the corresponding image category; the method includes: in response to a shooting instruction triggered by the first entry mode, acquiring the captured first target image, and inputting the first target image into a general recognition model to determine the recognition result; in response to the recognition result matching the vertical category recognition model corresponding to the first entry mode, the recognition result is input into the corresponding vertical category recognition model to obtain the corresponding response result; in response to the recognition result not matching the vertical category recognition model corresponding to the first entry mode, the recognition result is input into the corresponding diversion decision model to determine the corresponding vertical category recognition model; the recognition result is input into the determined vertical category recognition model to obtain the corresponding response result.

[0016] The image shooting interaction method provided in the present application is applied to a shooting terminal that provides a multi-entry shooting interface, which aggregates scattered functional entrances into a unified multi-entry shooting interface, so that the photographer can realize diverse functional requirements without cutting out the interface, eliminating the sense of fragmentation in the shooting operation; in addition, different functional entry modes in the multi-entry shooting interface correspond to a variety of background model programs, which can distinguish the actual functional requirements of the photographer and divert the obtained images to the appropriate background model program for processing, so as to improve the accuracy and reliability of the response results.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of an implementation example of the image shooting interaction method provided by an embodiment of the present invention; Figure 2 A flowchart of another implementation example of the image shooting interaction method provided by an embodiment of the present invention; Figure 3 A schematic diagram of a fully connected network for diversion decision making in an image shooting interaction method is provided for an embodiment of the present invention; Figure 4 A schematic diagram of another fully connected network for diversion decision-making in the image shooting interaction method provided by an embodiment of the present invention; Figure 5 A schematic diagram of the structure of a photographing terminal provided by an embodiment of the present invention; Figure 6 A schematic diagram of a multi-entry shooting interface of a shooting terminal provided in an embodiment of the present invention.

[0021] Reference numerals: 130 - processor, 131 - memory, 132 - bus, 133 - communication interface. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0023] To facilitate understanding of this embodiment, the application scenarios and design concepts of the embodiments of the present application are briefly introduced below.

[0024] The existing interactive interface requires frequent switching of interfaces when interacting with different entrances, which causes the photographer to feel a strong sense of disconnection. It is inconvenient to develop and maintain separate diversion algorithms for multiple entrances, and it is difficult to dynamically adjust the best image analysis results according to the scene and feedback high-quality responses.

[0025] Based on this, an embodiment of the present application provides an image shooting interaction method and a shooting terminal.

[0026] Example 1 The present application provides an image shooting interaction method, which is applied to a shooting terminal; the shooting terminal provides a multi-entry shooting interface; a first entry mode is set in the multi-entry shooting interface, and the first entry mode has a corresponding vertical category response model, and the vertical category response model matches the corresponding image category.

[0027] Combination Figure 1 As shown, the method includes: S110, in response to a shooting instruction triggered by the first entry mode, acquiring a shot target image, and inputting the target image into a universal recognition model to determine a recognition result.

[0028] S120, in response to the recognition result matching the vertical category recognition model corresponding to the first entry mode, the recognition result is input into the corresponding vertical category recognition model to obtain a corresponding response result.

[0029] S130, in response to the recognition result not matching the vertical category recognition model corresponding to the first entry mode, the recognition result is input into the corresponding diversion decision model to determine the corresponding vertical category recognition model.

[0030] S140, input the recognition result into the determined vertical category recognition model to obtain a corresponding response result.

[0031] The image shooting interaction method provided in the present application is applied to a shooting terminal, which provides a multi-entry shooting interface, and uniformly maps scattered multi-source entry requests to a single aggregate page. Through single-page interaction, the jump steps are reduced, the operation continuity is improved, and the sense of disconnection of the photographer's operation switching is eliminated. It can not only efficiently process different types of images, but also provide personalized responses and suggestions based on the recognition results, improve service accuracy, and improve service accuracy.

[0032] It can be understood that there is no limit to the number of first entry patterns. When there are multiple first entry patterns, there is a one-to-one correspondence between the multiple first entry patterns and the multiple vertical category response models. Each vertical category response model identifies and responds to a certain specified category of images. Therefore, after the first target image is identified based on the general recognition model to obtain the recognition result in step S110, if the recognition result matches the current vertical category recognition model, step S120 further identifies and responds to the first target image through the vertical category recognition model; if the recognition result does not match the current vertical category recognition model, it cannot be further accurately identified and responded to. At this time, the first target image is diverted through the diversion decision model in step S130 to match it with a suitable vertical category recognition model, so that step S140 further analyzes and responds based on the determined vertical category recognition model to obtain a response result. In addition, when the photographer forgets to select the entry mode, or selects the wrong first entry mode, after the general model performs preliminary recognition to obtain the recognition result, the recognition result is matched with the vertical category recognition model corresponding to the first entry mode. If there is no match, the recognition result is input into the diversion decision model to determine the corresponding vertical category recognition model for further recognition and response, thereby achieving automatic matching and correction, determining the appropriate vertical category recognition model for recognition and response, and improving accuracy and user experience.

[0033] In combination with the second aspect, the multi-entry shooting interface is further provided with a second entry mode. Figure 2 As shown, the method also includes: S210, in response to a shooting instruction triggered by the second entry mode, acquiring a shot second target image, and inputting the second target image into a universal recognition model to determine a recognition result.

[0034] S220: Input the recognition result into a corresponding diversion decision model to determine a corresponding vertical category recognition model.

[0035] S230, inputting the recognition result into the determined vertical category recognition model to obtain a corresponding response result.

[0036] In this embodiment, the multi-entry aggregation interface is provided with at least two types of entry modes, one or more first entry modes and one second entry mode, wherein each first entry mode corresponds to a vertical category recognition model; and the first entry mode also corresponds to a diversion decision model, which is used to divert the recognition result determined by the general recognition model to the vertical category recognition model that matches the image category of the second target image, so as to perform recognition and response, and obtain the corresponding response result. Among them, as an implementable method, the vertical category recognition model is a large language model, and at this time, recognition and response are both completed by the vertical category recognition model; as another implementable method, the vertical category recognition model can only complete the recognition, and the response is handed over to the subsequent large language model to complete. This is only an example and is not limited.

[0037] It can be understood that the number of first-type entry modes is not limited, and each first-type entry mode corresponds to a vertical classification recognition model, which can be adjusted and configured according to actual application requirements and is not limited here.

[0038] In this embodiment, there are three first-type entry modes, please refer to Figure 6 , the three first-type entry modes are named: "Photographing Ingredients", "AI Facial Diagnosis", and "AI Tongue Diagnosis" respectively; the second-type entry mode is named "Recognize Everything". The image captured in the "Recognize Everything" mode is the "second target image", and preliminary image recognition and analysis are performed through the general recognition model to provide preliminary recognition results, and the corresponding recognition results are dynamically diverted to the corresponding vertical recognition models in "Photographing Ingredients", "AI Facial Diagnosis", and "AI Tongue Diagnosis" through the diversion decision model for fine-grained recognition, analysis and response, so as to provide in-depth analysis in specific fields and obtain response results. Among them, the general recognition model has better versatility and can recognize images of various categories, but its recognition accuracy and granularity are not as good as the vertical recognition model that specializes in a certain image category.

[0039] Exemplarily, the general recognition model is used for preliminary and extensive classification and feature extraction of input images. It can identify the main object categories and basic features in the image, and can process images of multiple categories to provide preliminary classification results, but the accuracy may be limited. For example, there is a carrot on the table, and the recognition result obtained by the general recognition model may be: the image contains "food" or "vegetables". The vertical category recognition model focuses on the recognition of specific categories of images, providing more detailed and professional analysis. It can combine domain knowledge for in-depth analysis, with high accuracy and high specialization. Combining the above examples, the recognition result after identification by the vertical category recognition model may be: This is a fresh carrot, rich in vitamin A, suitable for cold or stir-frying.

[0040] It is understandable that different image recognition tasks require different processing methods and expertise. Through the preliminary classification of the general recognition module and the diversion of the diversion decision model, the most suitable vertical category recognition model can be quickly determined, reducing unnecessary calculations and time waste.

[0041] When the first type of entry mode ("Photographing Ingredients", "AI Facial Diagnosis", "AI Tongue Diagnosis") is triggered, the first target image obtained by the capture is subjected to preliminary image recognition and analysis by the general recognition model, and the recognition result is matched with the image category that can be recognized by the vertical recognition model. If there is a match, the current vertical recognition model can be used for recognition and response; if there is a mismatch, it is necessary to re-divert and determine the matching vertical recognition model, so as to perform targeted recognition and response for different categories of the first target image captured, which will help to obtain more fine-grained recognition results and provide strong data support for subsequent responses.

[0042] In combination with the first aspect, the second entry mode is a shooting and recognition diversion mode, and the diversion decision model corresponding to the second entry mode includes: a sequentially connected multimodal feature extraction module and a diversion decision fully connected network.

[0043] Among them, the multimodal feature extraction module is used to extract features from the image to be processed and the input text, and output an image feature vector (marked as "v" in this application).

[0044] The diversion decision fully connected network performs data processing on the input feature vector to determine the target second type entry mode.

[0045] Among them, the second entry mode is the shooting recognition diversion mode, and the second entry mode corresponds to the general recognition model and the diversion decision model; the diversion decision fully connected network includes: an input layer, a fully connected layer, and an output layer. A Softmax layer or a Sigmoid layer is also arranged between the fully connected layer and the output layer.

[0046] Please refer to Figure 3 An exemplary structure of a diversion decision fully connected network is proposed as an implementable method. The diversion decision fully connected network includes two cascaded fully connected layers, and finally outputs a probability value in the range of (0, 1) through the Softmax layer to map the feature H (the feature H can be an image feature vector v, or it can be a fusion of the image feature v and other feature vectors such as text feature vectors, speech feature vectors, etc.) to the diversion weights P of multiple different algorithm channels, so that the optimal algorithm channel can be determined. The "algorithm channel" refers to the vertical category recognition model. Among them, a ReLU activation function is set between the two fully connected layers to introduce nonlinear features.

[0047] Among them, multiple vertical classification recognition models correspond to multiple algorithm channels one by one. Combined with the above example, P=[P face , P tongue , P food ], where P face Corresponding to the algorithm channel of "AI face diagnosis", P tongue Corresponding to the algorithm channel of "AI tongue diagnosis", P food Corresponding to the algorithm channel of "Photographing Ingredients". For example, if P=[0.7,0.2,0.1], it means that there is a 70% probability of entering "AI Face Diagnosis", a 20% probability of entering AI Tongue Diagnosis, and a 10% probability of entering Ingredient Identification. At this time, the vertical recognition model corresponding to the first entry mode of "AI Face Diagnosis" is used for image recognition and response.

[0048] Understandably, Figure 4 A fully connected network structure for performing two tasks at the same time is provided for exemplary purposes only. The input layer is connected to a shared hidden layer, which is connected to the fully connected layer of task 1 and the fully connected layer of task 2. The fully connected layer of task 1 uses parameters W2⁽¹⁾ and b2⁽¹⁾, and the output is processed by the Softmax layer to obtain the probability P1 of task 1; the fully connected layer of task 2 uses parameters W2⁽²⁾ and b2⁽²⁾, and the output is processed by the Sigmoid layer to obtain the score S of task 2. In actual application, the number of tasks can be adjusted according to actual needs. This is only an example and is not limited. In this way, it is possible to recognize and respond to multiple images taken and / or uploaded.

[0049] In combination with the first aspect, the multi-entry shooting interface is further provided with a third entry mode. Figure 5 As shown, the method also includes: S310, in response to an upload instruction triggered by the third entry mode, obtaining the uploaded target image, and inputting the target image into a universal recognition model to determine a recognition result.

[0050] S320: Input the recognition result into a corresponding diversion decision model to determine a corresponding vertical category recognition model.

[0051] S330, input the recognition result into the determined vertical category recognition model to obtain a corresponding response result.

[0052] In this embodiment, the multi-entry shooting interface is also provided with a third entry mode. Figure 6 As shown, a third entry mode of "Upload Image" is set in the upper left corner of the multi-entry shooting interface, through which the picture is uploaded to obtain a third target image.

[0053] There are several situations at this point: In case 1, the photographer did not choose to trigger the first entry mode and the second entry mode, and the image upload operation was performed using the entry mode used last time or the entry mode set initially. If the entry mode used last time or the entry mode set initially was the first entry mode (combined with the above example, such as "AI Face-to-Face Consultation"), after obtaining the uploaded third target image, the third target image is initially identified by a general recognition model and the recognition result is matched with the vertical category recognition model of the first entry mode. If the match fails, step S320 is executed to re-divert, determine the appropriate vertical category recognition model, and then identify and respond.

[0054] In the second case, the photographer did not choose to trigger the first entry mode and the second entry mode, and the image upload operation was performed using the entry mode used last time or the entry mode set initially. If the entry mode used last time or the entry mode set initially is the first entry mode (combined with the above example, such as "AI Face Consultation"), after obtaining the uploaded third target image, the third target image is initially identified by the general recognition model and the recognition result is matched with the vertical category recognition model of the first entry mode. If the match is successful, step S330 is executed for further recognition and response by the vertical category recognition model.

[0055] Case 3: The photographer does not choose to trigger the first entry mode and the second entry mode, and performs the image upload operation using the entry mode used last time or the entry mode set initially. If the entry mode used last time or the entry mode set initially is the second entry mode (combined with the above example, such as "Knowing Everything"), after obtaining the uploaded third target image and performing preliminary recognition through the universal recognition model to obtain the recognition result, execute step S320 to perform the diversion processing to determine the appropriate vertical recognition model for recognition and response.

[0056] Case 4: The photographer has chosen to trigger a first entry mode, and the image is uploaded in the entry mode selected by the photographer. After obtaining the uploaded third target image, the recognition result obtained by preliminary recognition of the third target image by the universal recognition model is matched with the vertical category recognition model of the first entry mode. If the match fails, step S320 needs to be re-diverted to determine the appropriate vertical category recognition model for recognition and response. In this way, automatic error correction can be achieved to avoid the phenomenon that the uploaded image does not match the selected entry mode.

[0057] In case 5, the photographer has chosen to trigger a first entry mode, and the image is uploaded in the entry mode selected by the photographer. After obtaining the uploaded third target image, the third target image is initially recognized by the universal recognition model and the recognition result is matched with the vertical category recognition model of the first entry mode. If the match is successful, step S330 is executed to further recognize and respond by the vertical category recognition model.

[0058] Case six, if the entry mode is the second entry mode (combined with the above example, such as "know everything"), after the uploaded third target image is preliminarily recognized by the general recognition model to obtain the recognition result, execute step S320 to divert it to determine the appropriate vertical recognition model for recognition and response.

[0059] In combination with the first aspect, the recognition result includes the recognized image features and description text corresponding to the image features, and the response result is generated by a large language model.

[0060] It can be understood that the general recognition model can recognize basic features in the image, such as objects, text, scenes (such as natural, urban, indoor and other environmental types, activities, behaviors, time, weather, atmosphere, etc.), and image categories (such as faces, food, buildings, scenery, etc.), as well as descriptions corresponding to image features (i.e., descriptive text). For example, the image feature may be a "red round object", and the corresponding descriptive text may be "a red round object appears in the center of the picture."

[0061] The response results generated by the large language model refer to the use of large-scale pre-trained language models (such as GPT, BERT, etc.) to generate natural language descriptions or answers based on the input text or image recognition results. These models can generate fluent, coherent and logical text content by learning a large amount of text data.

[0062] Assuming the input image is a photo of a kitchen, which contains some ingredients and kitchen utensils, the general recognition model recognizes the following features: Ingredients: tomatoes, eggs, onions; Kitchen utensils: frying pan, spatula; Scene: Kitchen countertop.

[0063] The response generated by the large language model might be: "This photo shows a kitchen countertop with some common cooking ingredients and tools on it. You can see fresh tomatoes, eggs and onions, as well as a frying pan and a spatula next to it. It looks like you're ready to make a delicious home-cooked meal. If you need specific recipe suggestions or cooking tips, please let me know."

[0064] This generated response not only describes the specific content in the image, but also constitutes multimodal information together with the original image. This multimodal information combines visual data (image) and language data (description text) to more comprehensively reflect the image content. For example, the image itself may only show one device, while the generated description text can supplement the device's operating status, the possibility of failure, and other information, and further provide useful information or suggestions. In subsequent processing, multimodal information can be used for vertical category recognition (i.e., deep recognition for specific fields) to provide more comprehensive and in-depth services and enhance the photographer's experience.

[0065] In combination with the first aspect, a fourth entry mode is further provided in the multi-entry shooting interface, and the method further includes: S410, in response to a session instruction triggered by the fourth entry mode, obtaining corresponding session content, wherein the session content is used to be input into the general recognition model to determine the recognition result, and / or is used to be input into the vertical category recognition model to obtain a corresponding response result.

[0066] A fourth entry mode is also provided in the multi-entry shooting interface. When the photographer inputs a conversation instruction through the fourth entry mode in the multi-entry shooting interface and clicks "Send" in the fourth entry mode (combined with Figure 6 As shown in the figure, the system will obtain the corresponding conversation content. The "conversation content" here can be the photographer's voice input (such as "Please identify the operating status of the appliance in the image"), text input (such as "Analyze the problem in this picture"), or other forms of interactive information (such as gestures or button clicks).

[0067] The acquired conversation content is first input into the general recognition model to determine the preliminary recognition result. The general recognition model can process various types of input and extract key features or intents.

[0068] As needed, the conversation content can also be directly or indirectly input into a specific vertical recognition model to obtain a more professional response result. If the recognition result of the general recognition model matches a vertical recognition model, the result can be directly passed to the vertical recognition model; if it does not match, the diversion decision model can be used to decide which vertical model to use.

[0069] As an example, the user took a photo of a kitchen with ingredients and kitchen utensils through the first entry mode. The system identified the objects in the image and generated a description: "This is a photo of a kitchen with fresh tomatoes, eggs and onions, with a frying pan and a spatula next to it.

[0070] The photographer inputs a voice message through the fourth entry mode: "I need to make a simple home-cooked dish, please recommend a recipe." The system converts the voice message into text and inputs it into the general recognition model, recognizing that the photographer's intention is to find a recipe. The system then passes this request to a specialized vertical recognition model, which ultimately generates a response: "Based on the ingredients you provided, it is recommended that you try making scrambled eggs with tomatoes. This is a simple and delicious home-cooked dish.

[0071] In this way, the system can not only handle image recognition tasks, but also process text- or voice-based conversational instructions, providing richer and more flexible services.

[0072] In combination with the first aspect, the first entry mode is a shooting and recognition face mode, the vertical recognition model corresponding to the first entry mode is a face recognition model, and the corresponding image category is a face image; the recognition results include facial indicator information and skin quality indicator information, the facial indicator information includes at least one of nasolabial folds, dark circles, and skin color grading; the skin quality indicator information includes at least one of spots, acne marks, and blackheads.

[0073] In step S120, the recognition result is input into the corresponding vertical category recognition model to obtain the corresponding response result, which specifically includes: S121, input facial index information and skin index information into a face recognition model, and output a first response result; the first response result includes at least one of the photographer's physical health status, physical indicators, and improvement suggestions; wherein the face recognition model is trained based on a fatigue degree algorithm.

[0074] In this embodiment, after the photographer takes or uploads a face image, the general recognition model performs preliminary recognition to extract relevant features of the face and skin quality. Then, under the condition that the face image matches the currently selected or diverted face recognition model, the fatigue degree algorithm is used to perform recognition and generate a first response result.

[0075] Among them, facial indicator information includes but is not limited to the position and shape of facial feature points such as nasolabial folds, dark circles, eyes, eyebrows, mouth, nose, etc.; skin quality indicator information includes but is not limited to skin color, texture, glossiness, etc., as well as possible blemishes (such as spots, acne, dark circles, etc.).

[0076] In this embodiment, the face recognition model is trained based on the fatigue degree algorithm, which means that the model pays special attention to facial features related to fatigue during the training process, such as: the degree of eye opening and closing, the degree of dark circles, the relaxation of facial muscles, changes in skin gloss, etc.

[0077] In this embodiment, the first response result includes at least one of the photographer's physical health status, physical indicators, and improvement suggestions. Among them, based on changes in facial features, the face recognition model can evaluate the overall health of the photographer. For example, frequent dark circles may indicate lack of sleep or excessive stress; the face recognition model can also estimate some basic physical indicators, such as fatigue index, stress level, etc. This is usually achieved by analyzing facial features and skin changes. Furthermore, based on the evaluation results, the face recognition model can also provide personalized improvement suggestions. For example: if obvious dark circles are detected, it is recommended that the photographer ensure adequate sleep time; if dry skin is found, it is recommended to drink more water and use moisturizing skin care products; if signs of fatigue are detected, it is recommended to take proper rest and relax.

[0078] As an example, the first response may be: "Based on the analysis of your facial features, you may not have been getting enough sleep recently. It is recommended that you ensure 7-8 hours of adequate sleep every day, and pay attention to a balanced diet and appropriate exercise."

[0079] Furthermore, by combining the natural language processing capabilities of the large language model, feature scenes are extracted from facial images, and function cards are pushed based on the feature scenes; the function cards can be recommended recipes, health advice, shopping recommendations, etc. For example, when the feature scene is the kitchen, the system can push recipes that match the current health status, push methods for rest and relaxation based on facial fatigue signs, etc.

[0080] In combination with the first aspect, the first entry mode is a photographing and identifying tongue coating mode, the vertical category recognition model corresponding to the first entry mode is a tongue coating recognition model, and the corresponding image category is a tongue coating image; the recognition result includes tongue coating indicator information.

[0081] In step S120, the recognition result is input into the corresponding vertical category recognition model to obtain the corresponding response result, which specifically includes: S122, input the tongue coating index information into the tongue coating recognition model, and output a second response result; the second response result includes at least one of the photographer's physical index, body index, physical health status, and improvement suggestions; wherein the tongue coating recognition model is trained based on a health level algorithm.

[0082] In this embodiment, after the photographer takes or uploads the tongue coating image, it is initially identified by the general recognition model to extract the main features in the tongue coating image, namely the tongue coating index information, including the shape, color, texture, and vascular status of the tongue coating. Subsequently, under the condition that the tongue coating image matches the tongue coating recognition model currently selected or diverted, these features are input into a special tongue coating recognition model, which is trained based on the health degree algorithm, and the second response result generated may be: "Based on the analysis of your tongue coating color and vascular status, you may have damp heat in your body. It is recommended that you drink more water, eat a light diet, and avoid spicy food. In addition, proper exercise can also help regulate your physical condition."

[0083] The tongue coating recognition model is trained based on the health degree algorithm. This means that the model pays special attention to health-related tongue coating features during the training process, for example: different tongue coating colors may reflect different physical conditions (for example, yellow tongue coating may indicate heat in the body, and white tongue coating may indicate cold and dampness); the color and shape of the sublingual veins can reflect the blood circulation (for example, vasodilation may indicate inflammation or poor blood circulation in the body). Based on the above characteristics, the data processing of the health degree algorithm outputs the second response result. In this embodiment, the second response result includes at least one of the physical index, physical index, physical health status, and improvement suggestions of the shooter. Among them, based on the changes in the tongue coating features, the overall health status of the shooter can be evaluated. For example, the frequent appearance of yellow tongue coating may indicate heat in the body; the color and shape of the tongue coating can be used to evaluate the shooter's physical type, such as qi deficiency, blood stasis, etc., and can also be used to evaluate physical indicators, such as body temperature, blood pressure, etc. Further, based on the evaluation results, the model can provide personalized improvement suggestions. For example: If a yellow tongue coating is detected, the photographer is advised to drink more water, eat a light diet, and avoid spicy food; if dilated sublingual veins are found, the photographer is advised to do appropriate exercise to promote blood circulation; if a thick and greasy tongue coating is detected, the photographer is advised to regulate his stomach and avoid greasy food.

[0084] In combination with the first aspect, the first entry mode is a shooting and identifying food mode, the vertical category recognition model corresponding to the first entry mode is a food recognition model, and the corresponding image category is a food image; the recognition result includes at least food information.

[0085] In step S120, the recognition result is input into the corresponding vertical category recognition model to obtain the corresponding response result, which specifically includes: S123, in response to the category information of the ingredient information matching the sub-category in the preset ingredient name list, inputting the ingredient information into the ingredient recognition model, and outputting a third ingredient response result; wherein the ingredient recognition model is trained based on the g-Dino detection method and the ROI area extraction method.

[0086] In this embodiment, after the photographer takes or uploads the food image, the general recognition model performs preliminary recognition, and extracts the main features in the food image, namely the food information, including the shape, color, texture, etc. of the food. Subsequently, under the condition that the food image matches the food recognition model currently selected or diverted and the category information of the food image matches the sub-category in the food name list, these features are input into a special food recognition model, which is trained based on the g-Dino detection method and the ROI area extraction method, and the second response result generated may be: "The picture you took contains the following ingredients: tomatoes, beef and onions. It is recommended that you try to make a tomato and beef stew, which is a simple and delicious home-cooked dish. In addition, be sure to store the onions in a ventilated and dry place and freeze the remaining beef to extend its shelf life."

[0087] Among them, the g-Dino detection method is an advanced target detection algorithm that can accurately locate and identify food objects in images; the ROI region extraction method is used to extract the region of interest (Region of Interest), that is, the specific location and range of the food, to improve the recognition accuracy. According to the analysis results of the food recognition model, the system outputs the third response result, which at least includes: food name, food type (further subdivided food types, such as vegetables, fruits, meat, etc.), quantity or weight, recommended use (providing cooking suggestions or recipe recommendations based on food information), and storage suggestions (providing the best storage method and shelf life suggestions based on food characteristics).

[0088] As an example, the photographer took an image of food ingredients containing tomatoes, eggs and onions. The general recognition model preliminarily processes the image and extracts the main features, such as the color of tomatoes, the shape of eggs, the texture of onions, etc. At this point, the food image is matched with the food vertical category recognition model, and the above features (i.e., recognition results) recognized by the general recognition model are input into a special food recognition model, which is trained based on the g-Dino detection method and the ROI region extraction method. The generated third response result may be: "The picture you took contains the following ingredients: tomatoes, eggs and onions. It is recommended that you try to make scrambled eggs with tomatoes, which is a simple and delicious home-cooked dish. In addition, be sure to store onions in a ventilated and dry place to extend their shelf life.

[0089] In a second aspect, the present application provides a shooting terminal, combined with Figure 5 As shown, the photographing terminal includes a memory 131 and a processor 130, the memory 131 is used to store computer programs, and the processor 130 runs the computer programs to enable the photographing terminal to execute the above method.

[0090] Furthermore, combined with Figure 5The shooting terminal shown also includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .

[0091] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0092] The processor 130 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 130. The above processor 130 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the above embodiment in combination with its hardware.

[0093] In this embodiment, the shooting terminal provides a multi-entry shooting interface; a first entry mode is set in the multi-entry shooting interface, and the first entry mode has a corresponding vertical category response model, and the vertical category response model matches the corresponding image category.

[0094] In this way, the scattered functional entrances are aggregated into a unified multi-entry shooting interface, so that the photographer can realize diverse functional needs without cutting out the interface, eliminating the sense of fragmentation in the shooting operation; in addition, different functional entrance modes in the multi-entry shooting interface correspond to a variety of background model programs, which can distinguish the actual functional needs of the photographer and divert the obtained images to the appropriate background model program for processing, so as to improve the accuracy and reliability of the response results.

[0095] In a fourth aspect, an embodiment of the present application provides a readable storage medium, in which computer program instructions are stored. When the computer program instructions are read and executed by a processor, the above method is executed.

[0096] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0097] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0098] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0099] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0100] Finally, it should be noted that the above embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions recorded in the above embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An image shooting interactive method, characterized in that: The method is applied to a shooting terminal; the shooting terminal provides a multi-entry shooting interface; a first entry mode is set in the multi-entry shooting interface, the first entry mode has a vertical category response model corresponding thereto, and the vertical category response model matches the corresponding image category; The method comprises: In response to a shooting instruction triggered by the first entry mode, acquiring a first target image shot, and inputting the first target image into a universal recognition model to determine a recognition result; In response to the recognition result matching the vertical category recognition model corresponding to the first entry mode, the recognition result is input into the corresponding vertical category recognition model to obtain a corresponding response result; In response to the recognition result not matching the vertical category recognition model corresponding to the first entry mode, inputting the recognition result into a corresponding diversion decision model to determine a corresponding vertical category recognition model; The recognition result is input into the determined vertical category recognition model to obtain a corresponding response result.

2. The method according to claim 1, characterized in that The multi-entry shooting interface is further provided with a second entry mode, and the method further comprises: In response to a capture instruction triggered by the second entry mode, acquiring a captured second target image, and inputting the second target image into a universal recognition model to determine a recognition result; Inputting the recognition result into the corresponding diversion decision model to determine the corresponding vertical category recognition model; The recognition result is input into the determined vertical category recognition model to obtain a corresponding response result.

3. The method according to claim 1, characterized in that The multi-entry shooting interface is further provided with a third entry mode, and the method further comprises: In response to an upload instruction triggered by the third entry mode, acquiring the uploaded third target image, and inputting the third target image into a universal recognition model to determine a recognition result; Inputting the recognition result into the corresponding diversion decision model to determine the corresponding vertical category recognition model; The recognition result is input into the determined vertical category recognition model to obtain a corresponding response result.

4. The method according to any one of claims 1 to 3, characterized in that: The recognition result includes the recognized image features and description text corresponding to the image features, and the response result is generated by a large language model.

5. The method according to claim 4, characterized in that The multi-entry shooting interface is further provided with a fourth entry mode, and the method further comprises: In response to a session instruction triggered by the fourth entry mode, corresponding session content is obtained, and the session content is used to input into the general recognition model to determine the recognition result, and / or to input into the vertical recognition model to obtain a corresponding response result.

6. The method according to claim 1 or 3, characterized in that: The first entry mode is a shooting and recognizing face mode, the vertical recognition model corresponding to the first entry mode is a face recognition model, and the corresponding image category is a face image; the recognition result includes facial index information and skin quality index information; The step of inputting the recognition result into the corresponding vertical category recognition model to obtain the corresponding response result includes: The facial index information and the skin quality index information are input into the face recognition model, and a first response result is output; the first response result includes at least one of the photographer's physical health status, physical indicators, and improvement suggestions; wherein the face recognition model is trained based on a fatigue degree algorithm.

7. The method according to claim 1 or 3, characterized in that: The first entry mode is a tongue coating recognition mode of photographing, the vertical category recognition model corresponding to the first entry mode is a tongue coating recognition model, and the corresponding image category is a tongue coating image; the recognition result includes tongue coating index information; The step of inputting the recognition result into the corresponding vertical category recognition model to obtain the corresponding response result includes: The tongue coating index information is input into the tongue coating recognition model, and a second response result is output; the second response result includes at least one of the physical index, body index, physical health status, and improvement suggestions of the photographer; wherein the tongue coating recognition model is trained based on a health level algorithm.

8. The method according to claim 1 or 3, characterized in that: The first entry mode is a shooting and identifying food mode, the vertical category recognition model corresponding to the first entry mode is a food recognition model, and the corresponding image category is a food image; the recognition result at least includes food information; The step of inputting the recognition result into the corresponding vertical category recognition model to obtain the corresponding response result includes: In response to the category information of the ingredient information matching the subcategories in a preset ingredient name list, the ingredient information is input into the ingredient recognition model, and a third ingredient response result is output; wherein the ingredient recognition model is trained based on the g-Dino detection method and the ROI area extraction method.

9. The method according to claim 2, characterized in that: The second entry mode is a shooting recognition diversion mode, and the second entry mode corresponds to the general recognition model and the diversion decision model; the diversion decision model includes: A multimodal feature extraction module is used to extract features from the image to be processed and output a feature vector; The diversion decision fully connected network is used to perform data processing on the input feature vector to determine the target vertical category response model; wherein, the diversion decision fully connected network includes an input layer, a fully connected layer, and an output layer, and a Softmax layer or a Sigmoid layer is also arranged between the fully connected layer and the output layer.

10. A shooting terminal, characterized in that: The photographing terminal includes a memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the photographing terminal to execute any one of claims 1 to 9.

Images