Skin detection method and device based on large language model

Abstract

The invention provides a skin detection method and device based on a large language model, and the method comprises the steps: obtaining at least one image obtained by shooting a to-be-detected skin region, inputting the at least one image into a multi-modal model, and enabling the multi-modal model to carry out the detection of the to-be-detected skin region based on the at least one image, generating a first description text corresponding to the skin area; acquiring an audio for describing the skin area, and performing voice recognition on the audio so as to convert the audio into a second description text corresponding to the skin area; and based on the first description text and the second description text, constructing a prompt text, inputting the prompt text into a large language model, and generating a skin detection result text corresponding to the skin area by the large language model under the guidance of the prompt text.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of artificial intelligence, and in particular to a skin detection method and device based on a large language model. BACKGROUND

[0002] A large language model (LLM) is a natural language processing model based on deep learning technology, which has strong language understanding and generation capabilities. Nowadays, the application of large language models is becoming more and more widespread. For example, intelligent dialogue systems based on large language models have a wide range of applications in customer service, legal assistance, online education, medical consultation, and many other areas. Intelligent dialogue systems are a specific application form of large language models for users, which aims to understand and answer questions posed by users in natural language form and can generate concise and clear answers. Specifically, intelligent dialogue systems are based on large language models, which are used by large language models to understand and answer questions posed by users and generate corresponding answers.

[0003] For intelligent dialogue systems in the medical field, the core large language model mainly acquires knowledge in the medical field through statistical learning of static medical corpus in the training process, and generates relevant responses using the learned knowledge in the subsequent reasoning process. However, the representation process of this knowledge is usually limited by the coverage, timeliness, and knowledge density of the corpus, so that the model is prone to knowledge blind spots or reasoning biases when facing complex, rare, or highly contextual clinical problems, and may even have hallucination problems, resulting in inaccurate and unreliable responses. Therefore, it limits the adaptability and response accuracy of intelligent dialogue systems in actual medical scenarios. SUMMARY

[0004] One or more embodiments of the present specification provide technical solutions as follows: According to a first aspect of one or more embodiments of the present specification, a skin detection method based on a large language model is provided, the method comprising: obtaining at least one image obtained by photographing a skin region to be detected, and inputting the at least one image into a multi-modal model, and generating a first description text corresponding to the skin region based on the at least one image by the multi-modal model; obtaining an audio for describing the skin region, and performing speech recognition on the audio to convert the audio into a second description text corresponding to the skin region; construct a prompt text based on the first description text and the second description text, and input the prompt text into a large language model, and generate, by the large language model under the guidance of the prompt text, a skin detection result text corresponding to the skin region.

[0005] According to a second aspect of one or more embodiments of the present specification, a skin detection device based on a large language model is provided, and the device comprises: an image description generation module configured to obtain at least one image taken by photographing a skin region to be detected, and input the at least one image into a multi-modal model, and generate, by the multi-modal model based on the at least one image, a first description text corresponding to the skin region; a speech recognition module configured to obtain an audio for describing the skin region, and perform speech recognition on the audio to convert the audio into a second description text corresponding to the skin region; a detection result generation module configured to construct a prompt text based on the first description text and the second description text, and input the prompt text into a large language model, and generate, by the large language model under the guidance of the prompt text, a skin detection result text corresponding to the skin region.

[0006] According to a third aspect of one or more embodiments of the present specification, an electronic device is provided, and the device comprises: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of the above embodiments by running the executable instructions.

[0007] According to a fourth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, and the medium stores computer instructions which, when executed by a processor, implement the steps of the method according to any one of the above embodiments.

[0008] According to a fifth aspect of one or more embodiments of the present specification, a computer program product is provided, and the product comprises computer program / instructions which, when executed by a processor, implement the steps of the method according to any one of the above embodiments.

[0009] As can be known from the above embodiments, for a skin region to be detected, on the one hand, at least one image obtained by photographing the skin region can be acquired, and the at least one image is input into a multi-modal model, and the multi-modal model generates a first description text corresponding to the skin region based on the at least one image; on the other hand, an audio for describing the skin region can be acquired, and speech recognition is performed on the audio to convert the audio into a second description text corresponding to the skin region, so that a prompt text can be constructed based on the first description text and the second description text, and the prompt text is input into a large language model, and the large language model generates a skin detection result text corresponding to the skin region under the guidance of the prompt text.

[0010] In this way, the multi-modal data corresponding to the skin region to be detected can be integrated as the basis for the large language model to generate the skin detection result corresponding to the skin region through reasoning, which can improve the correctness and reliability of the skin detection result generated by the large language model, thereby improving the adaptability and response accuracy of the intelligent dialogue system based on the large language model in skin detection. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 FIG. 1 is a schematic diagram of an intelligent dialogue system provided by an example embodiment.

[0012] Figure 2 FIG. 2 is a flowchart of a skin detection method based on a large language model provided by an example embodiment.

[0013] Figure 3 FIG. 3 is a schematic diagram of a compilation and instrumentation process provided by an example embodiment.

[0014] Figure 4 FIG. 4 is a structural schematic diagram of a device provided by an example embodiment.

[0015] Figure 5 FIG. 5 is a block diagram of a skin detection apparatus based on a large language model provided by an example embodiment. DETAILED DESCRIPTION

[0016] In order to enable persons skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described below in conjunction with the drawings in the specification. Obviously, the described embodiments are only some of the embodiments of the specification, not all the embodiments. Based on the embodiments in the specification, all other embodiments obtained by persons skilled in the art without creative labor should be within the protection scope of the specification.

[0017] The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the specification are information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0018] Intelligent dialogue systems based on large language models have a wide range of applications in customer service, legal assistance, online education, medical consultation and many other fields. Intelligent dialogue systems are interactive systems developed using artificial intelligence technology, which provide services to users by simulating human communication. For example, in the medical field, intelligent dialogue systems have become a key tool for improving the quality and efficiency of medical services, providing users with medical-related dialogue, question and answer, query and other services, such as disease diagnosis, treatment recommendations, drug instructions, etc.

[0019] Intelligent dialogue systems are a specific application form of large language models for users, aiming to understand and answer questions posed by users in natural language form and generate concise and clear answers. Specifically, intelligent dialogue systems are based on large language models, which are used to understand and answer questions posed by users and generate corresponding answers.

[0020] Large language models are natural language processing models based on deep learning technology, with strong language understanding and generation capabilities. Large language models are usually deep learning models trained on large amounts of text data and can be used to understand the meaning of natural language text or generate natural language text. Large language models can handle a variety of natural language tasks, such as text classification, named entity recognition (NER), question and answer, dialogue, etc., and are an important way to artificial intelligence.

[0021] In the field of natural language processing (NLP), large-scale text data sets are usually referred to as corpora. Corpora can contain various types of text data, such as literary works, academic papers, legal documents, news reports, daily conversations, emails, online forum posts, etc. By learning the text data in the corpus, large language models can acquire and understand the rules and patterns of natural language, and then effectively process and generate human language.

[0022] Large language models are usually based on the Transformer architecture, i.e., large language models are usually deep learning models based on the Transformer architecture. Deep learning models based on the Transformer architecture are a class of neural network models that use the Transformer architecture, which have shown excellent performance in natural language processing and other fields.

[0023] The Transformer is a neural network model for sequence-to-sequence modeling. The Transformer does not rely on recursive structures and can parallelize training and inference, speeding up model processing. In deep learning models based on the Transformer architecture, a multi-layer Transformer encoder is usually used to extract features from input sequences, and a Transformer decoder is used to convert the extracted features into output sequences. At the same time, such models usually use self-attention mechanisms to capture long-distance dependencies in input sequences, and use residual connections and normalization methods to speed up training and improve model performance.

[0024] Pre-trained models are large language models pre-trained on large-scale unlabeled text data. Pre-trained models are general models that are not designed and optimized for specific tasks. In order to adapt pre-trained models to specific application scenarios and task requirements, fine-tuning is needed to improve the performance of the model on specific tasks. The large language models eventually put into use are usually further fine-tuned based on pre-trained models, and are supervised learning models based on labeled text data. Pre-training and fine-tuning are complementary processes, pre-training enables the model to have broad language understanding capabilities, and fine-tuning makes the model more specialized and accurate on specific tasks.

[0025] That is, the training process of a large language model can be divided into two stages: pre-training and fine-tuning. In the pre-training stage, unsupervised learning (e.g., self-supervised learning) can be used to pre-train on large-scale, unlabeled text datasets (e.g., web encyclopedias, web articles, books, etc.). Specifically, the model can predict missing parts or the next word based on context, learn statistical rules and language structures such as semantics and syntax, and iteratively update model parameters by minimizing prediction loss through backpropagation and optimization algorithms (e.g., gradient descent) to gradually improve the model's understanding of language. In the fine-tuning stage, according to the specific application scenario and task requirements, a corresponding supervised learning task (e.g., text classification, named entity recognition, question-answering systems, dialogue systems, etc.) can be selected, and a task-specific text dataset can be prepared. Then, the pre-trained model can be used as the starting point for fine-tuning, and supervised learning can be used to fine-tune on the task-specific text dataset. Specifically, the model can perform the task based on the text dataset, and iteratively update model parameters by minimizing the loss that measures the model's performance on the specific task through backpropagation and optimization algorithms (e.g., gradient descent) to gradually improve the model's performance on the specific task. In practical applications, fine-tuning can flexibly choose supervised learning, unsupervised learning, or semi-supervised learning based on the specific application scenario and the type of available data.

[0026] The language understanding ability learned by a large language model in the pre-training and fine-tuning stages enables the large language model to understand, analyze, and synthesize text information, make logical deductions, perform knowledge reasoning, or solve problems when faced with complex problems or tasks. This ability is often referred to as the reasoning ability of a large language model.

[0027] In practical applications, a pre-trained large language model is often referred to as a base model of a large language model, and a fine-tuned large language model is often referred to as a service model of a large language model.

[0028] Large language models are usually prompted or primed with a prompt, which can be an initial text or text snippet provided to the model, such as a sentence, a question, a dialogue, etc., to guide or prime the model to generate corresponding output. The prompt is a key tool to guide the model output and can be very simple or designed quite complex, such as including instructions, examples, descriptions of expected output format, etc. Through the prompt, the large language model can be explicitly told what task it is expected to perform, such as answering a question, simulating a dialogue, writing an article, translating text, etc. At the same time, the prompt can provide the necessary background information and context for the large language model to understand the logic, style, theme or position that should be followed when generating content. In addition, the prompt can also stimulate the large language model to exhibit its inherent knowledge reserve or specific language ability, such as explaining complex concepts, citing regulations, imitating the writing style of a specific author, etc.

[0029] Since large language models are mainly used for text-based processing to understand and generate human language, prompts are usually in the form of text. However, in practical applications, large language models can also accept other forms of input as prompts, such as images, audio, and even video, provided that the large language model is designed or trained to handle multi-modal data (e.g., text, images, audio, video, etc.).

[0030] Many tasks in practical applications require the combination of different types of information to obtain the best solution. For example, in a medical scenario, a doctor may need to combine a patient's medical record text and medical images for judgment, and a patient may need to consult a self-help medical text based on their own medical images or medical examination report photos. Multimodal large language models (MLLM) can process and understand multi-modal data, combining technologies from natural language processing and computer vision, aiming to achieve deep fusion and understanding of multi-modal data, and can reason, generate and interact between different modalities.

[0031] For intelligent dialogue systems in the medical field, the core large language model mainly acquires knowledge in the medical field through statistical learning of static medical corpus in the training process, and generates relevant responses in the subsequent reasoning process using the learned knowledge. However, the representation process of these knowledge is usually limited by the coverage, timeliness and knowledge density of the corpus, so that the model is prone to knowledge blind spots or reasoning biases when facing complex, rare or highly contextual clinical problems, and may even have hallucination problems, resulting in inaccurate and unreliable generated responses. Therefore, it limits the adaptability and response accuracy of intelligent dialogue systems in actual medical scenarios.

[0032] One or more embodiments of the present specification provide a technical solution for implementing skin detection based on a large language model, in which, for a skin region to be detected, on the one hand, at least one image obtained by photographing the skin region can be acquired, and the at least one image is input into a multi-modal model, and the multi-modal model generates a first description text corresponding to the skin region based on the at least one image, on the other hand, an audio for describing the skin region can be acquired, and the audio is subjected to speech recognition to convert the audio into a second description text corresponding to the skin region, so that a prompt text can be constructed based on the first description text and the second description text, and the prompt text is input into a large language model, and the large language model generates a skin detection result text corresponding to the skin region under the guidance of the prompt text.

[0033] In the above manner, the multi-modal data corresponding to the skin region to be detected can be integrated as the basis for the large language model to generate a skin detection result corresponding to the skin region through reasoning, which can improve the correctness and reliability of the skin detection result generated by the large language model, thereby improving the adaptability and response accuracy of the intelligent dialogue system based on the large language model in skin detection.

[0034] Reference Figure 1 , Figure 1 is a schematic diagram of an intelligent dialogue system provided by an exemplary embodiment.

[0035] As Figure 1 shown, the above intelligent dialogue system can include a server and at least one client accessing the server through any type of wired or wireless network.

[0036] The above server can correspond to a server including a single independent physical host, or a server cluster composed of multiple independent physical hosts; or, can correspond to a virtual server, a cloud server, etc. carried by a host cluster.

[0037] The above client can correspond to a terminal device such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a PC (Personal Computer), a PDA (Personal Digital Assistant), a wearable device (for example, smart glasses, a smart watch, etc.), a smart in-vehicle device, or a game console, etc.

[0038] A user can use the intelligent dialogue service provided by the above intelligent dialogue system through the above client; the above client and the above server can realize the intelligent dialogue service for the user through data interaction between each other.

[0039] Specifically, the above-mentioned server can be equipped with a large language model, and the above-mentioned intelligent dialogue system can be based on the large language model, and the large language model can infer based on the query text (which can be referred to as Query or Question) to understand and answer the question raised by the query text, and generate the answer text (which can be referred to as Response or Answer) corresponding to the query text.

[0040] For example, the above-mentioned client can output a corresponding user interface to the user, so that the user can perform operations such as inputting query text, uploading documents or pictures for assisting in asking questions, etc. in the user interface to ask questions to the above-mentioned intelligent dialogue system using the intelligent dialogue service provided by the intelligent dialogue system. The client can send the query text input by the user to the above-mentioned server, and the server can generate the corresponding answer text for the query text and output the answer text to the user, that is, return the answer text to the client, and the client displays the answer text to the user through the user interface for the user to view, thereby realizing the intelligent dialogue service for the user.

[0041] It should be noted that the query text can be regarded as a special Prompt. The query text describes the specific problem that the user wants to solve, and the problem is usually expressed through a well-designed Prompt.

[0042] In order to improve the adaptability and response accuracy of the intelligent dialogue system, the RAG (Retrieval-Augmented Generation) method can be used to combine information retrieval and model generation, so that the above-mentioned intelligent dialogue system no longer relies only on the knowledge obtained by the large language model in the training process through learning static corpus when answering the question raised by the user, but can first perform information retrieval in a large-scale document set based on the question, and then understand and answer the question according to the relevant documents retrieved, and generate the corresponding answer. That is, the document set and the large language model can be combined, and in the model generation process, relevant information is retrieved from the document set in real time to assist the model to make more accurate, comprehensive answers or decisions. Since the model generation process takes into account the retrieved information and the context of the question, it can ensure that the generated content meets the actual needs and is accurate, reliable, coherent and natural.

[0043] Specifically, the above-mentioned service end can also carry a knowledge base and an information retrieval component. Among them, the knowledge base is an external knowledge base relative to the large language model carried on the service end, that is, the data in the knowledge base is not the knowledge obtained by the large language model in the training process through learning, but as auxiliary information in the reasoning process of the large language model, assisting the large language model to generate an answer text corresponding to the query text. In the reasoning process of the large language model, the information retrieval component can perform information retrieval in the knowledge base based on the query text to assist the large language model to generate an answer text corresponding to the query text through the retrieved relevant information.

[0044] It should be noted that the above-mentioned service end can also carry image description generation components, speech recognition components, prompt generation components, user interaction components, voice playback components and other functional components or functional subsystems. These components or subsystems can work cooperatively with the large language model carried on the service end to jointly realize the generation of an answer text corresponding to the query text.

[0045] In addition, the large language model in the above-mentioned intelligent dialogue system can have and only have one, which can be used to perform dialogue tasks or question and answer tasks, and can also be used to perform image description generation tasks, speech recognition tasks, information retrieval tasks, similarity calculation tasks, etc. according to actual needs. Or, the large language model in the intelligent dialogue system can have multiple, which can include large language models for performing dialogue tasks or question and answer tasks, as well as large language models for performing image description generation tasks, large language models for performing speech recognition tasks, large language models for performing information retrieval tasks, and large language models for performing similarity calculation tasks.

[0046] Reference Figure 2 , Figure 2 is a flowchart of a skin detection method based on a large language model provided by an exemplary embodiment. The skin detection method based on a large language model can be applied to an intelligent dialogue system as shown in Figure 1 .

[0047] As shown in Figure 2 , the above-mentioned skin detection method based on a large language model can include the following steps: Step 202: acquiring at least one image obtained by photographing a skin region to be detected, and inputting the at least one image into a multi-modal model, and generating a first description text corresponding to the skin region by the multi-modal model based on the at least one image.

[0048] In this embodiment, the skin area to be detected can be determined according to actual conditions and needs. For example, the skin area with abnormalities in the whole body skin of a person, such as a skin area with rashes or papules, a skin area with blisters, a skin area with scales, a skin area with scabs, a skin area with erosion or ulcer, a skin area with redness or blackening, etc., can be taken as the skin area to be detected.

[0049] For the skin area to be detected, at least one image obtained by photographing the skin area can be acquired.

[0050] In actual application, a user can use his own terminal device (for example, a smart phone, a tablet computer, etc.) to photograph the skin area to be detected through the camera mounted on the terminal device, and obtain at least one image corresponding to the skin area.

[0051] In some embodiments, since a video is a dynamic image formed by a series of continuous static images (i.e., frames) played quickly in a certain time sequence, at least one image obtained by photographing the skin area to be detected can be at least one video frame in a video obtained by video photographing the skin area. These video frames can be continuous video frames, video frames obtained by sampling the video at a specific frame sampling interval or frame sampling rate, or video frames obtained by randomly sampling the video. Alternatively, different images in at least one image obtained by photographing the skin area to be detected can be images obtained by photographing the skin area from different angles. This specification does not specially limit this.

[0052] It should be noted that generating the above-mentioned first description text corresponding to the skin area on the basis of multiple images or videos corresponding to the skin area to be detected can expand the perception range of the skin area and capture the dynamic changes of the skin area, thereby improving the correctness and reliability of the skin detection result text corresponding to the skin area generated based on the first description text.

[0053] In the case where at least one image obtained by photographing the skin area to be detected is acquired, the at least one image can be input into a multi-modal model for processing. Specifically, the multi-modal model can perform an image description generation (Image Captioning) task on the basis of the at least one image. That is, the multi-modal model can generate a description text (which can be referred to as a first description text) corresponding to the skin area based on the at least one image. The image description generation task is an artificial intelligence task combining computer vision and natural language processing technology, and its main goal is to generate a natural language text that can accurately describe the content of an image.

[0054] It should be noted that the core feature of the multi-modal model is to be able to process and understand two or more different modalities. In the task of generating text from images, the model needs to process the visual modality (input image) and the text modality (output text). That is, the model not only needs to achieve visual understanding, i.e., "understand" the image content (visual understanding), but also needs to achieve language generation, i.e., express the understood image content in natural language. This involves establishing a connection and conversion between different modalities, which is a key ability of multi-modal learning.

[0055] The multi-modal model described above for performing the image description generation task can be a trained multi-modal model. During the training of the multi-modal model, a plurality of training samples can be prepared first. A training sample can include an image and a description text related to the image; for example, a training sample can include an image containing a "cat sleeping" and a text describing "a cat sleeping". Subsequently, the multi-modal model can be trained based on these training samples, and the model parameters of the multi-modal model can be adjusted so that the loss (Loss) is as small as possible, thereby completing the training of the multi-modal model.

[0056] In order to make the multi-modal model described above have better model effect in the image description generation task of skin detection, a plurality of training samples including an image corresponding to a skin region and a description text corresponding to the skin region can be prepared, and the multi-modal model can be further trained based on these training samples.

[0057] In some embodiments, the multi-modal model described above can be a multi-modal large language model, or any machine learning model capable of processing and understanding two or more different modalities (for example: a dual-flow neural network specially designed for images and text), and the present specification does not make special limitations.

[0058] Step 204: Obtain audio for describing the skin region, and perform speech recognition on the audio to convert the audio into a second description text corresponding to the skin region.

[0059] In the present embodiment, for the skin region to be detected, not only can the first description text corresponding to the skin region be obtained through at least one image corresponding to the skin region, but another description text (which can be referred to as a second description text) corresponding to the skin region can also be obtained through audio corresponding to the skin region.

[0060] Specifically, an audio for describing the skin region to be detected can be acquired. The content of the audio can specifically be a description of the skin region to be detected itself, such as a description of a skin surface state, a description of a current feeling of the skin (e.g., itching, pain, etc.), and the like. The content of the audio can also be a description of relevant information for assisting in skin detection, such as a description of a duration of an abnormality of the skin region to be detected, or a description of diet, medication, personal behavior, and the like before the abnormality of the skin region to be detected occurs, and the like.

[0061] In actual application, a user can use his own terminal device to collect a voice signal of the user through a microphone mounted on the terminal device to form an audio. The voice content of the user can be a description of the skin region to be detected by the user.

[0062] It should be noted that the above-mentioned video corresponding to the skin region to be detected and the above-mentioned audio for describing the skin region can be acquired separately. That is, the skin region can be photographed by a camera to obtain a silent video (containing only images, not containing audio) corresponding to the skin region, and the audio for describing the skin region can be recorded by a microphone.

[0063] Alternatively, the above-mentioned video corresponding to the skin region to be detected and the above-mentioned audio for describing the skin region can be acquired simultaneously. That is, the skin region can be photographed by a camera and a microphone simultaneously, and the audio for describing the skin region can be recorded to obtain a sound video (containing both images and audio) corresponding to the skin region. In this case, not only the video can be extracted from the sound video as the video corresponding to the skin region, but also the audio can be extracted from the sound video as the audio for describing the skin region.

[0064] In the case where the audio for describing the skin region to be detected is acquired, speech recognition can be performed on the audio to convert the audio into the above-mentioned second description text corresponding to the skin region. That is, the voice content can be recognized from the audio, and the recognized voice content can be output in the form of text. For example, an automatic speech recognition (ASR) task can be performed on the basis of the audio to convert the audio into the second description text.

[0065] In some embodiments, in order to improve the efficiency and accuracy of speech recognition, a model can be used to implement speech recognition on the audio.

[0066] Specifically, the speech recognition model can be obtained in advance through model training. In the training process of the speech recognition model, a plurality of training samples can be prepared first. One training sample can include a piece of speech and an accurate, word-by-word corresponding text transcription. Subsequently, the speech recognition model can be trained based on these training samples, and the model parameters of the speech recognition model can be adjusted so that the loss is as small as possible, thereby completing the training of the speech recognition model.

[0067] In order to make the above-mentioned speech recognition model have better model effect on the speech recognition task of skin detection, a plurality of training samples including a piece of speech for describing a skin region and a transcription text of the speech can be prepared, and the speech recognition model can be further trained based on these training samples.

[0068] And in the case of obtaining audio for describing the skin region to be detected, the audio can be input into the trained speech recognition model, and the speech recognition model can be used to perform speech recognition on the audio to convert the audio into the above-mentioned second description text corresponding to the skin region.

[0069] Step 206: constructing a prompt text based on the first description text and the second description text, and inputting the prompt text into a large language model, so that the large language model generates a skin detection result text corresponding to the skin region under the guidance of the prompt text.

[0070] In the embodiment, in the case of obtaining the first description text and the second description text, the prompt text (i.e. Prompt) can be further constructed based on the first description text and the second description text. The constructed prompt text can include the first description text and the second description text, so that the prompt text can be used to guide the large language model to reason based on the first description text and the second description text, and generate a skin detection result text corresponding to the skin region to be detected.

[0071] That is, the above-mentioned prompt text can be input into the large language model, and the large language model can reason based on the first description text and the second description text in the prompt text under the guidance of the prompt text, and generate a skin detection result text corresponding to the above-mentioned skin region to be detected.

[0072] In actual application, the skin detection can be detection of whether a skin disease is present, detection of the health status of the skin, detection of the performance form of the skin, and other skin property-related detection.

[0073] It should be noted that the above large language model can refer to a service model of the large language model. In actual application, the large language model can be pre-trained on a large-scale, unlabeled text dataset in an unsupervised learning manner to obtain a base model of the large language model; then, the dialogue task or the question and answer task for implementing skin detection can be used as a supervised learning task during fine-tuning, and a text dataset specific to the dialogue task or the question and answer task for implementing skin detection is prepared, and then the base model of the large language model can be used as a starting point for fine-tuning, and the dialogue task or the question and answer task for implementing skin detection is fine-tuned on the text dataset specific to the dialogue task or the question and answer task for implementing skin detection in a supervised learning manner to obtain the service model of the large language model.

[0074] When constructing the text dataset specific to the dialogue task or the question and answer task for implementing skin detection, the collected query texts for proposing skin detection related questions and including description texts corresponding to skin regions (these query texts are training samples) can be labeled, and skin detection result texts corresponding to the skin regions are labeled for each query text (the labeled skin detection result texts are labels of the training samples). In this way, the labeled query texts can be used as the text dataset specific to the dialogue task or the question and answer task for implementing skin detection, and used for supervised training of the large language model.

[0075] In some embodiments, in order to provide more context information for the large language model, thereby improving the correctness and reliability of the skin detection result text generated by the large language model, at least one image obtained by photographing the skin region to be detected can be input into the skin detection model, and the skin detection model can detect the skin characteristics corresponding to the skin region based on the at least one image.

[0076] For example, the classification model can be selected as the skin detection model. In this case, the collected image groups (including at least one image) corresponding to the skin region can be labeled, and the skin characteristics corresponding to the skin region are labeled for each image group. In this way, the labeled image groups can be used as labeled samples for supervised training of the skin detection model. After the training of the skin detection model is completed, the skin detection model can predict the probability of various skin characteristics of the skin region based on the input at least one image corresponding to the skin region to be detected, so as to determine the skin characteristic with the highest probability as the skin characteristic corresponding to the skin region.

[0077] In a case where the skin characteristics corresponding to the skin region are obtained by using the skin detection model, the first description text and the second description text can be combined, and the skin detection result text corresponding to the skin region can be generated by guiding the large language model to reason based on the first description text, the second description text, and the skin characteristics in text form.

[0078] In some embodiments, in a case where the skin detection result text is obtained by using the large language model, the skin detection result text can be directly output to the user in text form, so that the user can view the skin detection result in text form. Alternatively, the skin detection result text can be converted into audio, for example, automatically read aloud, and output to the user, so that the user can hear the skin detection result in the form of voice.

[0079] In actual applications, the skin detection result text can also be output to the user in text form and in the form of voice, so that the user can receive the skin detection result in text form and in the form of voice at the same time.

[0080] In some embodiments, in order to guide the user to provide more information helpful for skin detection to the large language model, thereby improving the correctness and reliability of the skin detection result text generated by the large language model, when the prompt text is constructed based on the first description text and the second description text, the constructed prompt text can be used to guide the large language model to generate question text for obtaining skin detection related information corresponding to the skin region to be detected. At this time, the first description text and the second description text can also be included in the prompt text, so that the large language model can refer to the first description text and the second description text to generate question text for obtaining skin detection related information corresponding to the skin region.

[0081] That is, the prompt text can be input into the large language model, and the large language model can generate question text for obtaining skin detection related information corresponding to the skin region to be detected under the guidance of the prompt text.

[0082] For example, assuming that the first description text is “Part: left cheek of face; Performance: erythema, mild desquamation; Preliminary judgment: suspected eczema (confidence 78%); Malignant risk: low.”, and the second description text is “It itches a little here, and it has been for three or four days.”, the constructed prompt text can be as follows: "You are a professional dermatology AI assistant. Please engage in a multi-round conversation with the user based on the following information to collect necessary information for providing more accurate recommendations.

[0083]

Image recognition result

Speech recognition result

[0084] Subsequently, the above answer text can be input into the above large language model, so that the large language model can generate a skin detection result text corresponding to the above skin area to be detected based on the above first description text, the above second description text and the answer text. For example, based on the first description text, the second description text and the answer text, a prompt text can be constructed and input into the large language model, and the large language model can infer based on the first description text, the second description text and the answer text in the prompt text to generate a skin detection result text corresponding to the skin area.

[0085] In some embodiments, the above question text can include question texts in multiple rounds of conversation. Wherein, a round of conversation refers to a complete round of communication, which usually contains a speech (which can be a question, statement, instruction, etc.) and a response (which can be an answer, reply, response, etc.). In this case, the question texts in each round of conversation can be output to the user in turn.

[0086] Specifically, in a case where the question text in the above-mentioned multi-turn dialogue is obtained by using the above-mentioned large language model, the question text in each turn of dialogue can be output to the user in turn; after outputting the question text in the current turn of dialogue (the i th turn of dialogue), the answer text input by the user corresponding to the question text in the current turn (the i th turn of dialogue) can be obtained, and the answer text is input into the large language model, and the large language model updates the question text in the next turn of dialogue (the i+1 th turn of dialogue) based on the answer text; the updated question text in the next turn of dialogue (the i+1 th turn of dialogue) is output to the user, so that the answer text input by the user corresponding to the updated question text in the next turn of dialogue (the i+1 th turn of dialogue) can be obtained, and the answer text is input into the large language model, and the large language model updates the question text in the next turn of dialogue (the i+1 th turn of dialogue) based on the answer text; and so on, until the question text in the multi-turn dialogue is completed.

[0087] For example, assuming that the above-mentioned large language model generates the question text in 3 turns of dialogue, wherein the question text in the 1st turn of dialogue is “Is this rash persistent or intermittent?”, the question text in the 2nd turn of dialogue is “Does the itching affect sleep?”, and the question text in the 3rd turn of dialogue is “Have you been sunburned recently, or have you been to places outdoors more often?”, if the answer text input by the user corresponding to the question text in the 1st turn of dialogue is “persistent, can't sleep because of itching”, the large language model can determine that there is no need to output the question text in the 2nd turn of dialogue to the user when updating the question text in the 2nd turn of dialogue based on the answer text, so the question text in the 2nd turn of dialogue can be skipped, and the question text in the 3rd turn of dialogue is output to the user.

[0088] In the technical solution provided by one or more embodiments of the present specification, for a skin region to be detected, on the one hand, at least one image obtained by photographing the skin region can be obtained, and the at least one image is input into a multi-modal model, and the multi-modal model generates a first description text corresponding to the skin region based on the at least one image; on the other hand, an audio for describing the skin region can be obtained, and the audio is subjected to speech recognition to convert the audio into a second description text corresponding to the skin region, so that a prompt text can be constructed based on the first description text and the second description text, and the prompt text is input into a large language model, and the large language model generates a skin detection result text corresponding to the skin region under the guidance of the prompt text.

[0089] In this way, the multimodal data corresponding to the skin region to be detected can be integrated, serving as the basis for the large language model to generate the skin detection result corresponding to the skin region through reasoning, which can improve the correctness and reliability of the skin detection result generated by the large language model, thereby improving the adaptability and response accuracy of the intelligent dialogue system based on the large language model in skin detection.

[0090] Reference Figure 3 , Figure 3 is a schematic diagram of a skin detection process based on a large language model provided by an example embodiment.

[0091] As shown in Figure 3 , in the above-mentioned skin detection process based on a large language model, for the skin region to be detected, at least one image (or video) obtained by photographing the skin region can be acquired, and the at least one image can be input into a multimodal model for processing. Specifically, the multimodal model can perform an image description generation task based on the at least one image. That is, the multimodal model can generate a first description text corresponding to the skin region based on the at least one image.

[0092] In the case where at least one image obtained by photographing the above-mentioned skin region to be detected is acquired, the at least one image can also be input into a skin detection model, and the skin detection model can detect the skin characteristics corresponding to the skin region based on the at least one image.

[0093] For the skin region to be detected, audio for describing the skin region can also be acquired. The content of the audio can be a description of the skin region to be detected itself, such as a description of the skin surface state, a description of the current feeling of the skin (e.g., itching, pain, etc.), etc. The content of the audio can also be a description of related information for assisting skin detection, such as a description of the duration of the abnormality of the skin region to be detected, or a description of the diet, medication, personal behavior, etc. before the abnormality of the skin region to be detected, etc. Further, the audio can be subjected to speech recognition to convert the audio into a second description text corresponding to the skin region. That is, the speech content can be recognized from the audio, and the recognized speech content can be output in the form of text.

[0094] In the case where the above-mentioned first description text, the above-mentioned second description text, and the above-mentioned skin characteristics are obtained, a prompt text (i.e., Prompt) can be further constructed based on the first description text, the second description text, and the skin characteristics. The constructed prompt text can be used to guide the large language model to generate a question text for obtaining skin detection related information corresponding to the above-mentioned skin region to be detected.

[0095] That is, the above prompt text can be input into the above large language model, and the large language model can generate a question text for obtaining skin detection related information corresponding to the skin region to be detected under the guidance of the prompt text. In this case, the question text can be output to the user, and the answer text corresponding to the question text input by the user is obtained.

[0096] Subsequently, the above answer text can be input into the above large language model, so that the large language model can perform reasoning based on the above first description text, the above second description text and the answer text to generate a skin detection result text corresponding to the skin region to be detected.

[0097] In actual application, the skin detection can be detection of whether a skin disease is present, detection of the health status of the skin, detection of the performance form of the skin, and the like.

[0098] Corresponding to the above embodiment of the skin detection method based on the large language model, the present specification also provides an embodiment of a skin detection device based on the large language model.

[0099] Reference Figure 4 , Figure 4 is a structural schematic diagram of an apparatus provided by an example embodiment. As shown in Figure 4 , the apparatus 400 mainly consists of a communication interface 402, a user interface 404, a processor 406 and a data storage 408, which are connected and communicate with each other through a system bus, a network or other connection mechanism 410. The communication interface 402 enables the apparatus 400 to communicate with other devices, access networks and transmission networks through analog or digital modulation. For example, the communication interface 402 can include a chipset and an antenna for wireless communication with a radio access network or an access point. In addition, the communication interface 402 can also be a wired interface such as an Ethernet, token ring or USB port, or a wireless interface such as Wifi, Bluetooth, Global Positioning System (GPS) or wide area wireless interface (e.g. WiMAX or LTE). Of course, the communication interface 402 can also support other forms of physical layer interface and standard or proprietary communication protocol. The communication interface 402 can also include multiple physical communication interfaces, such as a Wifi interface, a Bluetooth interface and a wide area wireless interface.

[0100] The user interface 404 includes input components such as a keypad, keyboard, touch- sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, among others, and output components such as a display screen (which can be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, other known or future developed equivalent devices, among others. The user interface 404 can also generate auditory output through a speaker, speaker jack, audio output port, audio output device, headphones, and other known or future developed equivalent devices. In some embodiments, the user interface 404 can include software, circuitry, or other forms of logic that enables the device 400 to transmit data to and receive data from external user input / output devices. In addition or alternatively, the device 400 can support remote access from other devices through the communication interface 402 or another physical interface (not shown). The user interface 404 can be configured to receive user input, the location and movement of which can be indicated by a pointer or cursor as described herein. The user interface 404 can also be configured as a display device for rendering or displaying a text segment.

[0101] The processor 406 can include one or more general-purpose processors and / or special-purpose processors.

[0102] The data storage 408 can include one or more volatile and / or non-volatile storage components, and can be integrated in whole or in part with the processor 406. The data storage 408 can include removable and / or non-removable components.

[0103] The processor 406 is capable of executing program instructions 418 (e.g., compiled or interpreted program logic and / or machine code) stored in the data storage 408 to implement various functionality described herein. The data storage 408 can include non-transitory computer-readable media having stored thereon program instructions that, when executed by the device 400, enable the device 400 to carry out any of the methods, processes, or functions disclosed in the specification and / or drawings. Execution of the program instructions 418 by the processor 406 can cause the processor 406 to utilize the data 412.

[0104] For example, the program instructions 418 can include an operating system 422 (e.g., operating system kernel, device drivers, and / or other modules) installed on the device 400, as well as one or more application programs 420 (e.g., a browser, social application, or game application). Similarly, the data 412 can include operating system data 416 and application data 414. The operating system data 416 is primarily accessible to the operating system 422, while the application data 414 is primarily accessible to the one or more application programs 420. The application data 414 can be located in a file system that is visible or hidden to a user of the device 400.

[0105] The application programs 420 can communicate with the operating system 422 through one or more application programming interfaces (APIs). These APIs can facilitate the application programs 420 reading and / or writing to the application data 414, communicating or receiving information via the communication interface 402, receiving or displaying information on the user interface 404, etc.

[0106] In some terminology, the application programs 420 can be referred to simply as “apps.” In addition, the application programs 420 can be downloaded to the device 400 through one or more online application stores or application markets. However, the application programs can also be installed on the device 400 through other means, such as through a web browser or a physical interface on the device 400 (e.g., a USB port).

[0107] Reference Figure 5 , Figure 5 is a block diagram of a skin detection device based on a large language model provided by an exemplary embodiment. The skin detection device based on a large language model can be applied to a device as shown in Figure 4 to implement the technical solutions of the present specification.

[0108] The above-mentioned skin detection device based on a large language model can include: An image description generation module 502 acquires at least one image obtained by photographing a skin region to be detected, and inputs the at least one image into a multi-modal model, and generates a first description text corresponding to the skin region based on the at least one image by the multi-modal model; A speech recognition module 504 acquires an audio for describing the skin region, and performs speech recognition on the audio to convert the audio into a second description text corresponding to the skin region; A detection result generation module 506 constructs a prompt text based on the first description text and the second description text, and inputs the prompt text into a large language model, and generates a skin detection result text corresponding to the skin region by the large language model under the guidance of the prompt text.

[0109] In some embodiments, the at least one image is at least one video frame in a video.

[0110] In some embodiments, the constructing a prompt text based on the first description text and the second description text, and inputting the prompt text into a large language model, and generating a skin detection result text corresponding to the skin region by the large language model under the guidance of the prompt text, includes: construct a prompt text based on the first description text and the second description text, and input the prompt text into a large language model, and generate, by the large language model under guidance of the prompt text, a question text for obtaining skin detection related information corresponding to the skin region; output the question text to a user, and obtain an answer text input by the user and corresponding to the question text; input the answer text into the large language model, and generate, by the large language model based on the first description text, the second description text and the answer text, a skin detection result text corresponding to the skin region.

[0111] In some embodiments, the question text includes question texts in multiple rounds of dialogues; The outputting the question text to the user and obtaining the answer text input by the user and corresponding to the question text includes: output the question texts in the rounds of dialogues to the user in sequence, and when obtaining the answer text input by the user and corresponding to the question text in the current round of dialogue, input the answer text into the large language model, and update, by the large language model based on the answer text, the question text in the next round of dialogue.

[0112] In some embodiments, the apparatus further includes: detecting, by a detection module, the at least one image, and detecting, by a skin detection model based on the at least one image, a skin characteristic corresponding to the skin region; The constructing the prompt text based on the first description text and the second description text includes: constructing the prompt text based on the first description text, the second description text and the skin characteristic in a text form.

[0113] In some embodiments, the apparatus further includes: outputting, by an output module, the skin detection result text to the user in a text form; and / or, converting the skin detection result text into an audio, and outputting the audio to the user.

[0114] In some embodiments, the performing speech recognition on the audio to convert the audio into the second description text corresponding to the skin region includes: inputting the audio into a speech recognition model, and performing, by the speech recognition model, speech recognition on the audio to convert the audio into the second description text corresponding to the skin region.

[0115] For ease of description, the above apparatus is described in various modules or units in terms of functions. Of course, when implementing one or more of the present specification, the functions of each module or unit can be implemented in the same or more software and / or hardware, or the modules implementing the same function can be combined by a combination of sub-modules or sub-units. The above-described apparatus embodiments are only illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0116] Based on the same concept as the above method, the present specification also provides an electronic device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method according to any one of the above embodiments by running the executable instructions.

[0117] Based on the same concept as the above method, the present specification also provides a computer readable storage medium, which stores computer instructions, and the instructions are executed by a processor to implement the steps of the method according to any one of the above embodiments.

[0118] Based on the same concept as the above method, the present specification also provides a computer program product, which comprises computer program / instructions, and the computer program / instructions are executed by a processor to implement the steps of the method according to any one of the above embodiments.

[0119] Those skilled in the art can understand that: In the present specification, the terms "comprise", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, product or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, product or device. Without more limitations, it does not exclude the presence of other same or equivalent elements in the process, method, product or device comprising the elements.

[0120] In the present specification, "one", "a" and "the" do not refer to the singular, but also include the plural.

[0121] In the present specification, the first, second, etc. ordinal numbers do not necessarily indicate the order, and many times are used for the convenience of distinguishing the objects. For example, the first server and the second server usually refer to two servers. In order to distinguish the two servers, they are expressed as the first server and the second server. Of course, sometimes the two servers can also be the same server.

[0122] In this specification, unless specifically stated otherwise, "receiving, sending of data" is not necessarily direct receiving and sending, and can be indirect receiving and sending. For example, A receives data sent by B, which can be understood as A directly receiving data sent by B, or A indirectly receiving data sent by B through C or other subjects. Similarly, B sends data to A, which can be understood as B directly sending data to A, or B indirectly sending data to A through C or other subjects. Here, C can be one subject, or two or more subjects.

[0123] In this specification, unless specifically stated otherwise, the association relationship generated between structures can be a direct association relationship or an indirect association relationship. For example, when describing "A is connected with B", unless it is specifically stated that A is directly connected with B, it should be understood that A can be directly connected with B, or A can be indirectly connected with B; for another example, when describing "A is on B", unless it is specifically stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B, or A can be indirectly above B (there are other elements between AB, and A is above B). By analogy.

[0124] This specification uses specific words to describe the embodiments of the specification. As "one embodiment", "an embodiment", and / or "some embodiments" means a certain feature, structure or characteristic related to at least one embodiment of the specification. Therefore, it should be emphasized and noted that the "an embodiment" or "one embodiment" or "one alternative embodiment" mentioned in different places in this specification does not necessarily refer to the same embodiment. In addition, the skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0125] Although one or more embodiments of the specification provide method steps as described in the embodiments or flowcharts, it can be understood that the order of steps listed in the embodiments or flowcharts is only one of the many execution orders, and does not represent the only execution order. Therefore, when the claims involve method steps, the adjustment of the order of the steps or the parallelism between the steps is also within the scope of protection of the claims.

Claims

1. A skin detection method based on a large language model, the method comprising: Acquiring at least one image obtained by photographing a skin area to be detected, and inputting the at least one image into a multimodal model, whereby the multimodal model generates a first description text corresponding to the skin area based on the at least one image; Acquiring audio for describing the skin area, and performing speech recognition on the audio to convert the audio into a second description text corresponding to the skin area; Based on the first description text and the second description text, a prompt text is constructed, and the prompt text is input into a large language model. Under the guidance of the prompt text, the large language model generates a skin detection result text corresponding to the skin area. The method according to claim 1 , wherein the at least one image is at least one video frame in a video.

3. The method according to claim 1, wherein constructing a prompt text based on the first description text and the second description text, inputting the prompt text into a large language model, and generating a skin detection result text corresponding to the skin area by the large language model under the guidance of the prompt text, comprises: constructing a prompt text based on the first description text and the second description text, and inputting the prompt text into a large language model, whereby the large language model, guided by the prompt text, generates a question text for obtaining skin detection-related information corresponding to the skin area; Outputting the question text to the user and obtaining the answer text corresponding to the question text input by the user; The answer text is input into the large language model, and the large language model generates a skin detection result text corresponding to the skin area based on the first description text, the second description text and the answer text.

4. The method according to claim 3, wherein the question text comprises question text in multiple rounds of dialogue; Outputting the question text to the user and obtaining an answer text input by the user corresponding to the question text includes: The question text in each round of dialogue is output to the user in sequence, and when the answer text corresponding to the question text in the current round of dialogue input by the user is obtained, the answer text is input into the large language model, and the large language model updates the question text in the next round of dialogue based on the answer text.

5. The method according to claim 1, further comprising: inputting the at least one image into a skin detection model, and having the skin detection model detect skin characteristics corresponding to the skin area based on the at least one image; The step of constructing a prompt text based on the first description text and the second description text includes: A prompt text is constructed based on the first description text, the second description text and the skin characteristics in text form.

6. The method according to claim 1, further comprising: Outputting the skin detection result text to the user in text form; and / or, The skin detection result text is converted into audio, and the audio is output to the user.

7. The method according to claim 1, wherein performing speech recognition on the audio to convert the audio into a second description text corresponding to the skin area comprises: The audio is input into a speech recognition model, and the speech recognition model performs speech recognition on the audio to convert the audio into a second description text corresponding to the skin area.

8. A skin detection device based on a large language model, the device comprising: an image description generation module, which obtains at least one image obtained by photographing the skin area to be detected, and inputs the at least one image into a multimodal model, whereby the multimodal model generates a first description text corresponding to the skin area based on the at least one image; a speech recognition module, which obtains audio used to describe the skin area and performs speech recognition on the audio to convert the audio into a second description text corresponding to the skin area; The detection result generation module constructs a prompt text based on the first description text and the second description text, and inputs the prompt text into a large language model. The large language model generates a skin detection result text corresponding to the skin area under the guidance of the prompt text.

9. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the steps of the method according to any one of claims 1 to 7 by running the executable instructions.

10. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program / instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images