Diagnosis method, device and equipment for autism spectrum disorder and medium

Abstract

The invention discloses an autism spectrum disorder diagnosis method and device, equipment and a medium, and the method comprises the steps: obtaining audio and video data of a test patient, and converting the audio and video data into behavior log data; and converting the behavior log data into script description data, inputting the script description data and doctor question data into a preset large language model, and obtaining a diagnosis reason and a diagnosis result of the autism spectrum disorder of the tested patient. Compared with the prior art, the method has the advantages that massive clinical data is not needed, and only video signals and audio signals of normal form interaction between a doctor and a tested patient during diagnosis in the testing process need to be converted into script description data which can be understood by a large language model; and then, a diagnosis reason and a diagnosis result of the autism spectrum disorder of the tested patient are generated by utilizing logical reasoning and text generation capability of the large language model, so that the diagnosis result is quickly obtained, and the diagnosis credibility of a doctor is further improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, equipment and medium for diagnosing autism spectrum disorder. Background Art

[0002] Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by social communication deficits and the presence of restricted, repetitive behaviors or interests, affecting approximately 2.3% of children and approximately 2.2% of adults. Individuals with ASD often experience additional developmental, behavioral, and mental health challenges, which can impose a lifelong burden on families.

[0003] Traditional ASD diagnostic methods include the Autism Diagnostic Interview (ADI) and the Autism Diagnostic Observation Schedule (ADOS). These methods require an experienced doctor to spend a lot of time on each assessment. In addition, the diagnosis relies on professional scales and is heavily dependent on the doctor's subjective judgment and clinical experience. Summary of the invention

[0004] The purpose of this application is to provide a diagnostic method, device, equipment and medium for autism spectrum disorder to solve the problems of high labor costs and subjective diagnostic results of traditional ASD methods and high data volume requirements of artificial intelligence methods in related technologies.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] In a first aspect, the present application provides a method for diagnosing autism spectrum disorder, comprising:

[0007] Acquire the audio and video data of the test patient, and convert the audio and video data into behavior log data; convert the behavior log data into script description data, and input the script description data and doctor's question data into a preset large language model to obtain the diagnostic reasons and diagnostic results of the test patient's autism spectrum disorder.

[0008] Preferably, the behavior log data includes: frame number, character identity, character position, character body language and character voice content, wherein the body language includes: character movements, eye direction, head posture, expression and gesture.

[0009] Preferably, the method of obtaining the audio and video data of the test patient and converting the audio and video data into behavior log data includes: obtaining the audio and video data of the test patient through a voice signal acquisition device and a video signal acquisition device; obtaining corresponding voice recognition data and image recognition data through a voice recognition model and an image recognition model; and arranging and classifying the voice recognition data and image recognition data to form behavior log data.

[0010] Preferably, the converting of the behavior log data into script description data comprises: converting the behavior log data into predefined response events; and converting the predefined response events into corresponding script description data.

[0011] Preferably, the predefined response events include: an event in which the test patient looks at a target object, an event in which the test patient points to a target object, an event in which the test patient smiles, an event in which the doctor or the test patient speaks, and an event in which the doctor or the test patient leaves the room.

[0012] Preferably, the script description data includes: system prompt words, script content, field prompt words and format prompt words.

[0013] In a second aspect, the present application also provides a diagnostic device for autism spectrum disorder, comprising:

[0014] The behavior conversion module is used to obtain the audio and video data of the test patient and convert the audio and video data into behavior log data; the diagnosis module is used to convert the behavior log data into script description data, and input the script description data and doctor's question data into a preset large language model to obtain the diagnosis reason and diagnosis result of the test patient's autism spectrum disorder.

[0015] Preferably, the behavior conversion module is specifically used for:

[0016] The audio and video data of autistic children are obtained through voice signal acquisition equipment and video signal acquisition equipment; the audio and video data are passed through a voice recognition model and an image recognition model to obtain corresponding voice recognition data and image recognition data; the voice recognition data and image recognition data are sorted and classified to form behavior log data.

[0017] In a third aspect, the present application further provides a computer electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above-mentioned methods for diagnosing autism spectrum disorders when executing the computer program.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for diagnosing autism spectrum disorders described in any one of the above are implemented.

[0019] The present application provides a method, device, equipment and medium for diagnosing autism spectrum disorder. The method obtains audio and video data of the test patient and converts the audio and video data into behavior log data; converts the behavior log data into script description data, and inputs the script description data and doctor's question data into a preset large language model to obtain the diagnostic reasons and diagnostic results of the autism spectrum disorder of the test patient. Compared with the related art, it has the following effects: the scheme of the present application does not require a large amount of clinical data, but only needs to convert the video signal and audio signal of the paradigm interaction between the doctor and the test patient during the test process into script description data that can be understood by the large language model, and then use the logical reasoning and text generation capabilities of the large language model to generate the diagnostic reasons and diagnostic results of the autism spectrum disorder of the test patient, while quickly obtaining the diagnostic results, it also further improves the credibility of the doctor's diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a method for diagnosing autism spectrum disorder according to an embodiment of the present application;

[0021] Figure 2 It is a data processing flow diagram of a method for diagnosing autism spectrum disorders in an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of text conversion in a method for diagnosing autism spectrum disorders in an embodiment of the present application;

[0023] Figure 4 It is a schematic diagram of the contents of a script description in a method for diagnosing autism spectrum disorders in an embodiment of the present application;

[0024] Figure 5 This is a schematic diagram of the contents of another script description in a method for diagnosing autism spectrum disorders in an embodiment of the present application;

[0025] Figure 6 It is a schematic diagram of the diagnostic reasons and diagnostic results of a method for diagnosing autism spectrum disorders in an embodiment of the present application;

[0026] Figure 7 This is a schematic diagram of another diagnostic reason and diagnostic result of a method for diagnosing autism spectrum disorder in an embodiment of the present application;

[0027] Figure 8is a schematic structural diagram of a diagnostic device for autism spectrum disorder according to an embodiment of the present application;

[0028] Fig. 9 It is a structural schematic diagram of a computer electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may also be a centered element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a centered element at the same time. In contrast, when an element is referred to as being "directly on" another element, there is no intermediate element. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0031] In this application, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0032] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0033] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms "a", "said" and "the" used in one or more embodiments of the present application are also intended to include plural forms, unless the context clearly indicates other meanings.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of the template are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the associated listed items.

[0035] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when the time of".

[0036] Currently, sensor technology and artificial intelligence technology show great potential in assisting ASD diagnosis. Mainstream artificial intelligence methods are divided into two categories: the first category uses machine learning technology to analyze behavior-related features from videos, such as body movements, head movements, and eye patterns; the second category involves deep learning methods that directly train raw video data to predict ASD labels. These two methods perform well on their respective datasets through supervised learning. However, the scarcity of ASD clinical data limits their accuracy. In addition, these methods can only provide predictions of diagnostic results and lack explanatory explanations.

[0037] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes are not repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0038] Please refer to Figure 1 and Figure 2 The present invention provides a method for diagnosing autism spectrum disorder, which is applied to the field of diagnosing autistic children and includes at least the following steps:

[0039] S10. Acquire audio and video data of the test patient, and convert the audio and video data into behavior log data.

[0040] Specifically, the audio data and video data of the test patient during the consultation process are obtained, and the audio data and video data are converted into behavior log data.

[0041] In a specific embodiment, audio and video data of the test patient are obtained through a voice signal acquisition device and a video signal acquisition device; the audio and video data are passed through a voice recognition model and an image recognition model to obtain corresponding voice recognition data and image recognition data; the voice recognition data and image recognition data are sorted and classified to form behavior log data.

[0042] It can be understood that in the room between the doctor and the test patient, there are voice signal acquisition equipment and video signal acquisition equipment. For example, the voice signal acquisition equipment can be a microphone and a recording device, and the video signal acquisition equipment can be a camera and a camcorder. The above-mentioned voice signal acquisition equipment and video signal acquisition equipment can collect the voice signals and video signals of the doctor and the test patient in the room, and perform corresponding storage processing after the acquisition is completed.

[0043] In a specific embodiment, the behavior log data includes: frame number, character identity, character position, character body language and character voice content, wherein the body language includes: character movements, eye direction, head posture, expression and gesture.

[0044] It can be understood that each RGB video is first passed through an instance segmentation model (SOLOv2) to extract the human body region (bounding box and mask). At the same time, the face detection model (ret inface) and the face recognition model (ArcFace) extract the face image and distinguish the identity of the test patient, the test patient's parents and the doctor. Then the HRNet model is used to identify the 17 types of human key points defined in the MS COCO dataset to determine each person's action. The head posture prediction model SYSUGaze model predicts each person's head posture (pitch, yaw and roll angle) and the horizontal and vertical angles of the eyes. The expression recognition model trained on the large dataset AffectNet recognizes the three expressions of neutral, happy and sad. The hand area is detected by the gesture detection model (YOLOv5), and the three most common basic pointing gestures in clinical practice are detected by the gesture recognition model ResNet-50.

[0045] S20, converting the behavior log data into script description data, and inputting the script description data and the doctor's question data into a preset large language model to obtain the diagnostic reasons and diagnostic results of the autism spectrum disorder of the test patient.

[0046] It can be understood that in order to convert these behavior log data into text that can be understood by the large language model, it is first necessary to convert the behavior log data into predefined response events; then convert the predefined response events into corresponding script description data, and finally input the script description data into the preset large language model to obtain the final diagnosis result.

[0047] In a specific embodiment, the predefined response events include: an event in which the test patient looks at a target object, an event in which the test patient points to a target object, an event in which the test patient smiles, an event in which the doctor or the test patient speaks, and an event in which the doctor or the test patient leaves the room.

[0048] Specifically, the behavioral log data is simplified by the above five basic response events to obtain the occurrence timestamps and durations of the basic response events in the interaction between the doctor and the test patient, and then the basic response events are converted into corresponding script text descriptions through text conversion.

[0049] It should be noted that if Figure 3 As shown in the figure, the text conversion process can be called the textualization process of the name-response paradigm, which includes four parts: 1. Gender and age, describing the basic information of the test subject, 2. [Timestamp, environment description], describing the positions of the doctor and the test patient and the props layout required for the paradigm, 3. [Timestamp, instruction description], describing the content of the doctor's instruction and its timestamp, 4. [Timestamp, reaction description], describing the test patient's reaction to the doctor's instruction and its timestamp.

[0050] See also Figure 4 and Figure 5 In a specific embodiment, the script description data includes: system prompt words, script content, field prompt words and format prompt words.

[0051] Specifically, the content of the script description data has the following functions: 1. System prompt words, which are used to describe the role played by the large language model and the capabilities of the script as a whole; 2. Script content, which is used to describe the overall content of the script; 3. Domain prompt words, which are used to increase the domain knowledge of autism spectrum disorders in the large language model; 4. Format prompt words, which are used to limit the format of the large language model output, which must include two aspects: diagnosis reasons and diagnosis results.

[0052] As an example, the diagnosis reasons and diagnosis results in this embodiment can refer to Figure 6 and Figure 7 .

[0053] It should be noted that the large language model in this application is an algorithm model that adopts an unsupervised learning learning method. Unsupervised learning is a machine learning method that discovers patterns or structures in data by learning the intrinsic structure of data without labels. In unsupervised learning, we only provide unlabeled text data, that is, training sets. By learning from the training set, the machine learning algorithm in this application can discover the semantics, knowledge and logic in the text, and ultimately enable the large language model to have the ability of logical reasoning and text generation.

[0054] Exemplarily, the large language model in the present application may include but is not limited to: any one of the closed source models gpt-4o, Claudé3.5-sonnet, Monnshot's API interface call and the open source models LLAMA3.1, Qwen2-72B-I nstruct models.

[0055] The present application provides a method, device, equipment and medium for diagnosing autism spectrum disorder. The method obtains audio and video data of the test patient and converts the audio and video data into behavior log data; converts the behavior log data into script description data, and inputs the script description data and doctor's question data into a preset large language model to obtain the diagnostic reasons and diagnostic results of the autism spectrum disorder of the test patient. Compared with the related art, it has the following effects: the scheme of the present application does not require a large amount of clinical data, but only needs to convert the video signal and audio signal of the paradigm interaction between the doctor and the test patient during the test process into script description data that can be understood by the large language model, and then use the logical reasoning and text generation capabilities of the large language model to generate the diagnostic reasons and diagnostic results of the autism spectrum disorder of the test patient, while quickly obtaining the diagnostic results, it also further improves the credibility of the doctor's diagnosis.

[0056] See also Figure 8 The present application also provides a diagnostic device 200 for autism spectrum disorder, comprising:

[0057] A behavior conversion module 201 is used to obtain audio and video data of a test patient and convert the audio and video data into behavior log data;

[0058] The diagnosis module 202 is used to convert the behavior log data into script description data, and input the script description data and the doctor's question data into a preset large language model to obtain the diagnosis reason and diagnosis result of the autism spectrum disorder of the test patient.

[0059] In one embodiment of the present application, the behavior conversion module 201 is specifically used for:

[0060] The audio and video data of autistic children are obtained through voice signal acquisition equipment and video signal acquisition equipment; the audio and video data are passed through a voice recognition model and an image recognition model to obtain corresponding voice recognition data and image recognition data; the voice recognition data and image recognition data are sorted and classified to form behavior log data.

[0061] See also Fig. 9The embodiment of the present application further provides a computer electronic device 400, including a memory 403 and a processor 402, wherein the memory 403 stores a computer program, and when the processor executes the computer program, the steps of the method for diagnosing autism spectrum disorder described in any one of the above items are implemented.

[0062] Specifically, the electronic device 400 includes: a transceiver 401, a bus interface and a processor 402, wherein the processor 402 is used to obtain audio and video data of the test patient and convert the audio and video data into behavior log data; convert the behavior log data into script description data, and input the script description data and doctor's question data into a preset large language model to obtain the diagnosis reason and diagnosis result of the test patient's autism spectrum disorder. .

[0063] In the embodiment of the present application, the electronic device 400 further includes a memory 403. Figure 4 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits of one or more processors represented by processor 402 and memory represented by memory 403. The bus architecture may also link various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface. The transceiver 401 may be a plurality of components, i.e., including a transmitter and a receiver, providing a unit for communicating with various other devices on a transmission medium. The processor 402 is responsible for managing the bus architecture and general processing, and the memory 403 may store data used by the processor 402 when performing operations.

[0064] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned methods for diagnosing autism spectrum disorders.

[0065] In this embodiment, the computer readable storage medium may be a non-volatile storage medium or a volatile storage medium. For example, the computer storage medium may include but is not limited to: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0066] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.

[0067] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0068] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0069] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0070] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a terminal device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application.

[0071] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A method for diagnosing autism spectrum disorder, characterized in that: include: Acquire audio and video data of the test patient, and convert the audio and video data into behavior log data; The behavior log data is converted into script description data, and the script description data and the doctor's question data are input into a preset large language model to obtain the diagnostic reasons and diagnostic results of the test patient's autism spectrum disorder.

2. The method for diagnosing autism spectrum disorder according to claim 1, characterized in that: The behavior log data includes: frame number, character identity, character position, character body language and character voice content, wherein the body language includes: character movements, eye direction, head posture, expression and gesture.

3. The method for diagnosing autism spectrum disorder according to claim 1, characterized in that: The step of obtaining audio and video data of the test patient and converting the audio and video data into behavior log data includes: Acquire audio and video data of the test patient through voice signal acquisition equipment and video signal acquisition equipment; Passing the audio and video data through a speech recognition model and an image recognition model to obtain corresponding speech recognition data and image recognition data; The voice recognition data and image recognition data are sorted and classified to form behavior log data.

4. The method for diagnosing autism spectrum disorder according to claim 1, characterized in that: The converting the behavior log data into script description data includes: Converting the behavior log data into predefined response events; The predefined response event is converted into corresponding script description data.

5. The method for diagnosing autism spectrum disorder according to claim 4, characterized in that: The predefined response events include: an event in which the test patient looks at a target object, an event in which the test patient points to a target object, an event in which the test patient smiles, an event in which the doctor or the test patient speaks, and an event in which the doctor or the test patient leaves the room.

6. The method for diagnosing autism spectrum disorder according to claim 4, characterized in that: The script description data includes: system prompt words, script content, field prompt words and format prompt words.

7. A diagnostic device for autism spectrum disorder, characterized in that: include: A behavior conversion module, used to obtain audio and video data of the test patient and convert the audio and video data into behavior log data; The diagnosis module is used to convert the behavior log data into script description data, and input the script description data and the doctor's question data into a preset large language model to obtain the diagnosis reasons and diagnosis results of the autism spectrum disorder of the test patient.

8. The diagnostic device for autism spectrum disorder according to claim 7, characterized in that: The behavior conversion module is specifically used for: Acquire audio and video data of autistic children through voice signal acquisition equipment and video signal acquisition equipment; Passing the audio and video data through a speech recognition model and an image recognition model to obtain corresponding speech recognition data and image recognition data; The voice recognition data and image recognition data are sorted and classified to form behavior log data.

9. A computer electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method for diagnosing autism spectrum disorders according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for diagnosing autism spectrum disorders according to any one of claims 1 to 6 are implemented.

Images