Psychological semantic scene image generation method and device, and computer readable storage medium

Abstract

The invention provides a psychological semantic scene image generation method, which comprises the following steps of: 1, processing data according to a psychological scale filled by a user to obtain a three-level level score of a psychological factor; 2, constructing a basic semantic concept group through hierarchical semantic mapping; 3, constructing structured cue words, wherein the structured cue words comprise a text cue word project and an image cue word project; 4, generating an image note through a large language model according to the text cue word, and generating a background image and page elements through a text-to-graph model according to the image cue word; 5, generating a semantic scene graph from the background picture and the page elements generated in the step 4 through a differentiable graph synthesizer; and step 6, generating a final psychological report in combination with the generated graph annotation and the semantic scene graph.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic image generation, and relates to a method, a device, and a computer-readable storage medium for generating a psychological semantic scene image. Background Art

[0002] In the prior art, psychological reports are usually presented in text form, which may be difficult for non-professionals to understand and interpret. To solve this problem, some studies have tried to use graphics or images to present data, such as using basic graphics such as bar charts and pie charts to display the results of the scale. However, these methods can usually only show the distribution and proportion of data, but cannot intuitively reflect the individual's psychological state.

[0003] Existing psychological scales, such as the Big Five Personality Traits Scale, Beck Anxiety Inventory, Beck Depression Inventory, etc., can quantify an individual's psychological state, but they have the following defects: (1) Subjectivity: The results of the scales often depend on the subjects' self-reports and may be affected by the subjects' subjective biases and misunderstandings; (2) Difficulty in interpretation: The results of the scales are usually presented in numerical form, which may be difficult for non-professionals to understand and interpret.

[0004] In addition, existing cross-modal psychology data visualization methods are mainly rule-based, and their results are limited by expert experience. It is difficult to cover all test situations, and the resulting infographics lack realism and readability. For example, projective tests (such as the Rosenseck inkblot test and the Thematic Apperception Test) and painting therapy in psychoanalysis emphasize the relationship between vision and psychological state, but the interpretation of their results usually requires professional training and experience. On the other hand, although expression recognition methods emphasize the relationship between vision and psychological state, their accuracy is usually limited by the complex relationship between expression and emotion. Summary of the invention

[0005] In order to address the deficiencies in the prior art, the purpose of the present invention is to provide a method, system and application for generating psychological semantic scene images. The present invention proposes a method for generating psychological semantic scene images from psychological scale results, which is used to visualize the user's psychological state with semantic metaphors in psychological reports. The method of the present invention applies a cross-modal generation model from text to graphics, and through scale data analysis, constructs a mapping from scale data to semantics, a method from semantics to text templates, a method from semantics to graphic elements, and randomized picture synthesis to achieve the generation of psychological semantic scene images. The input of the method of the present invention is the psychological scale evaluation result data, and the output is a semantic scene graphic and a corresponding description text that can represent the psychological state displayed by the psychological scale.

[0006] The present invention provides a method for generating a psychosemantic scene image, the method comprising the following steps:

[0007] Step 1: Based on the psychological scale filled in by the user, process the data to obtain the three-level scores of the psychological factors.

[0008] The three-level scores refer to the original scores of the factors of each dimension of the psychological scale, which are divided into three different levels to reflect the score strength of the factor:

[0009] Level 3: Normal level. The raw score of this factor is within the normal range, reflecting the normal psychological status of this dimension.

[0010] Level 2: Mild Abnormality. The raw score of this factor is slightly higher than the normal range, reflecting that there is a mild abnormality in the psychological status of this dimension. It needs to be taken seriously but is not serious.

[0011] Level 1: Severe Abnormality. The raw score of this factor is significantly higher than the normal range, reflecting that the psychological status of this dimension is severely abnormal. It requires high attention and intervention measures.

[0012] If the value is more serious, it will be classified as a crisis. The specific raw score range can be determined according to different psychological scales and experts. For example, if the raw score range of a factor is 0-100, 0-20 points can be set as the third level, 20-40 points as the second level, and 40-100 points as the first level.

[0013] Step 2: Construct basic semantic concept groups through hierarchical semantic mapping;

[0014] Step 3: construct structured prompt words including text prompt word engineering and image prompt word engineering;

[0015] Step 4: Generate a caption through a large language model based on the text prompt words obtained by the construction, and generate a background image and page elements through a text-to-graphics model based on the image prompt words obtained by the construction;

[0016] Step 5: Generate a semantic scene graph using the background image and page elements generated in step 4 through a differentiable graph synthesizer;

[0017] Step 6. Generate the final psychological report by combining the captions generated in step 4 and the semantic scene graph generated in step 5.

[0018] In step 1, the psychological scale assesses students’ mental health status through a comprehensive system perspective and provides dimensions that need attention. Theoretical models (such as Figure 4(as shown) fully considers the mental health level, resilience level and intervention mechanism, understands and identifies the different causes of risk warning, and provides direction for targeted counseling and intervention. This is not available in existing psychological scales. The scale is divided into 4 dimensions and 11 factors. The self-harm tendency dimension is composed of 1 factor of self-harm tendency; the mental health dimension is composed of 4 factors of emotional state, sleep state, Internet dependence, and aversion to school; the inducement dimension is composed of 1 factor of traumatic events; the personality factor dimension is composed of 5 factors of interpersonal pleasing, interpersonal sensitivity, persistence, sense of efficacy and family relationship. As long as the score of one factor in each dimension exceeds the norm reference score, it is considered to be detected. Then, according to the detection of the four dimensions and matching with the table below, the three levels A1, A2,…, A are finally determined. k , where k represents the number of factors, A i Represents the three-level intensity classification of the i-th factor in the scale results.

[0019] The complete scale is as follows:

[0020] Questionnaire for students' life, study and health status (admission version)

[0021]

[0022]

[0023]

[0024] The corresponding relationship between the items in the above scale and each factor:

[0025] Emotional status: items 12, 15, 16, 20, 27, 34

[0026] Sleep status: items 12 and 15

[0027] Internet dependence: items 23, 26, and 35

[0028] ·Tendency to dislike studying: Items 17, 30, and 33

[0029] Self-harm tendency: items 13, 22, and 31

[0030] Traumatic events: Items 24 and 25

[0031] Interpersonal ingratiation: Items 9, 18, and 28

[0032] Interpersonal sensitivity: items 7 and 19

[0033] Persistence: Items 2, 8, and 32

[0034] Sense of efficacy: Items 5 and 6

[0035] Family relationships: items 4, 21, and 36

[0036] The dimensions and factor structure of the scale are as follows:

[0037] Self-harm tendency dimension: Self-harm tendency factor

[0038] Mental health dimensions: emotional state, sleep condition, internet addiction, and factors of aversion to learning

[0039] · Predisposing dimension: Traumatic event factor

[0040] Personality factor dimensions: interpersonal pleasing, interpersonal sensitivity, persistence, sense of efficacy, family relationship factors

[0041] The scale is divided into 4 dimensions and 11 factors. The self-harm tendency dimension is composed of 1 factor of self-harm tendency; the mental health dimension is composed of 4 factors of emotional state, sleep state, Internet dependence, and aversion to learning; the inducement dimension is composed of 1 factor of traumatic events; the personality factor dimension is composed of 5 factors of interpersonal pleasing, interpersonal sensitivity, persistence, sense of efficacy, and family relationship. As long as the score of one factor in each dimension exceeds the norm reference score, it is considered to be detected. Then, according to the detection of the four dimensions, match it with the table below, and finally determine the three-level level.

[0042] Table 1 Classification of three levels

[0043] Classification results Self-harm tendencies Mental Health Causes Personality Factors Class III A √ √ √ Class III B √ √ Class III C √ √ Class III D √ Class II A √ √ Class B √ Class II C √ Level 1 crisis √

[0044] In step 2, the hierarchical semantic mapping refers to dividing the score strength of different dimensional factors into three levels (divided into 9 categories in Table 1), the score strength is obtained by weighted linear combination, and then mapped into different basic semantic concept groups; the basic semantic concept group is represented by {O k ,B k}, where B k Indicates the background style, O k Represents animal objects, the third level of the factor in the scale A k Through mapping hash tables, they are respectively associated with the basic semantic concept groups {O k ,B k}One-to-one correspondence; in the present invention, only the background style and animal objects are considered, without specifically considering their more general forms of expression.

[0045] In step 3, the generation of the structured prompt words further includes the following:

[0046] For the text prompt word project, the text prompt words include: role positioning, task objectives, context and language template; the context and language template are used to generate image captions for the large language model LLM, and the captions include: introduction of the semantics of the image and the scene and analysis of the user's psychological state; wherein,

[0047] Role positioning: Determine the role of the language model in the scenario as a psychological counselor for non-professional users.

[0048] Task Objective: Specify the goal or task in the scene and interpret the psychological test results through picture elements.

[0049] Context and language template: Provides background information and language templates for the scene to generate captions. The context includes the user's psychological scale results and psychological semantic concept groups, while the language template is designed based on the scene's picture interpretation method and the format of psychological interpretation.

[0050] For the image prompt word project, the image prompt words include role positioning, task objectives, and context, which are used to generate the required background images and psychologically corresponding page elements for the text-to-image generation model (T2I Model); the language template is part of the context and is used to define the answer form.

[0051] Role positioning: Determine the role of the language model in the scene as the prompt engineer of the Stable Difffusion image, including the format of the general Stable Diffsuion model input prompt words.

[0052] Task goal: Specify the goal or task in the scene, and write the prompt words for the generated image through semantic concept groups.

[0053] Context and language template: The context includes the current background or introduction corresponding to the page element, and the language template used to define the output format.

[0054] The generalized and specific construction process of text prompt words:

[0055] Determine the role positioning and task objectives: Position the language model as a psychological counselor for non-professional users, and specify the task objective as interpreting psychological test results through image elements.

[0056] Design context and language templates: Provide background information of the scene, including the user's psychological scale results and psychological semantic concept groups. Design appropriate language templates for generating captions, which include an introduction to the semantics of the image and the scene and an analysis of the user's psychological state.

[0057] Providing context and language templates to the Large Language Model (LLM): The context and language templates in the textual cues are provided to the Large Language Model (LLM) to generate captions associated with the psycho-semantic scene images.

[0058] The generalized and specific construction process of image prompt words is as follows:

[0059] Determine the role positioning and task objectives: Position the language model as the Prompt engineer of the Stablediffusion image, and specify the task goal as using semantic concept groups to write the prompt words required to generate pictures.

[0060] Design context and language templates: Provide appropriate context information, including an introduction to the current background or page element. Design a common language template to generate the required background images and psychologically corresponding page elements.

[0061] Providing context and language templates to the text-to-image generation model T2I: Provide the context and language templates in the image prompt words to the text-to-image generation model (T2I Model) to generate background images and page elements that conform to the psychological semantic scene.

[0062] In step 4, the captions are generated through the large language model based on the constructed text prompt words, including: input data, professional descriptions of the basic factors of the psychological scale, and interpretation of the elements in the image;

[0063] The input data refers to the three-level classification of each dimension factor calculated from the options of each question selected by the user;

[0064] The professional description of the basic factors of the psychological scale refers to the interpretation of the psychological scale results obtained by combining the psychological definition of each factor as the context of the prompt word with the introduction template;

[0065] The interpretation of elements in the image refers to parsing and describing the image elements appearing in the scene to express the meaning, emotion or state related to the results of the psychological scale.

[0066] The generation of the legend comprises the following steps:

[0067] Step 4.1: Give the large language model a specific role and provide specific task objectives;

[0068] Step 4.2, providing context information for generating a caption; the context information specifically refers to a simple and unprocessed introduction corresponding to the current background image or page element, including an animal object corresponding to the semantic concept group;

[0069] Step 4.3: Generate a caption using a preset function output template based on the role positioning, task objectives, and context information given above.

[0070] In step 4, generating background images and page elements through a text-to-graphics model according to the constructed image prompt words further includes the following steps:

[0071] Step 4.4: Give the text-to-image model specific task objectives and context information for generating page elements and background images;

[0072] Step 4.5, the text-to-image model generates page elements according to the task objectives and context information of generating page elements;

[0073] Step 4.6: The text-to-image model generates a background image based on the task objective and context information of generating a background image.

[0074] Among them, the page elements are animal objects, and the background is the name of the season scene.

[0075] Before and after generating captions, background images, and page elements in step 4, disturbances are introduced to make the generation richer;

[0076] The perturbation can be added before and after the generation model. Specifically, the pre-perturbation includes adding neutral semantic objects to the basic semantic concept group so that the generated result includes other animals that are adapted to the background image. The post-generation model perturbation is responsible for synthesizing the page elements into the background according to random numbers.

[0077] Pre-perturbation includes the following methods:

[0078] Add neutral semantic objects to the basic semantic concept group so that the generated results contain other animals that fit the background image. This can introduce more elements into the image and increase the diversity of the generated results.

[0079] Post-perturbation involves the following methods:

[0080] The generated page elements are shuffled according to random numbers and synthesized into the background. This can change the arrangement of page elements and make the generated semantic scene graph more diverse and changeable.

[0081] By introducing front and back perturbations, the generated captions, background images, and page elements can be richer and more diverse, increasing the diversity and variability of the generated results.

[0082] In step 5, the differentiable graph synthesizer is a tool for synthesizing semantic scene graphs. It has the following definitions and functions:

[0083] Definition: A differentiable graph synthesizer is an image synthesis model based on differentiable operations that can combine page elements and background images to generate a scene graph with semantic meaning.

[0084] Function: The function of the differentiable graphics synthesizer is to generate a semantic scene graph that meets the semantic requirements by combining and adjusting page elements and background images. It can synthesize images according to the input parameters and constraints, and maintain the differentiability of the model, so that the synthesis process can be optimized through gradient back propagation.

[0085] The specific process of combining page elements and background images to generate a semantic scene graph through a differentiable graph synthesizer is as follows:

[0086] 1. For a given page element and background image, first embed the page element into the appropriate position in the background image.

[0087] 2. According to the embedded page elements and background images, synthesis is performed through differentiable operations to adjust the appearance and position of the elements and merge them with the background image to generate a synthetic semantic scene graph.

[0088] 3. Iteratively optimize the synthesized semantic scene graph and use gradient backpropagation to minimize the difference with the target image until a semantic scene graph that meets the requirements is obtained.

[0089] The steps to set up / train a differentiable graph synthesizer include:

[0090] Step 5.1 The network architecture is represented as follows:

[0091] Input image -> encoder -> feature representation -> decoder -> output image

[0092] In addition, the specific structures of the encoder and decoder are as described in this specification.

[0093] Step 5.2: Design a loss function

[0094] The loss function is the key to ensuring the quality of synthesis. The design may include:

[0095] Pixel reconstruction loss: L_{pixel}=||I_c-I_{target}||_2

[0096] Content loss: L_{content}=||V(I_c)-V(I_{target})||_2

[0097] Adversarial loss: L_{adv} = log(1-D(G(I_c)))

[0098] Among them, I_c is the synthetic image, I_target is the target image, V is the feature extractor, G is the generation network, and D is the discriminant network.

[0099] Step 5.3: Construct training data

[0100] In order to train the network, a large number of input image and target image pairs need to be constructed as training data. The input image contains various elements and backgrounds, and the target image is a scene image with the elements correctly synthesized. Such training data can effectively help the network learn how to extract useful features from the input image and generate high-quality target images based on these features.

[0101] Step 5.4: Choose appropriate background images and page elements

[0102] In the process of generating images, it is very important to choose appropriate background images and page elements. It is necessary to consider the semantic content of the elements, as well as their layout and position in the background image. To achieve this, firstly, according to the input psychological scale results, a hierarchical semantic mapping method is used to map the psychological factor scores to different objects and backgrounds to form a basic semantic concept group. Then, according to the basic semantic concept group, appropriate background images and page elements are selected.

[0103] Step 5.5: Calculate the integration degree between the element and the background

[0104] Calculating the fusion degree of elements and background is an important step, which can help determine the matching degree of elements and background. The fusion degree of elements and background can be calculated by some metrics, such as the similarity of color, texture, shape, etc. Specifically, the fusion degree of elements and background can be calculated using methods such as color histogram, texture features, shape features, etc.

[0105] Step 5.6: Optimize the fusion effect

[0106] After calculating the degree of fusion between the element and the background, the fusion effect needs to be optimized. This is achieved by adjusting the position, size, shape and other attributes of the element, as well as possible background adjustments. Specifically, the position, size, shape and other attributes of the element can be adjusted through optimization methods such as gradient descent to maximize the degree of fusion between the element and the background. At the same time, the fusion effect can be further optimized by adjusting the brightness, contrast and other attributes of the background.

[0107] The differentiable composition method is a method of achieving image synthesis through differentiable operations, which applies the techniques in computer vision and graphics. Specifically, the differentiable composition method uses differentiable image operations, such as image superposition, scaling, rotation, translation, etc., as well as a series of differentiable functions and modules such as convolutional neural networks, to synthesize and adjust page elements and background images.

[0108] The soft-driven synthesis effect of constructing loss means constraining the synthesis result by designing loss function so that it meets the desired effect. Soft driving means that these constraints are not absolutely mandatory, but driven in a flexible way. The specific form of loss function can be designed according to specific application scenarios and requirements. Common ones include pixel-level difference loss, content loss, style loss, etc.

[0109] Image optimization and artifact removal are technical means and solutions for improving synthesis effects and reducing artifact problems in synthesis results. Specifically, image optimization is to continuously optimize the details, lighting, color, etc. of the synthesized image through iterative optimization to obtain better visual effects. Artifact removal is to use a series of algorithms and technical means, such as edge smoothing and fusion strategy adjustment, to reduce or eliminate the appearance of artifacts and improve the quality and realism of the synthesis results in order to solve the artifact problems that may occur during the synthesis process. The specific implementation method can be selected and designed according to the specific synthesis tasks and requirements.

[0110] In step five, differentiable graphics synthesizer, loss construction, image optimization and artifact elimination are involved. Examples of these mathematical formulas are shown in the figure:

[0111] 1. Differentiable Graphics Synthesizer:

[0112] -Image synthesis formula: Synthesized image I_c = f(I_b, E), where I_c represents the synthesized image, I_b represents the background image, and E represents the page element.

[0113] -Element position adjustment formula: E'=E+δE, where E' represents the adjusted page element, and δE represents the position adjustment vector.

[0114] 2. Loss construction:

[0115] -Pixel-level difference loss: L_pixel = Σ|I_c-I_target|, where I_target represents the target image.

[0116] - Content loss: L_content = ||V(I_c)-V(I_target)||, where V represents the feature extractor.

[0117] -Style loss: L_style = Σ||G(I_c)-G(I_target)||, where G represents the style feature extractor.

[0118] 3. Image Optimization:

[0119] -Optimization formula: I_optimized = argmin(L_total(I_c)), where L_total represents the comprehensive loss function and I_c represents the generated image.

[0120] 4. Artifact elimination:

[0121] -Edge smoothing formula: I_smooth = f(I_optimized), where f represents the smoothing operator.

[0122] -Fusion strategy adjustment: I_final = g(I_smooth, I_target)$, where g represents the fusion strategy adjustment function.

[0123] Based on the above method, the present invention also proposes a psychological semantic scene image generation device, including: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the above method is implemented.

[0124] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0125] The beneficial effects of the present invention include: the method of the present invention can present the psychological scale data in an intuitive and easy-to-understand manner, which helps users better understand and accept the psychological report. At the same time, the method can also provide more information for psychological counselors to help them better understand and interpret the user's psychological state.

[0126] 1. Intuitive and easy-to-understand display: The present invention applies semantic concept groups and generative models to convert psychological scale data into intuitive and easy-to-understand images and text displays. Through the combination of images and semantic prompts, users can more intuitively understand and accept the content of the psychological report.

[0127] 2. Personalized customization: Based on the user's psychological scale data and characteristics, the present invention uses semantic concept groups and generative models to generate personalized psychological semantic scene images. This personalized customization method helps users better understand and identify the information displayed in the psychological report.

[0128] 3. Provide more information support: The present invention not only shows the results of the psychological report to the user, but also provides more information support through structured prompt words and semantic scene images. This helps the psychological counselor to better understand and interpret the user's psychological state and provide more comprehensive and in-depth consulting services.

[0129] 4. Improvement of user participation and enthusiasm: Through intuitive image display and personalized content, the present invention can improve users' participation and enthusiasm in psychological testing and consultation, and enhance their recognition of their own psychological state and motivation to adjust.

[0130] 5. Enhancement of emotions and experience: The present invention converts psychological scale data into images and texts to create psychological semantic scenes with greater emotional resonance and immersion, thereby enhancing users' emotional experience and participation.

[0131] 6. Improved information transmission effect: Compared with traditional pure text or simple reports, the present invention can more intuitively and comprehensively transmit psychological test results and related information, thereby improving the effect and efficiency of information transmission.

[0132] At present, since the present invention is a new innovative technology, there is no specific comparative data with the prior art. Figure 5 , 6 As shown, the effects and advantages of the present invention can be evaluated through user feedback and evaluation studies to compare with traditional methods and prove its advantages in user understanding and acceptance of psychological reports. Such comparative studies can be conducted in further research in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0133] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0134] Figure 1 It is a flow chart of generating psychological semantic scene images of the present invention.

[0135] Figure 2 This is an example diagram of the hierarchical semantic mapping of the present invention.

[0136] Figure 3 It is a diagram of the three levels in the present invention.

[0137] Figure 4 It is the theoretical model in the present invention.

[0138] Figure 5 It is the feedback from the users of the present invention.

[0139] Figure 6 It is the feedback from the users of the present invention.

[0140] Figure 7 It is a schematic diagram of the scene diagram and text description in Example 1 of the present invention.

[0141] Figure 8 It is a schematic diagram of a scene diagram and text description in Example 2 of the present invention. DETAILED DESCRIPTION

[0142] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0143] This paper adopts an emerging research direction, namely Multimodal Large Language Model (MLLM). MLLM uses a powerful Large Language Model (LLM) as a "brain" to perform multimodal tasks. The emergence of MLLM, such as the ability to write stories based on images and mathematical reasoning without OCR, has made it possible to solve multimodal tasks. [1] , which is rare in traditional methods, indicating a potential path to artificial general intelligence. This invention applies this method paradigm to the metaphorical visualization of psychological reports for the first time. By combining the large language model (LLM) and the text-to-image model (T2I Model), it is the first time to generate semantic scene graphics and captions corresponding to the psychological state from the factor scores of the scale test results. This method can display the individual's psychological state more intuitively and vividly, which helps non-professionals understand and interpret the scale results. At the same time, this method breaks through the limitations of rule-based methods and can synthesize more realistic and readable results.

[0144] like Figure 1 As shown, the present invention designs a method for generating a semantic scene image from a psychological scale, and the method includes the following contents:

[0145] 1. Psychological scale: This paper proposes a psychological scale method, which evaluates students' mental health status from a comprehensive system perspective and provides appropriate solution dimensions. By combining online and offline questionnaires, primary and secondary school students are asked to fill in the questionnaire.

[0146] A total of 3,408 questionnaires were collected, of which 941 were eliminated based on the polygraph questions and the short test time. 2,467 valid questionnaires remained, with an effective rate of 72.4%.

[0147] Reliability and validity analysis

[0148] Reliability

[0149] The reliability and validity analysis was conducted on the data of this sample (n=2467), and the internal consistency coefficient (Cronbach's α) of the questionnaire was 0.855. The reliability coefficients of each dimension are shown in the table below.

[0150] Reliability table of each dimension (Cronbach's α)

[0151]

[0152] Construct validity

[0153] The data of this sample (n = 2467) were used for confirmatory factor analysis (CFA). The results showed that: 2 / df=3.86,TLI=0.929,CFI=0.941,RMSEA=0.034. According to the standards proposed by Wen Zhonglin and other psychology researchers, the results show that the 11-dimensional model fits well, is an ideal model, and has good structural validity.

[0154] Criterion-Related Validity

[0155] The criterion-related validity analysis was conducted on the important dimensions of this questionnaire. The criterion scales or dimensions were selected from the Symptom Checklist SCL-90, the Mental Health Scale CORE, the Health Questionnaire PHQ-9, the Post-Traumatic Stress Disorder Self-Assessment Scale PCL-C, the Psychological Resilience Scale CD-RISC-10, etc. The homogeneity reliability of the above scales is above 0.8, and they are well-known professional scales recognized at home and abroad.

[0156] The correlation coefficients between each important dimension and the criterion scale dimension are shown in the table below.

[0157] Criterion-related validity correlation coefficient table

[0158]

[0159] Note: ** indicates that r is statistically significant at the 0.01 level

[0160] The results show that the self-developed scale has good criterion validity and can effectively measure the mental health status of primary and secondary school students.

[0161] 2. Hierarchical semantic mapping ψ: Since the results generated by the T2I model lack controllability, each factor level A is defined k Corresponding background style B k and animal object O k Specific examples include Figure 2 The psychological factor scores output by the psychological scale are mapped to different objects according to classification, thus forming a basic semantic concept group {O k ,B kThe mapping rule is a hash table, and the specific method classification is iterated through discussions among psychology experts and feedback from test results.

[0162] 3. Prompt Engineering: A structured prompt word is constructed. The prompt word is used to (1) generate the page elements and background images required for the final semantic scene graph and (2) generate the caption corresponding to the semantic scene image. The caption is a descriptive text. All prompt engineering consists of three parts: role positioning, task objectives, and context.

[0163] 3.1. Synthesis of descriptive text: The generated descriptive text covers the input data, the professional description of the basic factors of the psychological scale, and the interpretation of the elements in the image.

[0164] In terms of the professional description of psychological factors, the function of OpenAI is used to build a description template for professional reports, and the corresponding positions are left blank, and then the function is required to return in json format to fill the blanks. The term explanation is used as context input to the text synthesis model to help the model understand the input value. The professional description is the output of the text synthesis model and is presented to the user. The following is an example of a prompt project:

[0165] The prompt words generated by the image in the present invention do not need to be positioned.

[0166] Role: “You are a professional psychoanalyst who will explain the results of the psychological scale to users in a graphic way with an image.”

[0167] Task objective: "You need to introduce a semantic scene image based on the results of the psychological scale. The introduction content includes the page elements in the scene and the corresponding psychological state, the scene background and the corresponding psychological factor state. Describe these results in a narrative and easy-to-understand context."

[0168] · Context: "Psychological scale is {A k :D k}, the user test result is {A1,A2,…,A n}, factor result A k The corresponding animal object is O k , background is B k "Among them, D k Represents factor A k Context is a part of the prompt word. The complete prompt word includes role positioning, task objectives and context. Generally, context refers to the necessary information to complete the task objectives. Each submission may change according to user information. Role positioning is mainly a description of rules.

[0169] Function output template: "Dear user, your mental picture is as shown in the figure, TEXT1based on{O k ,B k}, these show that you have encountered TEXT2based on A k , we recommend you TEXT2based on A k . The final output of the function output template is the caption text.

[0170] ·

[0171] 3.2. Page elements: for each group of elements (basic semantic concept groups, such as Figure 2 Each psychological scale level corresponds to a semantics) and automatically generates a prompt word template for generating images. Then the prompt word is input into the text-to-image model to generate basic semantic concept elements. After that, the segment anything algorithm is used to [2] The semantic entities are automatically segmented to form a bitmap with a transparent background. The page element refers to an animal object.

[0172] Role positioning is used to provide ChatGPT with reasonable prompts to better complete the task objectives. The training data source of ChatGPT ends in 2019, and the Text2Image models after that, such as StableDiffusion, are not used by ChatGPT. Therefore, in the prompt word engineering of page elements, role positioning needs to let ChatGPT understand the grammatical rules of the prompt words required by StableDiffusion.

[0173] · Character positioning: "StableDiffusion is a deep learning-based text-based image model that supports generating new images by using prompt words to describe elements to include or omit. I introduce the concept of Prompt in the StableDiffusion algorithm, also known as prompts. The following prompts are used to guide the AI ​​painting model to create images. They contain various details of the image, such as the appearance of the characters, the background, the color and lighting effects, as well as the theme and style of the image. The format of these prompts often contains weighted numbers in brackets to specify the importance or emphasis of certain details. For example, "(masterpiece: 1.5)" indicates that the quality of the work is very important, and multiple brackets have similar effects. In addition, if you use brackets, such as "{blue hair: white hair:0.3}", which means that blue hair and white hair are combined, and the proportion of blue hair is 0.3. The following are examples of using prompts to help AI models generate images: masterpiece, (best quality), highly detailed, ultra-detailed, cold, solo, (1 girl), detailed eyes, shining golden eyes) (long liver hair) expressionless, (long sleeves, puffy sleeves), (white wings), shining halo, (heavy metal:1.2), (metal jewelry), cross-laced footwear (chain), (white doves:1.2) Optional prompts include: Color - light, dark, pale, deep; Weather and time - golden hour lighting, strong rim light, intense shadows, in the rain, rainy days, sunset, cloudy; Buildings - in the baroque architecture (a decorative style in Renaissance Italy with a free form, pursuing motion, and fond of richness), in the romanesque architecture streets, in the palace, at the castle (with the exterior of the castle as the background), in the castle (with the interior of the castle as the background), in the street, in the cyberpunk city, rainy night in a cyberpunk city with glowing neon lights, at the lighthouse, in misty onsen, by the moon, in a bar, in bars, in a tavern, Japanese arch, in a locker room; Mountains - on a hill, the top of the hill; Seas - on the beach, over the sea, beautiful purple sunset at beach, in the ocean, on the ocean (on a ship).

[0174] · Task objective: "You need to generate a specific object based on the following text."

[0175] · Context: "The style of this object comes from {A k :D k ,O k}."

[0176] 3.2. Background image: In the background image section, modify the task objective to "You need to generate a scene picture based on the following text." Modify the context to "The style of this object comes from {A k :D k ,Bk}. "

[0177] 4. Perturbation of generated results: In order to improve the diversity of semantic images, work was done before and after the network, including adding neutral semantic objects to the basic semantic concept group, and allowing the generation model to add richer associative diagrams while ensuring overall semantic consistency.

[0178] Differentiable graphics synthesizer: It uses a differentiable composition method to merge the background image and the page elements corresponding to the basic semantic concept group according to the appropriate layout. Compared with the traditional direct combination method, this method can produce fewer combination artifacts, and softly drive the synthesis effect by constructing loss, thereby optimizing the entire image and eliminating various artifacts and other problems.

[0179] The key points of the technology of the present invention include the following aspects:

[0180] Cross-modal generative model: The technology of the present invention belongs to a cross-modal generative model from text to graphics. It takes the evaluation result data of the psychological scale as input to generate semantic scene graphics and descriptive text that can express the psychological state displayed by the psychological scale. The application of this cross-modal generative model is the first attempt in the metaphorical visualization of psychological reports.

[0181] Hierarchical semantic mapping: In order to control the controllability of the generated results, the technology of the present invention defines the background style and animal objects corresponding to each psychological factor level, and maps the psychological factor scores output by the psychological scale to different objects according to classification to form a basic semantic concept group. This hierarchical semantic mapping method makes the generated semantic scene image more interpretable and understandable.

[0182] ·Prompt Engineering: The technology of the present invention uses structured prompt engineering, including the synthesis of description text, the generation of page elements and the generation of background images. By reasonably constructing and filling description text templates, generating basic semantic concept elements and background images, more professional and readable semantic scene images and description texts can be generated.

[0183] Differentiable graphics synthesizer: In order to optimize the image synthesis effect, the method of the present invention adopts a differentiable composition method to merge the background image and the page elements corresponding to the basic semantic concept group according to a suitable layout. This method can reduce artifacts in the synthesis process, and softly drive the synthesis effect by constructing loss, optimize the entire image, and eliminate various artifacts and other problems.

[0184] These key points together constitute the core features and innovations of the technology of the present invention, which enable the generated semantic scene images to better reflect the user's psychological state and have higher interpretability and readability. Adaptive fine-tuning based on user interaction data makes the semantic direction more accurate.

[0185] Example 1

[0186] After a student answers 36 questions, the system automatically generates the following Figure 7 The scene diagram and text description are shown.

[0187] Example 2

[0188] After a student answers 36 questions, the system automatically generates the following Figure 7 The scene diagram and text description are shown.

[0189] √The report of students participating in the assessment is presented in the form of a scene diagram. We hope that children can take their own assessment reports and have further communication with the psychological teachers. Assessment is only a form, and communication is the goal. Through face-to-face communication between psychological teachers and students, we can further identify and clarify students’ psychological problems and levels.

[0190] √Thank you for answering all the questions carefully and telling us your feelings about growing up. We read in your report that you are somewhat unhappy and uncomfortable. Are you willing to talk to me about these things that make you feel uncomfortable?

[0191] √I read that you have had thoughts of self-harm. Is this true? How long have you had these thoughts? If 0-9 represents the degree of this thought, how serious do you think you are now?

[0192] √We hope that through assessment, we can encourage psychological teachers (every teacher) to have more contact with students, rather than simply referring students with problems.

[0193] The generation process of the psychological semantic scene image in this embodiment is as follows:

[0194] In the first step, students complete the 36 questions of the psychological scale and select the corresponding option for each question.

[0195] In the second step, the system collects all the options of the students and automatically calculates the original scores of all dimensional factors in the scale based on the preset question-factor correspondence. For example, the original score of the self-harm tendency factor calculated through relevant questions is 72.

[0196] In the third step, the system divides the intervals according to the original score set for each factor and determines the corresponding three-level level. For example, the original score of 72 belongs to the severe abnormal level, which is the first level. The three-level level of each factor is calculated in the same way.

[0197] In the fourth step, the system matches the corresponding semantic concept group according to the three-level results of each factor through the preset mapping relationship between the three-level level and the semantic concept group. For example, the first level of self-harm tendency is mapped to {background: cloudy day, animal: crow}.

[0198] In the fifth step, the system uses the matched semantic concept group, as well as the preset role positioning and task target templates, to automatically construct text prompt words and image prompt words.

[0199] In the sixth step, the system inputs the above prompt words and generates a text report and semantic scene image, including background pictures, animal elements and caption text, through a pre-trained neural network model.

[0200] In the seventh step, the system uses a differentiable image synthesis model to adjust and fuse the image generation results, and finally synthesizes a scene image with clear semantic meaning.

[0201] In the eighth step, the system combines the generated semantic scene image and the corresponding text report into a psychological assessment report according to the preset format and presents it to the user.

[0202] References

[0203] [1]Yin S, Fu C, Zhao S, et al.A Survey on Multimodal Large LanguageModels[J].arXiv preprint arXiv:2306.13549, 2023.

[0204] [2] Kirillov A, Mintun E, Ravi N, et al. Segment anything[J]. arXivpreprint arXiv: 2304.02643, 2023.

[0205] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.

Claims

1. A method for generating a psychosemantic scene image, characterized in that: The following steps are involved: Step 1: Based on the psychological scale filled in by the user, process the data to obtain the three-level scores of the psychological factors; Step 2: Construct basic semantic concept groups through hierarchical semantic mapping; Step 3: construct structured prompt words, including text prompt word engineering and image prompt word engineering; Step 4: Generate a caption based on the text prompt word through a large language model, and generate a background image and page elements based on the image prompt word through a text-to-graphics model; Step 5: Generate a semantic scene graph using the background image and page elements generated in step 4 through a differentiable graph synthesizer; Step 6: Generate a final psychological report by combining the generated captions and the semantic scene graph.

2. The method for generating a psychosemantic scene image according to claim 1, characterized in that: In step one, the psychological scale includes: 4 dimensions and 11 factors; among them, the self-harm tendency dimension is composed of 1 factor of self-harm tendency; the mental health dimension is composed of 4 factors of emotional state, sleep state, Internet dependence, and aversion to learning tendency; the inducement dimension is composed of 1 factor of traumatic events; the personality factor dimension is composed of 5 factors of interpersonal pleasing, interpersonal sensitivity, persistence, sense of efficacy and family relationship.

3. The method for generating a psychosemantic scene image according to claim 1, wherein: In step 2, the hierarchical semantic mapping refers to dividing the score strengths of different dimensional factors into three levels and then mapping them into different semantic concept groups; the basic semantic concept group is represented by {0 k ,B k }, where B k Indicates the background style, O k Represents animal objects, factor level A in the scale k Through mapping hash tables, they are respectively associated with the basic semantic concept groups {O k ,B k }One to one correspondence.

4. The method for generating a psychosemantic scene image according to claim 1, wherein: For the text prompt word project, the text prompt words include: role positioning, task objectives, context and language template; wherein, Role positioning is used to determine the role of the language model in the scenario as a psychological counselor for non-professional users; Task objectives are used to specify the goals or tasks in the scene and interpret the psychological test results through picture elements; The context and language template are used to provide background information and language templates of the scene, and are used to generate image captions for the large language model LLM; the captions include: an introduction to the semantics of the image and the scene and an analysis of the user's psychological state; wherein the context includes the user's psychological scale results and psychological semantic concept groups; the language template is designed according to the image interpretation method of the scene and the format of the psychological interpretation; and / or, For the image prompt word project, the image prompt words include: role positioning, task objectives, context; among them, Role positioning: Determine the role of the language model in the scenario as the prompt engineer of the Stable diffusion image, including the format of the general Stable Diffsuion model input prompt words; Task goal: specify the goal or task in the scene, and write the prompt words for the generated picture through the semantic concept group; Context and language template: The context includes the current background or the corresponding introduction of the page element; it is used to generate the required background image and the psychologically corresponding page elements for the text-to-image generation model (T2I Model).

5. The method for generating a psychosemantic scene image according to claim 4, characterized in that: The process of constructing text prompt words includes: Determine the role positioning and task objectives: Position the language model as a psychological counselor for non-professional users, and specify the task objective as interpreting psychological test results through image elements; Design context and language templates: Provide background information of the scene, including the user's psychological scale results and psychological semantic concept groups; design appropriate language templates for generating captions, which include an introduction to the semantics of the image and the scene and an analysis of the user's psychological state; Providing context and language templates to the large language model (LLM): The context and language templates in the text prompt words are provided to the large language model to generate captions related to the psycho-semantic scene image.

6. The method for generating a psychosemantic scene image according to claim 4, wherein: The construction process of image prompt words includes: Determine the role positioning and task objectives: Position the language model as the prompt engineer of the Stablediffusion image, and specify the task objective as using semantic concept groups to write the prompt words required to generate images; Design context and language templates: Provide appropriate context information, including the introduction of the current background or page elements. Design a common language template to generate the required background images and psychologically corresponding page elements; Providing context and language templates to the text-to-image generation model T2I: Provide the context and language templates in the image prompt words to the text-to-image generation model to generate background images and page elements that conform to the psychological semantic scene.

7. The method for generating a psychosemantic scene image according to claim 1, wherein: In step 4, the captions are generated through the large language model based on the constructed text prompts, including: input data, professional descriptions of the basic factors of the psychological scale, and interpretation of the elements in the image; The input data refers to the level classification of each dimension factor calculated from the options of each question selected by the user; The professional description of the basic factors of the psychological scale refers to the interpretation of the psychological scale results obtained by combining the psychological definition of each factor as the context of the prompt word with the introduction template; The interpretation of elements in the image refers to parsing and describing the image elements that appear in the scene to express the meaning, emotion or state related to the results of the psychological scale; The generation of the legend comprises the following steps: Step 4.1: Give the large language model a specific role and provide specific task objectives; Step 4.2, providing context information for generating a caption; the context information specifically refers to a simple and unprocessed introduction corresponding to the current background image or page element, including an animal object corresponding to the semantic concept group; Step 4.3: Generate a caption using a preset function output template based on the role positioning, task objectives, and context information given above.

8. The method for generating a psychosemantic scene image according to claim 1, wherein: In step 4, generating background images and page elements through a text-to-graphics model according to the constructed image prompt words includes the following steps: Step 4.4: Give the text-to-image model specific task objectives and context information for generating page elements and background images; Step 4.5, the text-to-image model generates page elements according to the task objectives and context information of generating page elements; Step 4.6: The text-to-image model generates a background image based on the task objective and context information of generating a background image.

9. The method for generating a psychosemantic scene image according to claim 7 or 8, characterized in that: Before and / or after generating the captions, background images, and page elements in step 4, disturbances are introduced to enrich the generation; wherein, The pre-perturbation includes: adding neutral semantic objects to the basic semantic concept group so that the generated results contain other animals that fit the background picture; and / or, Post-perturbation includes: shuffling the positions of the generated page elements according to random numbers and synthesizing them into the background.

10. The method for generating a psycho-semantic scene image according to claim 1, wherein: In step 5, the differentiable graph synthesizer is a tool for synthesizing semantic scene graphs, and the steps of establishing the differentiable graph synthesizer include: Step 5.1: The differentiable graph synthesizer architecture is: Input image -> encoder -> feature representation -> decoder -> output image; Step 5.2: Design loss function; The loss function is the key to ensure the quality of the synthesized image; when designing the loss function, the following parts are considered: -Pixel reconstruction loss: L_{pixel}=||I_c-I_{target}||_2 - Content loss: L_{content}=||V(I_c)-V(I_{target})||_2 - Adversarial loss: L_{adv} = log(1-D(G(I_c))) Among them, I_c represents the synthetic image, I_target represents the target image, V represents the feature extractor, G represents the generation network, and D represents the discriminant network; Step 5.3: Construct training data In order to train the network, a large number of input image and target image pairs are constructed as training data; the input image contains various elements and backgrounds, and the target image is a scene image with the elements correctly synthesized; the training data helps the network learn how to extract useful features from the input image and generate high-quality target images based on these features; Step 5.4: Choose appropriate background images and page elements In the process of generating images, appropriate background images and page elements are selected; the semantic content of the elements, as well as their layout and position in the background image, need to be considered; first, based on the input psychological scale results, the psychological factor scores are mapped to different objects and backgrounds using a hierarchical semantic mapping method to form a basic semantic concept group; then, appropriate background images and page elements are selected based on the basic semantic concept group; Step 5.5: Calculate the integration degree between the element and the background Calculating the degree of integration between elements and background is an important step to help determine the matching degree between elements and background; calculating the degree of integration between elements and background through some metrics, such as similarity of color, texture, and shape; using color histogram, texture features, shape features and other methods to calculate the degree of integration between elements and background; Step 5.6: Optimize the fusion effect After calculating the degree of fusion of the element and the background, the fusion effect is optimized; this is achieved by adjusting the position, size, shape attributes of the element, and possible background adjustments; the position, size, and shape attributes of the element are adjusted through the gradient descent optimization method to maximize the degree of fusion of the element and the background; at the same time, the fusion effect is further optimized by adjusting the brightness, contrast and other attributes of the background.

Images