Psychological semantic scene image generation system based on differentiable graph synthesizer and application

Abstract

The invention provides a psychological semantic scene image generation system based on a differentiable graph synthesizer, and the system comprises a level score calculation module which is used for processing data according to a psychological scale filled by a user to obtain a three-level level score of a psychological factor; the semantic construction module is used for constructing a basic semantic concept group through hierarchical semantic mapping; the cue word construction module is used for constructing structured cue words; the cue word generation module is used for generating an image note through a large language model according to the text cue word and generating a background image and page elements through a text-to-graph model according to the image cue word; the semantic scene generation module is used for generating a semantic scene graph from the generated background picture and page elements through a differentiable graph synthesizer; and the psychological report generation module is used for generating a final psychological report in combination with the generated graph annotation and the semantic scene graph.

Description

Technical Field

[0001] The present invention belongs to the technical field of semantic image generation, and relates to a psychological semantic scene image generation system based on a differentiable graphic synthesizer and an application thereof. Background Art

[0002] In existing technologies, psychological reports are typically presented in text format, which can be difficult for non-professionals to understand and interpret. To address this issue, some studies have attempted to present data using graphs or images, such as bar charts and pie charts, to display questionnaire results. However, these methods typically only display the distribution and proportion of data and fail to intuitively reflect an individual's psychological state.

[0003] Existing psychological scales, such as the Big Five Personality Traits Scale, Beck Anxiety Inventory, and Beck Depression Inventory, can quantify an individual's psychological state, but they have the following defects: (1) Subjectivity: The results of the scales often depend on the subjects' self-reports and may be affected by the subjects' subjective biases and misunderstandings; (2) Difficulty in interpretation: The results of the scales are usually presented in numerical form, which may be difficult for non-professionals to understand and interpret.

[0004] Furthermore, existing cross-modal psychology data visualization methods are primarily rule-based, and their results are limited by expert experience, making it difficult to cover all test scenarios. The resulting infographics lack realism and readability. For example, while projective tests in psychoanalysis (such as the Rosenseck Inkblot Test and the Thematic Apperception Test) and drawing therapy emphasize the relationship between visual and psychological states, interpreting their results typically requires specialized training and experience. On the other hand, while expression recognition methods emphasize the relationship between visual and psychological states, their accuracy is often limited by the complex relationship between expression and emotion. Summary of the Invention

[0005] In order to address the deficiencies in the prior art, the purpose of the present invention is to provide a method, system and application for generating psychological semantic scene images. The present invention proposes a method for generating psychological semantic scene images from psychological scale results, which is used to visualize the user's psychological state with semantic metaphors in psychological reports. The method of the present invention applies a text-to-graphic cross-modal generation model, and through scale data analysis, constructs a mapping of scale data to semantics, a method of semantics to text templates, a method of semantics to graphic elements, and randomized picture synthesis to achieve the generation of psychological semantic scene images. The input of the method of the present invention is the psychological scale evaluation result data, and the output is a semantic scene graphic and corresponding descriptive text that can represent the psychological state displayed by the psychological scale.

[0006] The present invention provides a psychosemantic scene image generation system, comprising:

[0007] The level score calculation module is used to process the data based on the psychological scale filled in by the user to obtain the three-level level scores of the psychological factors;

[0008] Semantic construction module, which constructs basic semantic concept groups through hierarchical semantic mapping;

[0009] A prompt word construction module, which is used to construct structured prompt words;

[0010] A prompt word generation module, which is used to generate a caption based on the text prompt word through a large language model, and generate a background image and page elements based on the image prompt word through a text-to-graphics model;

[0011] A semantic scene generation module, which is used to generate a semantic scene graph by using a differentiable graph synthesizer to generate the generated background image and page elements;

[0012] A psychological report generation module is used to generate a final psychological report by combining the generated annotations and the semantic scene graph.

[0013] In the present invention, the system adopts the following method, which includes the following steps:

[0014] Step 1: Based on the psychological scale filled out by the user, process the data to obtain the three-level scores of the psychological factors.

[0015] The three-level scores refer to the original scores of the factors of each dimension of the psychological scale, which are divided into three different levels to reflect the score strength of the factor:

[0016] Level 3: Normal level. The raw score of this factor is within the normal range, reflecting the normal psychological status of this dimension.

[0017] Level 2: Mild Abnormal. The raw score for this factor is slightly above the normal range, reflecting a mild abnormality in the psychological state of this dimension. This requires attention but is not serious.

[0018] Level 1: Severe Abnormality. The raw score for this factor is significantly higher than the normal range, reflecting a severe abnormality in the psychological status of this dimension. This requires high attention and intervention measures.

[0019] If the value is more severe, it will be classified as a crisis. The specific raw score range can be determined by different psychological scales and experts. For example, if the raw score range of a factor is 0-100, 0-20 points can be set as the third level, 20-40 points as the second level, and 40-100 points as the first level.

[0020] Step 2: Construct a basic semantic concept group through hierarchical semantic mapping;

[0021] Step 3: Constructing structured prompt words, including text prompt word engineering and image prompt word engineering;

[0022] Step 4: Generate captions based on the constructed text prompt words through the large language model, and generate background images and page elements based on the constructed image prompt words through the text-to-graphics model;

[0023] Step 5: Generate a semantic scene graph using the background image and page elements generated in step 4 through a differentiable graph synthesizer;

[0024] Step 6: Generate the final psychological report by combining the captions generated in step 4 and the semantic scene graph generated in step 5.

[0025] In step 1, the psychological scale assesses students’ mental health status through a comprehensive system perspective and provides dimensions that need attention. Theoretical models (such as Figure 4 (as shown) fully considers the level of mental health, level of resilience and intervention mechanism, understands and identifies the different causes of risk warning, and provides direction for targeted counseling and intervention. This is not available in existing psychological scales. The scale is divided into 4 dimensions and 11 factors. The self-harm tendency dimension is composed of 1 factor of self-harm tendency; the mental health dimension is composed of 4 factors of emotional state, sleep state, Internet dependence, and tendency to dislike school; the inducement dimension is composed of 1 factor of traumatic events; the personality factor dimension is composed of 5 factors of interpersonal pleasing, interpersonal sensitivity, persistence, sense of efficacy and family relationship. As long as the score of one factor in each dimension exceeds the norm reference score, it is considered to be detected. Then, according to the detection situation of the four dimensions and matching with the table below, the three levels A1, A2,…, A are finally determined. k , where k represents the number of factors, A i Represents the three-level strength classification of the i-th factor in the scale results.

[0026] The complete scale is as follows:

[0027] Student Life, Study, and Health Status Questionnaire (Admission Version)

[0028]

[0029]

[0030] The corresponding relationship between the items in the above scale and each factor:

[0031] Emotional status: items 12, 15, 16, 20, 27, and 34

[0032] Sleep status: items 12 and 15

[0033] Internet dependence: items 23, 26, and 35

[0034] ·Tendency to dislike studying: Items 17, 30, and 33

[0035] Self-harm tendency: items 13, 22, and 31

[0036] Traumatic events: Items 24 and 25

[0037] Interpersonal Ingratiation: Items 9, 18, and 28

[0038] Interpersonal sensitivity: items 7 and 19

[0039] Persistence: Items 2, 8, and 32

[0040] Sense of efficacy: Items 5 and 6

[0041] Family relationships: items 4, 21, and 36

[0042] The dimensions and factor structure of the scale are as follows:

[0043] Self-harm tendency dimension: Self-harm tendency factor

[0044] Mental health dimensions: emotional state, sleep condition, internet addiction, and factors related to aversion to learning

[0045] · Predisposing dimension: Traumatic event factor

[0046] Personality factor dimensions: interpersonal pleasing, interpersonal sensitivity, persistence, sense of efficacy, and family relationship factors

[0047] The scale is divided into four dimensions and 11 factors. The self-harm tendency dimension consists of one factor, namely, self-harm tendency; the mental health dimension consists of four factors, namely, emotional state, sleep state, internet addiction, and school aversion; the trigger dimension consists of one factor, namely, traumatic events; and the personality factor dimension consists of five factors, namely, interpersonal pleasing, interpersonal sensitivity, persistence, sense of efficacy, and family relationships. A score exceeding the norm for any one factor in each dimension is considered detected. The detected scores for the four dimensions are then compared with the table below to determine the final three-level level.

[0048] Table 1 Three-level classification

[0049] Classification results Self-harm tendencies Mental Health Causes Personality factors Class III A √ √ √ Class III B √ √ Class III C √ √ Class III D √ Class II A √ √ Class B √ Class II C √ Level 1 crisis √

[0050] In step 2, the hierarchical semantic mapping refers to dividing the score strength of different dimensional factors into three levels (divided into 9 categories in Table 1), and the score strength is obtained by weighted linear combination, and then mapped into different basic semantic concept groups; the basic semantic concept group is represented by {O k ,B k}, where B k Indicates the background style, O kRepresents animal objects, the third level A of the factor in the scale k By mapping hash tables respectively with the basic semantic concept group {O k ,B k}One-to-one correspondence; in the present invention, only the background style and animal objects are considered, without specifically considering their more general forms of expression.

[0051] In step 3, the generation of the structured prompt words further includes the following:

[0052] For the text prompt word project, the text prompt words include: role positioning, task objectives, context and language templates; the context and language templates are used to generate image captions for the large language model (LLM), and the captions include: an introduction to the semantics of the image and the scene and an analysis of the user's psychological state;

[0053] Role positioning: Determine the role of the language model in the scenario as a psychological counselor for non-professional users.

[0054] Task Objective: Specify the goal or task in the scene and interpret the psychological test results through picture elements.

[0055] Context and Language Template: This provides background information and language templates for generating captions. Context includes the user's psychological scale results and psychological semantic concept groups, while the language template is designed based on the scene's image interpretation and the format of psychological explanations.

[0056] For the image prompt word project, the image prompt word includes role positioning, task objectives, and context, which are used to generate the required background image and psychologically corresponding page elements for the text-to-image generation model (T2I Model); the language template is part of the context and is used to define the answer format.

[0057] Role positioning: Determine the role of the language model in the scenario as the prompt engineer of the stable diffusion image, including the format of the general Stable Diffsuion model input prompt word.

[0058] Task goal: Specify the goal or task in the scene and write prompt words to generate images through semantic concept groups.

[0059] Context and language template: The context includes the current background or the introduction corresponding to the page element, and the language template used to define the output format.

[0060] The generalized specific construction process of text prompt words:

[0061] Determine the role positioning and task objectives: Position the language model as a psychological counselor for non-professional users, and specify the task objective as interpreting psychological test results through image elements.

[0062] Design context and language templates: Provide background information for the scene, including the user's psychological scale results and psychological semantic concept groups. Design appropriate language templates for generating captions, including an introduction to the image and scene semantics and an analysis of the user's psychological state.

[0063] Providing context and language templates to the Large Language Model (LLM): The context and language templates in the text prompt words are provided to the Large Language Model (LLM) to generate captions related to the psychosemantic scene images.

[0064] The generalized and specific construction process of image prompt words:

[0065] Determine role positioning and task objectives: Position the language model as the Prompt engineer for Stablediffusion images, and specify the task objective as using semantic concept groups to write the prompt words required to generate images.

[0066] Design context and language templates: Provide appropriate context information, including an introduction to the current background or page elements. Design a common language template to generate the required background images and psychologically corresponding page elements.

[0067] Providing context and language templates to the text-to-image generation model T2I: Provide the context and language templates in the image prompt words to the text-to-image generation model (T2I Model) to generate background images and page elements that conform to the psychological semantic scene.

[0068] In step 4, the captions are generated using a large language model based on the constructed text prompts, including: input data, professional descriptions of the basic factors of the psychological scale, and interpretations of the elements in the image;

[0069] The input data refers to the three-level classification of each dimension factor calculated based on the options of each question selected by the user;

[0070] The professional description of the basic factors of the psychological scale refers to the interpretation of the psychological scale results obtained by combining the psychological definition of each factor as the context of the prompt word with the introduction template;

[0071] The interpretation of elements in the image refers to parsing and describing the image elements appearing in the scene to express the meaning, emotion or state related to the results of the psychological scale.

[0072] The generation of the legend comprises the following steps:

[0073] Step 4.1: Give the large language model a specific role and provide specific task objectives;

[0074] Step 4.2: providing context information for generating captions; the context information specifically refers to a simple, unprocessed introduction corresponding to the current background image or page element, including the animal object corresponding to the semantic concept group;

[0075] Step 4.3: Generate a caption using a preset function output template based on the role positioning, task objectives, and context information given above.

[0076] In step 4, generating background images and page elements through a text-to-graphics model based on the constructed image prompt words further includes the following steps:

[0077] Step 4.4: Give the text-to-image model specific task objectives and contextual information for generating page elements and background images;

[0078] Step 4.5: The text-to-image model generates page elements based on the task objectives and context information.

[0079] Step 4.6: The text-to-image model generates a background image based on the task objective and contextual information of generating a background image.

[0080] Among them, the page elements are animal objects and the background is the name of the season scene.

[0081] Before and after generating the captions, background images, and page elements in step 4, perturbations are introduced to enrich the generation.

[0082] Perturbations can be added before and after the generative model. Specifically, pre-perturbations involve adding neutral semantic objects to the basic semantic concept group, allowing the generated result to include other animals that fit the background image. Post-perturbations are responsible for synthesizing page elements into the background using random numbers to shuffle their positions.

[0083] Pre-perturbation includes the following methods:

[0084] Adding neutral semantic objects to the basic semantic concept group allows the generated results to include other animals that fit the background image. This can introduce more elements into the image and increase the diversity of the generated results.

[0085] Post-perturbation involves the following methods:

[0086] The generated page elements are randomly shuffled and composited into the background. This changes the arrangement of page elements and makes the generated semantic scene graph more diverse and changeable.

[0087] By introducing front-end and back-end perturbations, the generated captions, background images, and page elements can be richer and more diverse, increasing the diversity and variability of the generated results.

[0088] In step 5, the differentiable graph synthesizer is a tool for synthesizing semantic scene graphs. It has the following definitions and functions:

[0089] Definition: A differentiable graph compositor is an image synthesis model based on differentiable operations that can combine page elements and background images to generate a scene graph with semantic meaning.

[0090] Purpose: The Differentiable Graphics Synthesizer generates a semantic scene graph that meets semantic requirements by combining and adjusting page elements and background images. It synthesizes images based on input parameters and constraints while maintaining the model's differentiability, enabling the synthesis process to be optimized through gradient backpropagation.

[0091] The specific process of combining page elements and background images to generate a semantic scene graph through a differentiable graph synthesizer is as follows:

[0092] 1. For a given page element and background image, first embed the page element into the appropriate position in the background image.

[0093] 2. Based on the embedded page elements and background images, synthesis is performed through differentiable operations to adjust the appearance and position of the elements and fuse them with the background image to generate a synthetic semantic scene graph.

[0094] 3. Iteratively optimize the synthesized semantic scene graph and use gradient backpropagation to minimize the difference with the target image until a semantic scene graph that meets the requirements is obtained.

[0095] The steps to build / train a differentiable graph synthesizer include:

[0096] Step 5.1 The network architecture is shown as follows:

[0097] Input image -> encoder -> feature representation -> decoder -> output image

[0098] In addition, the specific structures of the encoder and decoder are as described in this specification.

[0099] Step 5.2: Design a loss function

[0100] The loss function is key to ensuring synthesis quality. The design can include:

[0101] Pixel reconstruction loss: L_{pixel}=||I_c-I_{target}||_2

[0102] Content loss: L_{content}=||V(I_c)-V(I_{target})||_2

[0103] Adversarial loss: L_{adv}=log(1-D(G(I_c)))

[0104] Among them, I_c is the synthetic image, I_target is the target image, V is the feature extractor, G is the generation network, and D is the discriminant network.

[0105] Step 5.3: Construct training data

[0106] To train the network, a large number of input and target image pairs are required as training data. The input images contain various elements and backgrounds, and the target images are scenes in which these elements are correctly synthesized. This training data effectively helps the network learn how to extract useful features from the input images and generate high-quality target images based on these features.

[0107] Step 5.4: Choose the right background image and page elements

[0108] When generating images, choosing appropriate background images and page elements is crucial. The semantic content of the elements, as well as their layout and position within the background image, must be considered. To achieve this, we first use a hierarchical semantic mapping method based on the input psychological scale results to map psychological factor scores to different objects and backgrounds, forming a basic semantic concept group. Then, based on this basic semantic concept group, we select appropriate background images and page elements.

[0109] Step 5.5: Calculate the degree of integration between the element and the background

[0110] Calculating the degree of fusion between elements and background is an important step, as it helps determine the degree of match between the elements and background. This degree of fusion can be calculated using metrics such as color, texture, and shape similarity. Specifically, methods such as color histograms, texture features, and shape features can be used to calculate the degree of fusion between elements and background.

[0111] Step 5.6: Optimize the fusion effect

[0112] After calculating the degree of integration between the element and the background, the integration needs to be optimized. This is achieved by adjusting the element's position, size, shape, and other properties, as well as the background, if necessary. Specifically, optimization methods such as gradient descent can be used to adjust the element's position, size, and shape to maximize integration. Furthermore, adjustments to background properties such as brightness and contrast can be made to further optimize the integration.

[0113] Differentiable composition methods use differentiable operations to synthesize images, drawing on techniques from computer vision and graphics. Specifically, they use differentiable image operations such as image overlay, scaling, rotation, and translation, along with a series of differentiable functions and modules like convolutional neural networks, to synthesize and adjust page elements and background images.

[0114] Softly driving the synthesis effect by constructing a loss function means constraining the synthesis result to meet the desired effect by designing a loss function. Soft driving means that these constraints are not absolutely mandatory, but rather driven in a flexible manner. The specific form of the loss function can be designed based on the specific application scenario and requirements. Common ones include pixel-level difference loss, content loss, style loss, etc.

[0115] Image optimization and artifact removal are technical means and solutions used to improve synthesis quality and reduce artifacts in the resulting image. Specifically, image optimization involves iteratively optimizing the details, lighting, and color of the synthesized image to achieve superior visual quality. Artifact removal addresses artifacts that may arise during the synthesis process by employing a series of algorithms and techniques, such as edge smoothing and blending strategy adjustments, to reduce or eliminate them, thereby improving the quality and realism of the resulting image. Specific implementation methods can be selected and designed based on the specific synthesis task and requirements.

[0116] In step five, differentiable graph synthesizer, loss construction, image optimization, and artifact removal are involved. Examples of these mathematical formulas are shown in the figure:

[0117] 1. Differentiable Graphics Synthesizer:

[0118] -Image synthesis formula: Synthesized image I_c = f(I_b, E), where I_c represents the synthesized image, I_b represents the background image, and E represents the page element.

[0119] -Element position adjustment formula: E'=E+δE, where E' represents the adjusted page element and δE represents the position adjustment vector.

[0120] 2. Loss construction:

[0121] -Pixel-level difference loss: L_pixel = Σ|I_c-I_target|, where I_target represents the target image.

[0122] - Content loss: L_content = ||V(I_c)-V(I_target)||, where V represents the feature extractor.

[0123] -Style loss: L_style = Σ||G(I_c)-G(I_target)||, where G represents the style feature extractor.

[0124] 3. Image Optimization:

[0125] -Optimization formula: I_optimized = argmin(L_total(I_c)), where L_total represents the comprehensive loss function and I_c represents the generated image.

[0126] 4. Artifact elimination:

[0127] -Edge smoothing formula: I_smooth=f(I_optimized), where f represents the smoothing operator.

[0128] -Fusion strategy adjustment: I_final = g(I_smooth, I_target)$, where g represents the fusion strategy adjustment function.

[0129] The beneficial effects of the present invention include: the method of the present invention can present psychological scale data in an intuitive and easy-to-understand manner, helping users better understand and accept psychological reports. At the same time, the method can also provide more information for psychological counselors to help them better understand and interpret the user's psychological state.

[0130] 1. Intuitive and easy-to-understand presentation: This invention applies semantic concept groups and generative models to transform psychological scale data into intuitive and easy-to-understand images and text. By combining images and semantic cues, users can more intuitively understand and accept the content of the psychological report.

[0131] 2. Personalized customization: Based on the user's psychological scale data and characteristics, the present invention uses semantic concept groups and generative models to generate personalized psychological semantic scene images. This personalized customization method helps users better understand and identify with the information presented in the psychological report.

[0132] 3. Provide more information support: This invention not only displays the results of the psychological report to the user, but also provides more information support through structured prompt words and semantic scene images. This helps psychological counselors better understand and interpret the user's psychological state, providing more comprehensive and in-depth consulting services.

[0133] 4. Improved user participation and motivation: Through intuitive image display and personalized content, the present invention can improve users' participation and motivation in psychological testing and consultation, and enhance their understanding of their own psychological state and motivation to adjust.

[0134] 5. Enhancement of emotions and experience: This invention converts psychological scale data into images and text to create psychological semantic scenes with greater emotional resonance and immersion, thereby enhancing users' emotional experience and participation.

[0135] 6. Improved information transmission effect: Compared with traditional pure text or simple reports, the present invention can transmit psychological test results and related information more intuitively and comprehensively, improving the effect and efficiency of information transmission.

[0136] At present, since the present invention is a new innovative technology, there is no specific comparative data with the existing technology. Figure 5 、 6 As shown, the effects and advantages of the present invention can be evaluated through user feedback and evaluation studies to compare with traditional methods and prove its advantages in user understanding and acceptance of psychological reports. Such comparative studies can be conducted in further research in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0137] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0138] Figure 1 This is a flow chart of the psychological semantic scene image generation process of the present invention.

[0139] Figure 2 This is an example diagram of the hierarchical semantic mapping of the present invention.

[0140] Figure 3 It is a diagram of the three levels in the present invention.

[0141] Figure 4 It is the theoretical model in this invention.

[0142] Figure 5 It is the feedback from users of the present invention.

[0143] Figure 6 It is the feedback from users of the present invention.

[0144] Figure 7 This is a schematic diagram of the scene diagram and text description in Example 1 of the present invention.

[0145] Figure 8 This is a schematic diagram of a scene diagram and text description in Example 2 of the present invention. DETAILED DESCRIPTION

[0146] The present invention is further described in detail with reference to the following specific examples and accompanying drawings. The processes, conditions, experimental methods, etc. for implementing the present invention, except for those specifically mentioned below, are common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0147] This paper adopts an emerging research direction, namely Multimodal Large Language Model (MLLM). MLLM uses a powerful Large Language Model (LLM) as a "brain" to perform multimodal tasks. The emergence of MLLM, such as the ability to write stories based on images and mathematical reasoning without OCR, has [1] , which is rare in traditional methods, indicating a potential path to artificial general intelligence. This invention applies this method paradigm to the metaphorical visualization of psychological reports for the first time. By combining the large language model (LLM) and the text-to-image model (T2I Model), it is the first time to realize the generation of semantic scene graphics and captions corresponding to psychological states from the factor scores of the scale test results. This method can display the individual's psychological state more intuitively and vividly, which helps non-professionals understand and interpret the scale results. At the same time, this method breaks through the limitations of rule-based methods and can synthesize more realistic and readable results.

[0148] like Figure 1 As shown, the present invention designs a method for generating a semantic scene image from a psychological scale, which includes the following contents:

[0149] 1. Psychological scale: This paper proposes a psychological scale method, which uses a comprehensive system perspective to assess students' mental health and provide appropriate solution dimensions. By combining online and offline questionnaires, primary and secondary school students are asked to fill in the questionnaire.

[0150] A total of 3,408 questionnaires were collected, of which 941 were eliminated based on the polygraph questions and the short test time. The remaining valid questionnaires were 2,467, with an efficiency of 72.4%.

[0151] Reliability and validity analysis

[0152] Reliability

[0153] Reliability and validity were analyzed using data from this sample (n=2467), and the questionnaire's internal consistency coefficient (Cronbach's α) was 0.855. The reliability coefficients for each dimension are shown in the table below.

[0154] Reliability table of each dimension (Cronbach's α)

[0155]

[0156] Construct validity

[0157] The data of this sample (n=2467) were used for confirmatory factor analysis (CFA). The results showed that: 2 / df=3.86, TLI=0.929, CFI=0.941, RMSEA=0.034. According to the criteria proposed by Wen Zhonglin and other psychology researchers, the results show that the 11-dimensional model fits well, is an ideal model, and has good construct validity.

[0158] Criterion-related validity

[0159] Criterion-related validity analysis was conducted on the key dimensions of this questionnaire. The criterion scales or dimensions were selected from the Symptom Checklist-90 (SCL-90), the CORE Mental Health Inventory (CORE), the PHQ-9, the Post-Traumatic Stress Disorder Inventory (PCL-C), and the CD-RISC-10 Resilience Scale. All of these scales have homogeneity reliabilities exceeding 0.8, making them well-known and recognized professional scales both domestically and internationally.

[0160] The correlation coefficients between each important dimension and the criterion scale dimension are shown in the table below.

[0161] Criterion-related validity correlation coefficient table

[0162]

[0163]

[0164] Note: ** indicates that r is statistically significant at the 0.01 level

[0165] The results show that the self-developed scale has good criterion validity and can effectively measure the mental health status of primary and secondary school students.

[0166] 2. Hierarchical semantic mapping ψ: Due to the lack of controllability of the generated results of the T2I model, each factor level A is defined k Corresponding background style B k and animal object O k Specific examples include Figure 2 The psychological factor scores output by the psychological scale are mapped to different objects according to the classification, thus forming a basic semantic concept group {O k ,B k The mapping rule is a hash table, and the specific method classification is iterated through discussions among psychology experts and feedback from test results.

[0167] 3. Hint Engineering: A structured set of hint words is constructed. These hint words are used to (1) generate the page elements and background images required for the final semantic scene graph, and (2) generate the captions corresponding to the semantic scene images. The captions are descriptive text. All hint engineering consists of three parts: role positioning, task objectives, and context.

[0168] 3.1. Synthesis of descriptive text: The generated descriptive text covers the input data, professional descriptions of the basic factors of the psychological scale, and interpretations of elements in the image.

[0169] For the professional description of psychological factors, we used OpenAI's function to construct a description template for a professional report. We left the corresponding fields blank and then asked the function to return the content in JSON format, filling in the blanks. The term explanations serve as contextual input to the text synthesis model, helping the model understand the input values. The professional descriptions are the output of the text synthesis model and are presented to users. The following is an example of a prompt project:

[0170] The prompt words generated by the image in the present invention do not need to be positioned.

[0171] Role: "You are a professional psychoanalyst who will explain the results of a psychological questionnaire to users using an image."

[0172] Task Objective: "You need to describe a semantic scene image based on the results of the psychological scale. The description should include the page elements within the scene and the corresponding psychological states, the scene background and the corresponding psychological factor states. Describe these results in a storytelling and easy-to-understand context."

[0173] ·Context: "Psychological scale is {A k :D k}, the user test result is {A1,A2,…,A n}, factor result A k The corresponding animal object is O k , background B k "Among them, D k Indicates factor A k Context is part of the prompt. A complete prompt includes role positioning, task objectives, and context. Generally, context represents the necessary information to complete the task objectives and may change with each submission based on user information. Role positioning primarily describes the rules.

[0174] Function output template: "Dear user, your mental picture is as shown in the figure, TEXT1based on{O k ,B k}, these show that you have encountered TEXT2 based on A k , recommend you TEXT2based on A k The final output of the function output template is the caption text.

[0175] ·

[0176] 3.2. Page elements: for each group of elements (basic semantic concept group, such as Figure 2 Each psychological scale level corresponds to a semantic) and automatically generates a prompt word template for generating images. Then the prompt word is input into the text-to-image model to generate basic semantic concept elements. After that, the segment anything algorithm is used to [2] Semantic entities are automatically segmented to form a bitmap with a transparent background. The page element refers to an animal object.

[0177] Role localization is used to provide ChatGPT with appropriate prompts to better achieve its task objectives. ChatGPT's training data source expired in 2019, and subsequent Text2Image models, such as StableDiffusion, are not used by ChatGPT. Therefore, when designing prompts for page elements, role localization is crucial to ensuring ChatGPT understands the grammatical rules for the prompts required by StableDiffusion.

[0178] · Character Positioning: "StableDiffusion is a text-based image model that uses deep learning to generate new images using prompts to describe elements to include or omit. I introduce the concept of Prompt in the StableDiffusion algorithm here, also known as prompts. The following prompts are used to guide the AI ​​painting model to create images. They contain various details of the image, such as the appearance of the character, the background, the color and lighting effects, as well as the theme and style of the image. The format of these prompts often contains weighted numbers in brackets to specify the importance or emphasis of certain details. For example, "(masterpiece: 1.5)" indicates that the quality of the work is very important, and multiple brackets have similar effects. In addition, if brackets are used, such as "{blue hair: white hair: 0.3}", which means that blue hair and white hair are combined, with the blue hair accounting for 0.3%. The following are examples of using prompts to help AI models generate images: masterpiece, (best quality), highly detailed, ultra-detailed, cold, solo, (1 girl), detailed eyes, shine golden eyes) (long liver hair) expressionless, (long sleeves, puffy sleeves), (white wings), shine halo, (heavy metal: 1.2), (metal jewelry), cross-laced footwear (chain), (White doves: 1.2) Optional prompts include: Color - light, dark, pale, deep; Weather and Time - golden hour lighting, strong rim light, intense shadows, in the rain, rainy days, sunset, cloudy; Buildings - in the baroque architecture (a decorative style in Renaissance Italy with a free form, pursuing motion, and fond of richness), in the romanesque architecture streets, in the palace, at the castle (with the exterior of the castle as the background), in the castle (with the interior of the castle as the background), in the street, in the cyberpunk city, rainy night in a cyberpunk city with glowing neon lights, at the lighthouse, in misty onsen, by the moon, in a bar, in bars, in a tavern, Japanese arch, in a locker room; Mountains - on a hill, the top of the hill; Seas - on the beach, over the sea, beautiful purple sunset at beach, in the ocean, on the ocean (on a ship).

[0179] · Task objective: "You need to generate a specific object based on the following text."

[0180] · Context: "The style of this object comes from {A k :D k ,O k}."

[0181] 3.2. Background Image: In the background image part, modify the task objective to "You need to generate a scene picture based on the following text." Modify the context to "The style of this object comes from {A k :D k ,Bk}.”

[0182] 4. Perturbation of generated results: To improve the diversity of semantic images, work was done before and after the network, including adding neutral semantic objects to the basic semantic concept group, and allowing the generative model to add richer associative diagrams while ensuring overall semantic consistency.

[0183] Differentiable Graphics Synthesizer: This uses a differentiable composition method to combine a background image with page elements corresponding to a basic semantic concept group in an appropriate layout. Compared to traditional direct composition methods, this method reduces the occurrence of composition artifacts. It also softly drives the composition effect by constructing a loss, thereby optimizing the entire image and eliminating various artifacts.

[0184] The key points of the technology of the present invention include the following aspects:

[0185] Cross-modal generative model: This technology, a text-to-graphics cross-modal generative model, takes psychological scale assessment data as input and generates semantic scene graphics and descriptive text that represent the psychological states presented by the scale. This application of a cross-modal generative model is a first for metaphorical visualization of psychological reports.

[0186] Hierarchical semantic mapping: To ensure controllability of the generated results, this technology defines the background styles and animal objects corresponding to each psychological factor level. The psychological factor scores output by the psychological scale are then mapped to different objects according to categories, forming a basic semantic concept group. This hierarchical semantic mapping method makes the generated semantic scene images more interpretable and understandable.

[0187] Hint Engineering: This technology utilizes a structured hint engineering process, including the synthesis of description text, the generation of page elements, and the creation of background images. By rationally constructing and filling description text templates, generating basic semantic concept elements and background images, it can generate more professional and readable semantic scene images and description text.

[0188] Differentiable Graphics Synthesizer: To optimize image synthesis, this method uses a differentiable composition approach to fuse a background image with page elements corresponding to a basic semantic concept group in a suitable layout. This approach reduces artifacts during the synthesis process and softly drives the synthesis effect by constructing a loss function, optimizing the overall image and eliminating various artifacts.

[0189] These key points together constitute the core features and innovations of the present invention, enabling the generated semantic scene images to better reflect the user's psychological state and have higher interpretability and readability. Adaptive fine-tuning based on user interaction data makes the semantic direction more accurate.

[0190] Example 1

[0191] After a student answers 36 questions, the system automatically generates the following Figure 7 The scene diagram and text description are shown.

[0192] Example 2

[0193] After a student answers 36 questions, the system automatically generates the following Figure 8 The scene diagram and text description are shown.

[0194] √The report of students participating in the assessment is presented in the form of a scene diagram. We hope that children can take their own assessment reports and have further communication with the psychological teachers. Assessment is only a form, and communication is the goal. Through face-to-face communication between psychological teachers and students, we can further explore and clarify the students' psychological problems and their degree.

[0195] √Thank you for answering all the questions carefully and telling us about your feelings during your growth. We read in your report that you were somewhat unhappy and uncomfortable. Would you like to talk to me about these things that made you feel uncomfortable?

[0196] I read that you've had thoughts of self-harm. Is this true? How long have you had these thoughts? On a scale of 0-9, how serious would you say these thoughts are now?

[0197] √We hope that through assessment, we can encourage psychological teachers (every teacher) to have more contact with students, rather than simply referring students with problems.

[0198] The generation process of the psychosemantic scene image in this embodiment is as follows:

[0199] In the first step, students complete the 36 questions on the psychological scale and select the corresponding option for each question.

[0200] In the second step, the system collects all the student's responses and automatically calculates the raw scores for all the factors in the scale based on the pre-set item-factor correspondence. For example, the raw score for the self-harm tendency factor calculated from the relevant items is 72.

[0201] In the third step, the system divides each factor's raw score into intervals and determines the corresponding three-level level. For example, a raw score of 72 is considered severely abnormal, which is Level 1. The three-level level of each factor is calculated in the same way.

[0202] In the fourth step, the system uses the preset mapping relationship between the three-level level and the semantic concept group to match the three-level results of each factor to the corresponding semantic concept group. For example, the first level of self-harm tendency is mapped to {background: cloudy sky, animal: crow}.

[0203] In the fifth step, the system uses the matched semantic concept group and the preset role positioning and task target templates to automatically construct text prompt words and image prompt words.

[0204] In the sixth step, the system inputs the above prompt words and generates a text report and semantic scene image, including background pictures, animal elements and caption text, through a pre-trained neural network model.

[0205] In the seventh step, the system uses a differentiable image synthesis model to adjust and fuse the image generation results, and finally synthesizes a scene image with clear semantic meaning.

[0206] In the eighth step, the system combines the generated semantic scene image and the corresponding text report into a psychological assessment report according to the preset format and presents it to the user.

[0207] References

[0208] [1]Yin S, Fu C, Zhao S, et al.A Survey on Multimodal Large LanguageModels[J].arXiv preprint arXiv:2306.13549, 2023.

[0209] [2] Kirillov A, Mintun E, Ravi N, et al. Segment anything[J]. arXivpreprint arXiv: 2304.02643, 2023.

[0210] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the appended claims.

Claims

1. A psychosemantic scene image generation system based on a differentiable graphics synthesizer, characterized in that: include: A level score calculation module is used to process data based on the psychological scale filled in by the user to obtain the three-level level scores of the psychological factors; Semantic construction module, which constructs basic semantic concept groups through hierarchical semantic mapping; A prompt word construction module, which is used to construct structured prompt words; A prompt word generation module, which is used to generate a caption according to the text prompt word through a large language model, and to generate a background image and page elements according to the image prompt word through a text-to-graphics model; A semantic scene generation module, which is used to generate a semantic scene graph by using a differentiable graph synthesizer to generate the generated background image and page elements; A psychological report generation module is used to generate a final psychological report by combining the generated annotations and the semantic scene graph.

2. The psycho-semantic scene image generation system based on a differentiable graphics synthesizer as claimed in claim 1, characterized in that: The system adopts a psychosemantic scene image generation method, which includes the following steps: Step 1: Based on the psychological scale filled in by the user, process the data to obtain the three-level scores of the psychological factors; Step 2: Construct basic semantic concept groups through hierarchical semantic mapping; Step 3: construct structured prompt words, including text prompt word engineering and image prompt word engineering; Step 4: Generate a caption based on the text prompt word through a large language model, and generate a background image and page elements based on the image prompt word through a text-to-graphics model; Step 5: Generate a semantic scene graph using the background image and page elements generated in step 4 through a differentiable graph synthesizer; Step 6: Generate a final psychological report by combining the generated captions and the semantic scene graph.

3. The psycho-semantic scene image generation system based on a differentiable graphics synthesizer as claimed in claim 2, characterized in that: In step one, the psychological scale includes: 4 dimensions and 11 factors; among them, the self-harm tendency dimension is composed of 1 factor of self-harm tendency; the mental health dimension is composed of 4 factors of emotional state, sleep state, Internet dependence, and aversion to learning tendency; the inducement dimension is composed of 1 factor of traumatic events; the personality factor dimension is composed of 5 factors of interpersonal pleasing, interpersonal sensitivity, persistence, sense of efficacy and family relationship.

4. The psycho-semantic scene image generation system based on a differentiable graphics synthesizer as claimed in claim 2, characterized in that: In step 2, the hierarchical semantic mapping refers to dividing the score strengths of different dimensional factors into three levels and then mapping them into different semantic concept groups; the basic semantic concept group is represented by {0 k ,B k }, where B k Indicates the background style, O k Represents animal objects, factor level A in the scale k Through mapping hash tables, they are respectively associated with the basic semantic concept groups {O k ,B k }One to one correspondence.

5. The psycho-semantic scene image generation system based on a differentiable graphics synthesizer as claimed in claim 2, characterized in that: For the text prompt word project, the text prompt words include: role positioning, task objectives, context and language template; wherein, Role positioning is used to determine the role of the language model in the scenario as a psychological counselor for non-professional users; Task objectives are used to specify the goals or tasks in the scene and interpret the psychological test results through picture elements; The context and language template are used to provide background information and language templates of the scene, and are used to generate image captions for the large language model LLM; the captions include: an introduction to the semantics of the image and the scene and an analysis of the user's psychological state; wherein the context includes the user's psychological scale results and psychological semantic concept groups; the language template is designed according to the image interpretation method of the scene and the format of the psychological interpretation; and / or, For the image prompt word project, the image prompt words include: role positioning, task objectives, and context; Role positioning: Determine the role of the language model in the scene as the prompt engineer of the Stable Difffusion image, including the format of the general Stable Diffsuion model input prompt word; Task goal: specify the goal or task in the scene, and write the prompt words for the generated picture through the semantic concept group; Context and language template: The context includes the current background or the corresponding introduction of the page element; it is used to generate the required background image and the psychologically corresponding page elements for the text-to-image generation model (T2I Model).

6. The psycho-semantic scene image generation system based on a differentiable graphics synthesizer as claimed in claim 5, characterized in that: The process of constructing text prompt words includes: Determine the role positioning and task objectives: Position the language model as a psychological counselor for non-professional users, and specify the task objective as interpreting psychological test results through image elements; Design context and language templates: Provide background information of the scene, including the user's psychological scale results and psychological semantic concept groups; design appropriate language templates for generating captions, which include an introduction to the semantics of the image and the scene and an analysis of the user's psychological state; Providing context and language templates to the large language model LLM: Providing the context and language templates in the text prompt words to the large language model to generate captions related to the psycho-semantic scene image; and / or, The construction process of image prompt words includes: Determine the role positioning and task objectives: Position the language model as the prompt engineer of the Stablediffusion image, and specify the task objective as using semantic concept groups to write the prompt words required to generate images; Design context and language templates: Provide appropriate context information, including the introduction of the current background or page elements. Design a common language template to generate the required background images and psychologically corresponding page elements; Providing context and language templates to the text-to-image generation model T2I: Provide the context and language templates in the image prompt words to the text-to-image generation model to generate background images and page elements that conform to the psychological semantic scene.

7. The psychosemantic scene image generation system based on a differentiable graphics synthesizer as claimed in claim 2, characterized in that: In step 4, the captions are generated through the large language model based on the constructed text prompts, including: input data, professional descriptions of the basic factors of the psychological scale, and interpretation of the elements in the image; The input data refers to the level classification of each dimension factor calculated from the options of each question selected by the user; The professional description of the basic factors of the psychological scale refers to the interpretation of the psychological scale results obtained by combining the psychological definition of each factor as the context of the prompt word with the introduction template; The interpretation of elements in the image refers to parsing and describing the image elements that appear in the scene to express the meaning, emotion or state related to the results of the psychological scale; The generation of the legend comprises the following steps: Step 4.1: Give the large language model a specific role and provide specific task objectives; Step 4.2, providing context information for generating a caption; the context information specifically refers to a simple and unprocessed introduction corresponding to the current background image or page element, including an animal object corresponding to the semantic concept group; Step 4.3: Generate a caption using a preset function output template based on the role positioning, task objectives, and context information given above.

8. The psycho-semantic scene image generation system based on a differentiable graphics synthesizer as claimed in claim 2, characterized in that: In step 4, generating background images and page elements through a text-to-graphics model according to the constructed image prompt words includes the following steps: Step 4.4: Give the text-to-image model specific task objectives and context information for generating page elements and background images; Step 4.5, the text-to-image model generates page elements according to the task objectives and context information of generating page elements; Step 4.6: The text-to-image model generates a background image based on the task objective and context information of generating a background image.

9. The psycho-semantic scene image generation system based on a differentiable graphic synthesizer as claimed in claim 7 or 8, characterized in that: Before and / or after generating the captions, background images, and page elements in step 4, disturbances are introduced to enrich the generation; wherein, The pre-perturbation includes: adding neutral semantic objects to the basic semantic concept group so that the generated results contain other animals that fit the background picture; and / or, Post-perturbation includes: shuffling the positions of the generated page elements according to random numbers and synthesizing them into the background.

10. The psycho-semantic scene image generation system based on a differentiable graphic synthesizer according to claim 1, characterized in that: In step five, the differentiable graphics synthesizer is a tool for synthesizing semantic scene graphs; it is an image synthesis model based on differentiable operations, which can combine page elements and background images to generate a scene graph with semantic meaning.

Images