Method, device, computer equipment and readable storage medium for planning target area viewing scenes based on multi-modality

Abstract

The present invention discloses a method, device, computer equipment and readable storage medium for planning a viewing scene of a target area based on multimodality, comprising: firstly obtaining viewing video data of a target area, including a plurality of preset viewing scene pictures. Then, by obtaining and analyzing the multimodal feedback results of users for the viewing video data, the visual landscape quality of each preset viewing scene picture is evaluated. At the same time, considering the pre-set visual impact factor of the landscape of the target area, the correlation analysis is performed on the visual landscape quality evaluation result and the visual impact factor of the landscape, and finally the landscape scene planning result of the target area is obtained. Such a design improves the personalization and accuracy of the viewing scene planning and optimizes the user's viewing experience.

Description

Technical Field

[0001] The present invention relates to the field of landscape planning, and in particular to a multi-modal target area landscape scene planning method, device, computer equipment and readable storage medium. Background Art

[0002] Traditional viewing scene planning methods for target areas mostly rely on expert experience or simple user surveys. These methods often ignore the multimodal feedback of users during the actual viewing process and fail to fully consider the impact of landscape visual influencing factors on the viewing experience. Summary of the invention

[0003] The object of the present invention is to provide a method, device, computer equipment and readable storage medium for planning a target area viewing scene based on multimodality.

[0004] In a first aspect, an embodiment of the present invention provides a method for planning a target area viewing scene based on multi-modality, comprising:

[0005] Acquire viewing video data for a target area, wherein the viewing video data includes a plurality of preset viewing scene images of the target area;

[0006] Obtaining a multimodal feedback result of a user for the viewing video data, and obtaining a visual landscape quality evaluation result of each of the preset viewing scene images according to the multimodal feedback result;

[0007] Obtaining a preset landscape visual impact factor of the target area;

[0008] A correlation analysis is performed on the visual landscape quality evaluation result and the landscape visual impact factor to obtain a viewing scene planning result for the target area.

[0009] In a possible implementation, the multimodal feedback result includes an eye movement feedback result, a user expression feedback result, a physiological signal result, and a satisfaction survey result, and the obtaining of the multimodal feedback result of the user for the viewing video data, and obtaining the visual landscape quality evaluation result of each preset viewing scene picture according to the multimodal feedback result, includes:

[0010] Respectively obtaining the eye movement feedback result, the user expression feedback result, the physiological signal result and the satisfaction survey result;

[0011] The eye movement feedback results, the user expression feedback results, the physiological signal results and the satisfaction survey results are aligned according to the time axis to obtain the visual landscape quality evaluation results of each preset viewing scene picture.

[0012] In a possible implementation, obtaining the eye movement feedback result includes:

[0013] Obtaining, by means of a virtual reality eye tracker, the user's gaze duration, first gaze time, pupil size, eye movement hotspot area, and gaze trajectory for a target preset viewing scene picture; the target preset viewing scene picture is any one of the multiple preset viewing scene pictures;

[0014] The gaze duration, the first gaze time, the pupil size, the eye movement hotspot area and the gaze trajectory are used as the eye movement feedback result of the target preset viewing scene picture.

[0015] In a possible implementation, obtaining the user expression feedback result includes:

[0016] Acquiring, by means of a preset expression recognition system, changes in the user's eyebrows, eyes, eyelids, and lips for a target preset viewing scene picture; the target preset viewing scene picture is any one of the multiple preset viewing scene pictures;

[0017] The change is used as a user expression feedback result of the target preset viewing scene picture.

[0018] In a possible implementation, obtaining the physiological signal result includes:

[0019] Collecting the user's electrocardiogram data, electroencephalogram data, electromyography data, electrodermal conductivity data, respiratory wave data and heart rate for a target preset viewing scene screen by a multi-channel physiological instrument; the target preset viewing scene screen is any preset viewing scene screen among the multiple preset viewing scene screens;

[0020] The electrocardiogram data, the electroencephalogram data, the electromyography data, the electrodermal conductivity data, the respiratory wave data and the heart rate are used as physiological signal results of the target preset viewing scene picture.

[0021] In a possible implementation manner, the performing a correlation analysis on the visual landscape quality evaluation result and the landscape visual impact factor to obtain the landscape scene planning result of the target area includes:

[0022] An IPA two-dimensional matrix analysis diagram is drawn according to the visual landscape quality evaluation results; the IPA two-dimensional matrix analysis diagram includes the first quadrant, the second quadrant, the third quadrant and the fourth quadrant; the indicators corresponding to the first quadrant include landscape space comfort, landscape vision breadth, landscape space comfort, building sketch attractiveness, landscape sketch attractiveness, architectural attractiveness, landscape cultural expression and plant and environment coordination; the indicators corresponding to the second quadrant include road square color and plant diversity; the indicators corresponding to the third quadrant include building color, road square material, building sketch color, landscape sketch color, building material, the coordination degree between building sketch material and environment, the coordination degree between landscape sketch material and environment and plant landscape color richness; the indicators corresponding to the fourth quadrant include waterscape attractiveness;

[0023] Normalizing the IPA two-dimensional matrix analysis diagram to obtain a comprehensive landscape score corresponding to each of the preset viewing scene images in the target area;

[0024] The landscape visual impact factors are subjected to correlation analysis according to the landscape comprehensive score corresponding to each of the preset viewing scene images to obtain the viewing scene planning result of the target area.

[0025] In a possible implementation manner, the obtaining of a preset landscape visual impact factor of the target area includes:

[0026] Obtain historical landscape planning and design cases associated with the target area;

[0027] Obtain landscape construction elements and tourists' landscape needs;

[0028] Combining the historical landscape planning and design cases, the landscape creation elements and the tourists' landscape needs, the landscape visual impact factor of the target area is obtained by using network word frequency search and expert analysis method.

[0029] In a second aspect, an embodiment of the present invention provides a target area viewing scene planning device based on multi-modality, comprising:

[0030] An acquisition module is used to acquire viewing video data for a target area, wherein the viewing video data includes a plurality of preset viewing scene images of the target area; acquire a multimodal feedback result of a user for the viewing video data, and obtain a visual landscape quality evaluation result of each of the preset viewing scene images according to the multimodal feedback result; and acquire a landscape visual impact factor of the target area that is preset;

[0031] The planning module is used to perform correlation analysis on the visual landscape quality evaluation result and the landscape visual impact factor to obtain the landscape scene planning result of the target area.

[0032] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device executes the method described in at least one possible implementation of the first aspect.

[0033] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method described in at least one possible implementation manner of the first aspect.

[0034] Compared with the prior art, the beneficial effects provided by the present invention include: adopting a method, device, computer equipment and readable storage medium for planning a viewing scene of a target area based on multimodality disclosed in the present invention, by acquiring viewing video data of the target area, including multiple preset viewing scene pictures. Then, by acquiring and analyzing the multimodal feedback results of the user for the viewing video data, the visual landscape quality of each preset viewing scene picture is evaluated. At the same time, considering the pre-set visual impact factor of the landscape of the target area, the correlation analysis of the visual landscape quality evaluation result and the visual impact factor of the landscape is performed, and finally the landscape scene planning result of the target area is obtained. Such a design improves the personalization and accuracy of the viewing scene planning and optimizes the user's viewing experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0036] Figure 1 A schematic diagram of the steps of a multi-modal target area viewing scene planning method provided by an embodiment of the present invention;

[0037] Figure 2 A schematic block diagram of the structure of a target area viewing scene planning device based on multi-modality provided by an embodiment of the present invention;

[0038] Figure 3 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0040] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0041] In order to solve the technical problems in the aforementioned background technology, Figure 1 A flowchart of a method for planning a scene viewing a target area based on multimodality is provided in an embodiment of the present disclosure. The method for planning a scene viewing a target area based on multimodality is introduced in detail below.

[0042] Step S201, obtaining viewing video data for a target area, wherein the viewing video data includes a plurality of preset viewing scene images of the target area;

[0043] Step S202, obtaining a multimodal feedback result of a user for the viewing video data, and obtaining a visual landscape quality evaluation result of each of the preset viewing scene images according to the multimodal feedback result;

[0044] Step S203, obtaining a preset landscape visual impact factor of the target area;

[0045] Step S204: performing a correlation analysis on the visual landscape quality evaluation result and the landscape visual impact factor to obtain a viewing scene planning result of the target area.

[0046] In an embodiment of the present invention, exemplarily, the server first retrieves the viewing video data for a specific target area from a database or cloud storage. For example, the target area is "X Mountain Scenic Area", and the server obtains video data of multiple preset viewing scene images including X Mountain Sunrise, Sea of ​​Clouds, Strange Pine and Rocks. The server designs and implements a user interface through which users can watch viewing videos and provide multimodal feedback, including clicks, slides, facial expressions, voice comments, etc. For example, the user expressed a high interest in the "X Mountain Sunrise" scene, clicked the "Like" button, and watched the "Sea of ​​Clouds" scene for a long time and made positive voice comments. The server collects these multimodal feedback results and analyzes the user's preferences and behavior patterns through a machine learning algorithm. The server uses multimodal data analysis technology, combined with the user's clicks, slide behaviors, facial expressions, voice comments, etc., to evaluate the visual landscape quality of each preset viewing scene image. For example, the "X Mountain Sunrise" scene has a high visual landscape quality score due to its magnificent scenery and high evaluation by users. The server reads the pre-set landscape visual impact factors from the configuration file, which may include season, weather, time (such as sunrise, sunset), tourist flow, etc. For example, for the X Mountain Scenic Area, the sea of ​​flowers in spring and the snow scene in winter are important visual influencing factors. The server uses data analysis tools to perform correlation analysis on the visual landscape quality evaluation results and landscape visual influencing factors. For example, the analysis found that the "X Mountain Sunrise" scene has the highest visual landscape quality in the early morning of spring, while the "sea of ​​clouds" scene is most spectacular when it is sunny after rain. Based on the analysis results, the server generates viewing scene planning results, recommending users to watch the "X Mountain Sunrise" in the early morning of spring and the "sea of ​​clouds" when it is sunny after rain, thereby providing personalized viewing experience planning. Through the above steps, the server can provide personalized viewing scene planning results for the target area based on multimodal feedback and landscape visual influencing factors.

[0047] In the embodiment of the present invention, the multimodal feedback result includes eye movement feedback result, user expression feedback result, physiological signal result and satisfaction survey result. The aforementioned step S202 can be implemented through the following example.

[0048] Respectively obtaining the eye movement feedback result, the user expression feedback result, the physiological signal result and the satisfaction survey result;

[0049] The eye movement feedback results, the user expression feedback results, the physiological signal results and the satisfaction survey results are aligned according to the time axis to obtain the visual landscape quality evaluation results of each preset viewing scene picture.

[0050] In an embodiment of the present invention, exemplarily, the server collects the user's eye movement data through an eye tracking device. For example, when the user watches the "X Mountain Sunrise" scene, the movement of the eyeball and the gaze point are recorded to form an eye movement feedback result.

[0051] Through facial expression recognition technology, the server captures and analyzes the changes in the user's expression when watching the video, such as smiling, frowning, etc., and obtains the user's expression feedback results. For example, the user showed a happy expression when watching the "sea of ​​clouds" scene.

[0052] The server also collects the user's physiological signals, such as heart rate changes, through a physiological signal monitoring device (such as a heart rate monitor) as a physiological signal result. For example, the user's heart rate increases when watching certain shocking scenes.

[0053] Finally, the server collects user satisfaction survey results for each preset viewing scene through online surveys or questionnaires, such as user satisfaction scores for the "strange pines and rocks" scene.

[0054] The server aligns the above four multimodal feedback results according to the time axis to ensure that each feedback result corresponds to a specific viewing scene. For example, the user's eye movement data, expression changes, physiological signals and satisfaction scores when watching "X Mountain Sunrise" are matched and associated in time.

[0055] The server uses data analysis algorithms to comprehensively consider eye movement feedback results (such as gaze duration, gaze point distribution), user expression feedback results (such as pleasure level), physiological signal results (such as heart rate changes) and satisfaction survey results to evaluate the visual landscape quality of each preset viewing scene. For example, the "X Mountain Sunrise" scene received a high visual landscape quality evaluation due to its high gaze duration, pleasant expression, accelerated heart rate and high satisfaction score.

[0056] Through the above steps, the server can comprehensively and accurately obtain and analyze the user's multimodal feedback, thereby obtaining the visual landscape quality evaluation result of each preset viewing scene.

[0057] In the embodiment of the present invention, obtaining the eye movement feedback result may be implemented through the following example.

[0058] Obtaining, by means of a virtual reality eye tracker, the user's gaze duration, first gaze time, pupil size, eye movement hotspot area, and gaze trajectory for a target preset viewing scene picture; the target preset viewing scene picture is any one of the multiple preset viewing scene pictures;

[0059] The gaze duration, the first gaze time, the pupil size, the eye movement hotspot area and the gaze trajectory are used as the eye movement feedback result of the target preset viewing scene picture.

[0060] In the embodiment of the present invention, the server collects detailed eye movement data for each preset viewing scene (such as the scene of "Sunrise at Mount X") viewed by the user through a professional device, a virtual reality eye tracker. The data includes:

[0061] Gaze duration:

[0062] The server records the length of time the user's eyes stay in each specific area or point of interest. For example, when the user is watching the "X Mountain Sunrise", he or she stares at the light part of the sunrise for a long time.

[0063] First fixation time:

[0064] The server records the time when the user first casts his / her gaze toward each specific area or point of interest. For example, when the user first watches the video "Sunrise at Mount X", the first thing he / she looks at is the position where the sun rises.

[0065] Pupil size:

[0066] Through the eye tracker, the server can capture the changes in the user's pupils, which reflects the user's interest and excitement. For example, when watching the magnificent view of "Sunrise at Mount X", the user's pupils may dilate slightly.

[0067] Eye movement hot spots:

[0068] The server analyzes the user's eye movement data to determine which areas or points of interest attract the most attention of the user. For example, in the "Sunrise at Mount X" scene, the sun and the surrounding clouds may be eye movement hot spots.

[0069] Gaze Track:

[0070] The server tracks and records the movement path of the user's eyes during viewing. For example, when a user watches "Sunrise on Mount X", he or she may move from the sun to the surrounding clouds and then to the mountains, forming a specific gaze trajectory.

[0071] After completing the above data collection, the server uses these data (gaze duration, first gaze time, pupil size, eye movement hotspots, and gaze trajectory) as the eye movement feedback results of the target preset viewing scene "X Mountain Sunrise". These data provide an important basis for evaluating the visual landscape quality of the scene. By comprehensively analyzing the data of multiple users, the server can more accurately evaluate the attractiveness of different viewing scenes and provide strong support for viewing planning.

[0072] In the embodiment of the present invention, obtaining the user expression feedback result may be implemented through the following example.

[0073] Acquiring, by means of a preset expression recognition system, changes in the user's eyebrows, eyes, eyelids, and lips for a target preset viewing scene picture; the target preset viewing scene picture is any one of the multiple preset viewing scene pictures;

[0074] The change is used as a user expression feedback result of the target preset viewing scene picture.

[0075] In the embodiment of the present invention, the server uses a preset expression recognition system to collect detailed expression data for each preset viewing scene screen (such as the scene of "X Mountain Sea of ​​Clouds") viewed by the user. These data mainly include changes in key parts of the user's face, as follows:

[0076] Eyebrow changes:

[0077] The server captures the changes in the shape of the user's eyebrows through the expression recognition system. For example, when the user sees the spectacular sea of ​​clouds in "X Mountain Sea of ​​Clouds", his eyebrows may rise slightly to express surprise or pleasure.

[0078] Changes in eyes:

[0079] The server records the size, shape, and degree of opening of the user's eyes. When watching "X Mountain Sea of ​​Clouds", the user's eyes may be opened wide to more fully appreciate the beautiful scenery in front of them.

[0080] Changes in eyelids:

[0081] The server pays attention to the degree of closure of the user's eyelids and whether there are any actions such as blinking. For example, when the user is deeply attracted by the beautiful scenery of "X Mountain Sea of ​​Clouds", the frequency of blinking may be reduced.

[0082] Lip changes:

[0083] The server also captures the changes in the user's lips, such as the corners of the mouth rising to indicate a smile, or the lips slightly opening to indicate surprise. When watching "X Mountain Sea of ​​Clouds", the user may smile with satisfaction.

[0084] After completing the above data collection, the server uses the changes in eyebrows, eyes, eyelids, and lips as the user's expression feedback results for the target preset viewing scene of "X Mountain Sea of ​​Clouds". These data reflect the user's emotional response and preference for the scene, providing an important reference for evaluating the visual quality of the scene. By comprehensively analyzing the expression data of multiple users, the server can more accurately understand the attractiveness of different viewing scenes to users and provide a strong basis for viewing planning.

[0085] In the embodiment of the present invention, obtaining the physiological signal result may be implemented through the following examples.

[0086] Collecting the user's electrocardiogram data, electroencephalogram data, electromyography data, electrodermal conductivity data, respiratory wave data and heart rate for a target preset viewing scene screen by a multi-channel physiological instrument; the target preset viewing scene screen is any preset viewing scene screen among the multiple preset viewing scene screens;

[0087] The electrocardiogram data, the electroencephalogram data, the electromyography data, the electrodermal conductivity data, the respiratory wave data and the heart rate are used as physiological signal results of the target preset viewing scene picture.

[0088] In the embodiment of the present invention, the server collects detailed physiological signal data for each preset scene of the target viewed by the user (such as the scene of "X Mountain Strange Pines and Rocks") through a connected multi-channel physiological instrument. These data include multiple physiological indicators, as follows:

[0089] ECG data:

[0090] The server receives the user's ECG signal from the physiological instrument and analyzes the electrical activity of the heart. For example, when the user watches "X Mountain Strange Pines and Rocks", the ECG data may show that the heart activity is stable, indicating that the user is in a relaxed state.

[0091] EEG data:

[0092] The server records and analyzes the user's brain waves to understand the state of brain activity. When watching "X Mountain Strange Pines and Rocks", the user's brain wave data may show an increase in alpha waves, indicating that the user is in a state of pleasure and relaxation.

[0093] EMG data:

[0094] The server monitors the electrical activity of the user's muscles to assess the user's tension or relaxation. For example, when the user is attracted by the beautiful scenery of "X Mountain's strange pines and rocks", the electromyographic data may show that the muscles are in a relaxed state.

[0095] Skin electrical data:

[0096] The server measures the user's skin conductivity, which can reflect the user's emotional state. For example, when watching "X Mountain Strange Pines and Rocks", if the user feels excited or agitated, the skin conductivity data may rise.

[0097] Respiratory wave data:

[0098] The server records the user's breathing frequency and depth to analyze the user's relaxation level. When watching "X Mountain Strange Pines and Rocks", the user's breathing may become smoother and deeper.

[0099] Heart rate:

[0100] The server continuously monitors the user's heart rate changes. For example, when the user sees the spectacular scenery of "X Mountain Strange Pines and Rocks", the heart rate may rise slightly, indicating that the user is in an excited state.

[0101] After completing the above data collection, the server uses these ECG data, EEG data, EMG data, skin conductance data, respiratory wave data, and heart rate as the physiological signal results of the preset viewing scene of "X Mountain Strange Pines and Rocks". These data provide information on the physiological reactions of users when viewing different viewing scenes, and provide an objective basis for evaluating the visual quality of the scene. By comprehensively analyzing the physiological signal data of multiple users, the server can more accurately understand the impact of different viewing scenes on the physiological state of users, and provide a scientific basis for viewing planning.

[0102] In the embodiment of the present invention, the aforementioned step S204 may be implemented through the following example execution.

[0103] An IPA two-dimensional matrix analysis diagram is drawn according to the visual landscape quality evaluation results; the IPA two-dimensional matrix analysis diagram includes the first quadrant, the second quadrant, the third quadrant and the fourth quadrant; the indicators corresponding to the first quadrant include the comfort of landscape space, the breadth of landscape vision, the attractiveness of building sketches, the attractiveness of landscape sketches, the attractiveness of architectural shapes, the expressiveness of landscape culture and the coordination between plants and the environment; the indicators corresponding to the second quadrant include the color of roads and squares and the diversity of plants; the indicators corresponding to the third quadrant include the color of buildings, the material of roads and squares, the color of building sketches, the color of landscape sketches, the material of buildings, the coordination between the material of building sketches and the environment, the coordination between the material of landscape sketches and the environment and the richness of plant landscape colors; the indicators corresponding to the fourth quadrant include the attractiveness of waterscapes;

[0104] Normalizing the IPA two-dimensional matrix analysis diagram to obtain a comprehensive landscape score corresponding to each of the preset viewing scene images in the target area;

[0105] The landscape visual impact factors are subjected to correlation analysis according to the landscape comprehensive score corresponding to each of the preset viewing scene images to obtain the viewing scene planning result of the target area.

[0106] In the embodiment of the present invention, the server first draws an IPA (Importance-Performance Analysis) two-dimensional matrix analysis chart based on the visual landscape quality evaluation results obtained in the previous step. This analysis chart is divided into four quadrants, each of which corresponds to a different landscape visual impact factor index:

[0107] Quadrant 1:

[0108] The indicators included are the comfort of landscape space, the breadth of viewing field, the attractiveness of building sketches, the attractiveness of landscape sketches, the attractiveness of architectural shape, the expression of landscape culture and the coordination between plants and the environment.

[0109] For example, in the "Sunrise at X Mountain" viewing scene, these indicators may all perform well, showing high comfort and a wide field of vision, as well as attractive structures, landscape pieces, architectural shapes, and rich plant landscapes.

[0110] Quadrant II:

[0111] Indicators included are road and square colour and plant diversity.

[0112] In the "X Mountain Sea of ​​Clouds" viewing scene, the colors of the road square may be coordinated with the natural environment, and the plant diversity may also be very rich.

[0113] Quadrant III:

[0114] The indicators included are building color, road and square material, building sketch color, landscape sketch color, building material, the degree of coordination between building sketch material and the environment, the degree of coordination between landscape sketch material and the environment, and the color richness of plant landscape.

[0115] The “X Mountain Strange Pines and Rocks” viewing scene may perform averagely in these aspects and needs to be improved to better coordinate with the environment.

[0116] Quadrant 4:

[0117] The indicators included are mainly water feature attractiveness.

[0118] If there is a water feature in the target area, such as "X Mountain Waterfall", then this indicator will be very important. If not, this quadrant may not be a major consideration.

[0119] Next, the server normalizes the IPA two-dimensional matrix analysis chart to ensure comparability between different indicators. Through this step, the server can obtain the comprehensive landscape score corresponding to each preset viewing scene in the target area.

[0120] Finally, the server conducts a correlation analysis of the factors affecting the landscape vision based on the comprehensive landscape score corresponding to each preset viewing scene. Through this analysis, the server can identify which factors have an important impact on the landscape quality, which factors need to be improved, and which factors have performed well. Based on this information, the server finally generates the viewing scene planning results for the target area, providing a scientific basis for future landscape design and improvement.

[0121] In the embodiment of the present invention, the aforementioned step S203 can be implemented through the following example execution.

[0122] Obtain historical landscape planning and design cases associated with the target area;

[0123] Obtain landscape construction elements and tourists' landscape needs;

[0124] Combining the historical landscape planning and design cases, the landscape creation elements and the tourists' landscape needs, the landscape visual impact factor of the target area is obtained by using network word frequency search and expert analysis method.

[0125] In the embodiment of the present invention, illustratively, first, the server obtains historical landscape planning and design cases associated with the target area. For example, for the target area of ​​X Mountain Scenic Area, the server will collect all landscape planning and design cases about X Mountain Scenic Area in the past few decades, including design drawings, planning reports, implementation effect evaluations, etc.

[0126] Secondly, the server obtains landscape construction elements and tourist landscape needs. Landscape construction elements may include terrain, vegetation, water bodies, buildings, small pieces, etc., while tourist landscape needs may include viewing comfort, visual breadth, cultural expression, etc. For the X Mountain Scenic Area, the server will analyze the terrain characteristics, vegetation distribution, water body conditions, and existing buildings and small pieces of X Mountain, and understand tourists' landscape needs through tourist surveys or online comment analysis.

[0127] Finally, the server combines historical landscape planning and design cases, landscape construction elements and tourist landscape needs, and uses network word frequency search and expert analysis to obtain the landscape visual impact factors of the target area. Specifically, the server will analyze which elements in historical cases have an important impact on landscape quality and which elements are of particular concern to tourists. At the same time, the server will also use network word frequency search tools to analyze online comments and travel notes about the X Mountain Scenic Area, and extract the landscape elements and evaluation indicators most frequently mentioned by tourists. Finally, the server will invite experts in the field of landscape planning to conduct interviews or questionnaires to obtain their professional opinions on the landscape visual impact factors of the X Mountain Scenic Area. Combining this information, the server can obtain the landscape visual impact factors of the X Mountain Scenic Area, such as terrain undulation, vegetation richness, water clarity, architectural style coordination, and landscape facility perfection.

[0128] In order to more clearly describe the solution provided by the embodiment of the present application, a more detailed implementation method is provided below.

[0129] (1) Landscape preference analysis based on eye movement data

[0130] Eye movement data refers to the position and movement of the eyes recorded and tracked by an infrared camera. It can objectively reflect which visual elements people look at, when they look at certain visual elements, how long they stare at each element, the fixed order in which they look at visual elements, whether they repeatedly look at visual elements from a certain point of view, etc., thereby quantifying visual attention.

[0131] A virtual reality eye tracker was used to accurately capture the subjects' gaze behavior in the virtual reality scene. A matching system was used on the computer to automatically analyze the video viewers' gaze duration on different landscape nodes, the gaze trajectory of each scene, hot spots and other information to explore the subjects' visual landscape preferences. The selected indicators can be referred to Table 1.

[0132] Table 1

[0133]

[0134]

[0135] The embodiment of the present invention uses HTR-Eyet racking virtual reality eye tracker, with a sampling frequency of 120Hz for both eyes and a total resolution of 2880×1600. Its advantage is that it can make the experimental participants feel as if they are in the scene, and indoor experiments avoid the influence of uncontrollable environmental factors such as light intensity and weather changes, and the interference of the head and body shaking of the subjects when walking outdoors on data collection.

[0136] (2) Emotion recognition method based on facial expression

[0137] The facial expression detection system is used to observe and record the changes in the subjects' facial expressions while watching the three-dimensional animation of the service area landscape in real time and to characterize the emotional state. The HTR-gFace facial expression detection system selected in the embodiment of the present invention adopts a three-dimensional model, which can use facial muscles to automatically identify, calculate, classify and analyze the facial expressions of the subjects at multiple locations. The system can recognize 10 expressions (smile, size, fake smile, kiss, gesture, disappointment, rage, relaxation, shouting, excitement), and output 7 basic emotions (happiness, anger, surprise, fear, sadness, disgust, contempt) and 2 advanced emotions (immersion level, psychological valence: positive / negative).

[0138] (3) Emotion recognition methods based on physiological signals

[0139] The multi-channel physiological instrument is a wearable, low-interference wireless human physiological data recording system that can monitor and record the physiological changes of subjects in real time under real natural conditions, and analyze the individual's emotional state based on the physiological change trends under visual stimulation.

[0140] The embodiment of the present invention uses the HRT-Physio multi-channel physiological instrument to monitor and record the subjects' ECG, EEG, EMG, EDA, RESP, HR and other indicators when they watch the three-dimensional animation of the service area landscape. Please refer to Table 2. Each host of the instrument has 9 sensor channels (non-unipolar analog channels), the sampling rate is 4000Hz (per channel), and 9 channels of physiological data can be recorded simultaneously. The software supports 3 hosts to work synchronously, forming a 27-channel physiological data recording system.

[0141] Table 2

[0142]

[0143]

[0144] The ECG data were processed, the R wave peak was extracted, and the heart rate variability (HRV) indicators of each group were compared using time domain analysis and frequency domain analysis. Time domain analysis refers to the function of ECG signal changing with time. The overall standard deviation of RR interval (SDNN), the square root of the mean square of the difference between adjacent RR intervals (RMSSD) and Lorenz scatter plot were used as quantitative indicators. The emotional perception of people in different scenarios was judged. When the pressure was relieved and the emotions developed positively, the values ​​of SDNN and RMSSD would decrease. The Lorenz scatter plot uses an iterative method to record continuous ECG RR intervals, reflecting the linear and nonlinear change trends of HRV. Its normal heart rhythm should be concentrated near the 45° angle bisector and distributed symmetrically in a comet shape. Under the influence of positive or negative emotions, the scatter distribution deviates from the 45° angle bisector. Among them, under the negative emotional trend, the heart rate increases, the scatter point shape is conical, the major axis is longer, the minor axis is shorter, and the dispersion is low; under the positive emotional trend, the heart rate slows down, the scatter point shape is elliptical, the major axis is shorter, the minor axis is longer, and the dispersion is high. The ECG time domain analysis signal is converted into a frequency domain analysis signal using fast Fourier transform, and then the power spectrum density analysis of the ECG signal is performed: an increase in HF means an increase in the activity of the parasympathetic nerves; the LF / HF value is related to the excitement of the sympathetic nerves of the human body, and the increase in the LF / HF value means an increase in the degree of arousal. Among them, Where N is the number of RR intervals, RR i is the value of the i-th RR interval, ΔRR i is the difference between the i-th adjacent RR intervals.

[0145] (4) Comprehensive analysis method based on multimodal software system

[0146] The accuracy of multimodal emotion recognition is higher than that of single-modal recognition of eye movement, expression or physiological signal. Through the multimodal software system, the various indicators collected by the above-mentioned instruments are synchronized to the same timeline. The three types of evaluation results are comprehensively analyzed using a two-dimensional emotion model, which is divided into four areas: high arousal and high valence, high arousal and low valence, low arousal and high valence, and low arousal and low valence, to judge the attractiveness, information content, and characteristics of the landscape scheme, as well as the emotional valence and emotional arousal of tourists, reflecting the tourists' experience in the landscape environment of the service area.

[0147] The embodiment of the present invention adopts the HRT multimodal software system to synchronously playback, count and analyze the data of all the above modules, and can also carry out experimental plan formulation, experimental task setting, questionnaire survey and automatic report generation.

[0148] (5) Importance-Performance Analysis

[0149] Importance-Performance Analysis (IPA) is an ideal tool for analysis and evaluation. It compares the user's expected expectations (importance) with actual perception (performance) to distinguish the advantages, disadvantages and priority of transformation and improvement of existing resources, thereby assisting in the proposal of optimization strategies. The data obtained from the analysis is reflected in the two-dimensional matrix analysis diagram in the form of (x, y) coordinates, and the two-dimensional plane is divided into four quadrants: "Advantage Area", "Maintenance Area", "Opportunity Area" and "Improvement Area" with the mean of each axis as the boundary.

[0150] In summary, the design concept of "no trace" focuses on the protection of the ecological environment, taking the ecological environment as part of the design elements, following the natural geographical characteristics of the site and the inherent mechanism and succession law of the ecosystem, mainly restoring the authenticity and integrity of the ecosystem, improving the ecological service function, and enabling the ecosystem to achieve a sustainable and virtuous cycle; on the other hand, it focuses on people's yearning for nature, taking people's most primitive pursuit of beauty into consideration, supplemented by artificial restoration, laying out landscape elements, sorting out functional spaces, and configuring service facilities, reproducing the ecological landscape and the ecology of the landscape in the simplest way, and arousing people's aesthetic pleasure. The use of this concept can minimize people's demands and consumption of natural resources, and thus achieve harmonious coexistence between man and nature.

[0151] Please refer to Figure 2 , Figure 2 A multi-modal target area viewing scene planning device 110 provided in an embodiment of the present invention includes:

[0152] The acquisition module 1101 is used to acquire viewing video data for a target area, wherein the viewing video data includes a plurality of preset viewing scene images of the target area; acquire a multimodal feedback result of a user for the viewing video data, and obtain a visual landscape quality evaluation result of each of the preset viewing scene images according to the multimodal feedback result; and acquire a preset landscape visual impact factor of the target area;

[0153] The planning module 1102 is used to perform a correlation analysis on the visual landscape quality evaluation result and the landscape visual impact factor to obtain a landscape scene planning result of the target area.

[0154] It should be noted that the implementation principle of the aforementioned target area viewing scene planning device 110 based on multimodality can refer to the implementation principle of the aforementioned target area viewing scene planning method based on multimodality, which will not be repeated here. It should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be fully implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the target area viewing scene planning device 110 based on multimodality can be a separately established processing element, or it can be integrated in a chip of the above device to implement. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the target area viewing scene planning device 110 based on multimodality. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or an instruction in software form.

[0155] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0156] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned multi-modal target area viewing scene planning device 110. Figure 3 As shown, Figure 3The computer device 100 is a block diagram of a structure of a computer device 100 provided in an embodiment of the present invention. The computer device 100 includes a multi-modal target area viewing scene planning device 110 , a memory 111 , a processor 112 and a communication unit 113 .

[0157] To achieve data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these elements can be achieved through one or more communication buses or signal lines. The multimodal target area viewing scene planning device 110 includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the multimodal target area viewing scene planning device 110 stored in the memory 111, such as the software function modules and computer programs included in the multimodal target area viewing scene planning device 110.

[0158] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is running, the computer device where the readable storage medium is located is controlled to execute the aforementioned target area viewing scene planning method.

[0159] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. A method for planning a target area viewing scene based on multi-modality, characterized in that: include: Acquire viewing video data for a target area, wherein the viewing video data includes a plurality of preset viewing scene images of the target area; Obtaining a multimodal feedback result of a user for the viewing video data, and obtaining a visual landscape quality evaluation result of each of the preset viewing scene images according to the multimodal feedback result; Obtaining a preset landscape visual impact factor of the target area; Performing a correlation analysis on the visual landscape quality evaluation result and the landscape visual impact factor to obtain a viewing scene planning result of the target area; The multimodal feedback results include eye movement feedback results, user expression feedback results, physiological signal results, and satisfaction survey results. The obtaining of the multimodal feedback results of the user for the viewing video data, and obtaining the visual landscape quality evaluation results of each of the preset viewing scene images according to the multimodal feedback results, include: Respectively obtaining the eye movement feedback result, the user expression feedback result, the physiological signal result and the satisfaction survey result; Aligning the eye movement feedback results, the user expression feedback results, the physiological signal results, and the satisfaction survey results according to the time axis to obtain a visual landscape quality evaluation result of each preset viewing scene picture; Obtaining the eye movement feedback result includes: Obtaining, by means of a virtual reality eye tracker, the user's gaze duration, first gaze time, pupil size, eye movement hotspot area, and gaze trajectory for a target preset viewing scene picture; the target preset viewing scene picture is any one of the multiple preset viewing scene pictures; The fixation duration, the first fixation time, the pupil size, the eye movement hotspot area and the fixation trajectory are used as the eye movement feedback result of the target preset viewing scene picture; Obtaining the eye movement feedback result includes: Obtaining, by means of a virtual reality eye tracker, the user's gaze duration, first gaze time, pupil size, eye movement hotspot area, and gaze trajectory for a target preset viewing scene picture; the target preset viewing scene picture is any one of the multiple preset viewing scene pictures; The fixation duration, the first fixation time, the pupil size, the eye movement hotspot area and the fixation trajectory are used as the eye movement feedback result of the target preset viewing scene picture; Obtaining the user expression feedback result includes: Acquiring, by means of a preset expression recognition system, changes in the user's eyebrows, eyes, eyelids, and lips for a target preset viewing scene picture; the target preset viewing scene picture is any one of the multiple preset viewing scene pictures; The change is used as a user expression feedback result of the target preset viewing scene picture; Acquiring the physiological signal result includes: Collecting the user's electrocardiogram data, electroencephalogram data, electromyography data, electrodermal conductivity data, respiratory wave data and heart rate for a target preset viewing scene screen by a multi-channel physiological instrument; the target preset viewing scene screen is any preset viewing scene screen among the multiple preset viewing scene screens; The electrocardiogram data, the electroencephalogram data, the electromyography data, the electrodermal conductivity data, the respiratory wave data, and the heart rate are used as physiological signal results of the target preset viewing scene picture; The correlation analysis of the visual landscape quality evaluation result and the landscape visual impact factor is performed to obtain the landscape scene planning result of the target area, including: An IPA two-dimensional matrix analysis diagram is drawn according to the visual landscape quality evaluation results; the IPA two-dimensional matrix analysis diagram includes the first quadrant, the second quadrant, the third quadrant and the fourth quadrant; the indicators corresponding to the first quadrant include landscape space comfort, landscape vision breadth, landscape space comfort, building sketch attractiveness, landscape sketch attractiveness, architectural attractiveness, landscape cultural expression and plant and environment coordination; the indicators corresponding to the second quadrant include road square color and plant diversity; the indicators corresponding to the third quadrant include building color, road square material, building sketch color, landscape sketch color, building material, the coordination degree between building sketch material and environment, the coordination degree between landscape sketch material and environment and plant landscape color richness; the indicators corresponding to the fourth quadrant include waterscape attractiveness; Normalizing the IPA two-dimensional matrix analysis diagram to obtain a comprehensive landscape score corresponding to each of the preset viewing scene images in the target area; Performing a correlation analysis on the landscape visual impact factors according to the landscape comprehensive score corresponding to each of the preset viewing scene images to obtain the viewing scene planning result of the target area; The step of obtaining a preset landscape visual impact factor of the target area includes: Obtain historical landscape planning and design cases associated with the target area; Obtain landscape construction elements and tourists' landscape needs; Combining the historical landscape planning and design cases, the landscape creation elements and the tourists' landscape needs, the landscape visual impact factor of the target area is obtained by using network word frequency search and expert analysis method.

2. A target area viewing scene planning device based on multi-modality, characterized in that: include: An acquisition module, used to acquire viewing video data for a target area, wherein the viewing video data includes a plurality of preset viewing scene images of the target area; Obtaining a multimodal feedback result of a user for the viewing video data, and obtaining a visual landscape quality evaluation result of each of the preset viewing scene images according to the multimodal feedback result; obtaining a landscape visual impact factor of the preset target area; A planning module, used to perform correlation analysis on the visual landscape quality evaluation result and the landscape visual impact factor to obtain a landscape scene planning result of the target area; The multimodal feedback results include eye movement feedback results, user expression feedback results, physiological signal results and satisfaction survey results. The acquisition module is specifically used for: Respectively obtaining the eye movement feedback result, the user expression feedback result, the physiological signal result, and the satisfaction survey result; aligning the eye movement feedback result, the user expression feedback result, the physiological signal result, and the satisfaction survey result according to the time axis to obtain a visual landscape quality evaluation result of each preset viewing scene picture; The virtual reality eye tracker is used to obtain the user's gaze duration, first gaze time, pupil size, eye movement hotspot area and gaze trajectory for the target preset viewing scene; The target preset viewing scene picture is any preset viewing scene picture among the multiple preset viewing scene pictures; the fixation duration, the first fixation time, the pupil size, the eye movement hotspot area and the fixation trajectory are used as the eye movement feedback result of the target preset viewing scene picture; The preset expression recognition system is used to obtain changes in the user's eyebrows, eyes, eyelids, and lips for the target preset viewing scene; The target preset viewing scene picture is any preset viewing scene picture among the multiple preset viewing scene pictures; the change situation is used as the user expression feedback result of the target preset viewing scene picture; Collecting the user's electrocardiogram (ECG) data, electroencephalogram (EEG) data, electromyogram (EMG) data, electrodermal conductivity (EGD) data, respiratory wave data, and heart rate for a target preset viewing scene screen by using a multi-channel physiological instrument; the target preset viewing scene screen is any one of the multiple preset viewing scene screens; and using the electrocardiogram (ECG) data, electromyogram (EMG) data, electrodermal conductivity (EGD) data, respiratory wave data, and heart rate as physiological signal results for the target preset viewing scene screen; Obtain historical landscape planning and design cases related to the target area; obtain landscape construction elements and tourist landscape requirements; Combining the historical landscape planning and design cases, the landscape construction elements and the tourist landscape needs, using network word frequency search and expert analysis methods, the landscape visual impact factor of the target area is obtained; The planning module is specifically used for: An IPA two-dimensional matrix analysis diagram is drawn according to the visual landscape quality evaluation results; the IPA two-dimensional matrix analysis diagram includes the first quadrant, the second quadrant, the third quadrant and the fourth quadrant; the indicators corresponding to the first quadrant include landscape space comfort, landscape vision breadth, landscape space comfort, building sketch attractiveness, landscape sketch attractiveness, architectural attractiveness, landscape cultural expression and plant and environment coordination; the indicators corresponding to the second quadrant include road square color and plant diversity; the indicators corresponding to the third quadrant include building color, road square material, building sketch color, landscape sketch color, building material, the coordination degree between building sketch material and environment, the coordination degree between landscape sketch material and environment and plant landscape color richness; the indicators corresponding to the fourth quadrant include waterscape attraction; the IPA two-dimensional matrix analysis diagram is normalized to obtain the landscape comprehensive score corresponding to each of the preset viewing scene pictures in the target area; the landscape visual impact factors are correlated according to the landscape comprehensive score corresponding to each of the preset viewing scene pictures to obtain the viewing scene planning results of the target area.

3. A computer device, characterized in that: The computer device comprises a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device performs the method of claim 1.

4. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method according to claim 1.

Images