Multimedia interactive imaging method, device, storage medium and computer program product

Abstract

The present invention discloses a multimedia interactive imaging method, device, storage medium and computer program product. The method comprises: obtaining EEG data of a target object, determining the current state recognition result of the target object according to the EEG data; the current state recognition result is used to characterize the characteristic attributes of the target object's emotion and / or concentration; in response to the current state recognition result of the target object, determining the target display imaging strategy in the correspondence between each state recognition result and the display imaging strategy; and performing interactive imaging on the target object according to the target display imaging strategy. The method is used to realize interactive control of multimedia interactive imaging equipment through state recognition results.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimedia interactive imaging method, device, storage medium and computer program product. Background Art

[0002] Brain Computer Interface (BCI) is a connection pathway directly established between the brain (or brain cells) and external devices. The EEG data of the target object can be obtained through the brain-computer interface technology, and then the EEG data can be identified and analyzed to obtain the EEG data recognition result of the target object. For example, the EEG data recognition result can be emotion classification, etc. Then, the target object is assisted in adjusting his or her own emotional state based on the EEG data recognition result.

[0003] Since the EEG data recognition results are abstract and imperceptible, the current application of EEG data recognition results often requires technicians to adjust the target object's emotional state based on the EEG data recognition results of the target object. The target object cannot achieve autonomous interaction with the EEG data recognition results, nor can it make further application control based on the EEG data recognition results. Summary of the invention

[0004] A technical problem to be solved by the embodiments of the present invention is to realize interactive imaging of EEG data recognition results.

[0005] In a first aspect, the present application provides a multimedia interactive imaging method. The method comprises:

[0006] Acquire EEG data of a target object, and determine a current state recognition result of the target object based on the EEG data; the current state recognition result is used to characterize characteristic attributes of the emotion and / or concentration of the target object;

[0007] In response to the current state recognition result of the target object, determining a target display imaging strategy in the corresponding relationship between each state recognition result and the display imaging strategy;

[0008] According to the target display imaging strategy, interactive imaging is performed on the target object.

[0009] In one embodiment, determining the current state recognition result of the target object according to the EEG data includes:

[0010] Performing filtering processing on the EEG data to obtain processed target EEG data;

[0011] Extracting EEG features from the target EEG data according to a preset feature extraction algorithm; the EEG features contain state information of the target object;

[0012] According to the EEG characteristics and a preset state recognition algorithm, a current state recognition result of the target object is determined.

[0013] In one embodiment, the state recognition result of the target object is an emotion recognition result, and the interactive imaging of the target object according to the target display imaging strategy includes:

[0014] According to the target display imaging strategy, determining display particles corresponding to the emotion recognition result and target attribute parameters of the display particles; the display particles are used to mark the interaction trajectory of the target object;

[0015] identifying an interaction trajectory of the target object according to the image data of the target object;

[0016] Interactive imaging is performed on the target object according to the display particles, the target attribute parameters of the display particles, the interactive trajectory and the image data of the target object.

[0017] In one embodiment, the state recognition result of the target object is a concentration recognition result, and the interactive imaging of the target object according to the target display imaging strategy includes:

[0018] According to the target display imaging strategy, when the concentration recognition result of the target object is high concentration, the light intensity is controlled to be greater than or equal to a preset light threshold, so as to reflect the image of the target object through the mirror surface of the plane glass to perform mirror interactive imaging;

[0019] When the concentration recognition result of the target object is low concentration, the light intensity is controlled to be less than a preset light threshold so as to display interactive imaging of the flat glass transmission display device.

[0020] In one of the embodiments, after controlling the light intensity to be less than a preset light threshold, the method further includes:

[0021] Acquire image data and depth data of the target object;

[0022] Determining the imaging position of the target object according to the depth data, and processing the image data to obtain imaging content;

[0023] The display interactive imaging of the target object on the display device is determined according to the emotion recognition result of the target object, the imaging position, the imaging content and the target display imaging strategy.

[0024] In one embodiment, the depth data includes the position coordinates of the target feature point of the target object; the position coordinates of the target feature point are coordinate values ​​in a coordinate system with a preset tilt angle of the depth camera; and determining the imaging position of the target object according to the depth data includes:

[0025] Performing a smoothing process on the position coordinates of the target feature point to obtain a processed first position coordinate of the target feature point;

[0026] According to the preset tilt angle, the first position coordinates are transformed to obtain second position coordinates in a vertical coordinate system;

[0027] The second position coordinates are reduced according to a preset ratio to obtain the imaging position of the target object.

[0028] In one embodiment, after determining the display interactive imaging of the target object on the display device according to the emotion recognition result of the target object, the imaging position, the imaging content and the target display imaging strategy, the method further includes:

[0029] In response to a displacement operation on the display interactive imaging, determining updated position information of the display interactive imaging after the displacement;

[0030] Output the interactive imaging of the target object after displacement according to the updated position information.

[0031] In one embodiment, the state recognition result of the target object is a concentration recognition result, and the method further includes:

[0032] When the concentration recognition result of the target object is high concentration, controlling the volume parameter of the audio device to meet the first preset audio condition;

[0033] When the concentration recognition result of the target object is low concentration, the volume parameter of the audio device is controlled to meet a second preset audio condition.

[0034] In a second aspect, the present application also provides a multimedia interactive imaging device. The multimedia interactive imaging device comprises: a supporting component and a display device;

[0035] The supporting member is used to support the display device;

[0036] The display device comprises a housing, a depth camera and a display device; the display device and the depth camera are fixed inside the housing; the transparent surface of the housing is parallel to the display interface of the display device;

[0037] The display device is used to respond to the current state recognition result of the target object, determine the target display imaging strategy in the corresponding relationship between each state recognition result and the display imaging strategy; and perform interactive imaging on the target object according to the target display imaging strategy.

[0038] In one embodiment, the multimedia interactive device further includes:

[0039] Light source equipment, used to provide display light source;

[0040] The controller is used to control the light intensity of the light source device in response to the concentration recognition result of the target object and according to the target display imaging strategy when the state recognition result of the target object is a concentration recognition result.

[0041] In one embodiment, the multimedia interactive device further includes:

[0042] An audio device, comprising a control unit and a speaker, wherein the control power supply is used to control a volume parameter of the speaker according to a state recognition result of the target object;

[0043] The speaker is used to play audio according to the volume parameter.

[0044] In one embodiment, the display device further includes:

[0045] A flat glass is arranged in the housing in parallel with the display device and is located between the display device and the light-transmitting surface of the housing, and is used to display the interactive imaging of the target object according to a preset second display imaging strategy and the intensity of the display light source.

[0046] In one of the embodiments, the depth camera is used to obtain image data and depth data of the target object;

[0047] The display device is used to determine the imaging position of the target object according to the depth data, and process the image data to obtain imaging content; and

[0048] The digital interactive imaging of the target object is determined according to the state recognition result of the target object, the imaging position, the imaging content and a preset second display imaging strategy.

[0049] In one embodiment, the depth data includes position coordinates of a target feature point of the target object; the position coordinates of the target feature point are coordinate values ​​in a coordinate system of the depth camera at a preset tilt angle;

[0050] The display device is further used to perform smoothing processing on the position coordinates of the target feature point to obtain a first position coordinate of the processed target feature point; the first position coordinate is symmetrical to the position coordinate of the mirror image of the target object;

[0051] According to the preset tilt angle, the first position coordinates are transformed to obtain second position coordinates in a vertical coordinate system;

[0052] The second position coordinates are reduced according to a preset ratio to obtain the imaging position of the target object.

[0053] In one of the embodiments, the display device is further used to provide an imaging position control button, and the imaging position control button is used to adjust the imaging position of the display interactive imaging of the target object in response to a trigger operation of the user.

[0054] In one embodiment, the support component includes a support rod and a base; the support rod is provided with a movable buckle for connecting the display device, and the height of the display device is adjusted by the movable buckle;

[0055] The base is provided with a moving component for moving the positions of the support rod and the display device.

[0056] In a third aspect, the present application further provides a multimedia interactive imaging system, the system comprising a brain-computer device and a multimedia interactive imaging device according to any one of the second aspects, wherein:

[0057] The brain-computer device is used to collect EEG data of a target object, determine a current state recognition result of the target object based on the EEG data; and send the current state recognition result to the multimedia interactive imaging device; the current state recognition result is used to characterize characteristic attributes of the emotion and / or concentration of the target object;

[0058] The multimedia interactive imaging device is used to respond to the current state recognition result of the target object, determine the target display imaging strategy in the corresponding relationship between each state recognition result and the display imaging strategy; and perform interactive imaging on the target object according to the target display imaging strategy.

[0059] In a fourth aspect, the present application further provides a multimedia interactive imaging device, the device comprising:

[0060] A first determination module is used to obtain EEG data of a target object, and determine a current state recognition result of the target object according to the EEG data; the state recognition result is used to characterize characteristic attributes of the emotion and / or concentration of the target object;

[0061] A second determination module, configured to determine a target display imaging strategy in a correspondence between each state recognition result and a display imaging strategy in response to a current state recognition result of the target object;

[0062] An imaging module is used to perform interactive imaging on the target object according to the target display imaging strategy.

[0063] In one embodiment, the first determining module is used to:

[0064] Performing filtering processing on the EEG data to obtain processed target EEG data;

[0065] Extracting EEG features from the target EEG data according to a preset feature extraction algorithm; the EEG features contain state information of the target object;

[0066] According to the EEG characteristics and a preset state recognition algorithm, a current state recognition result of the target object is determined.

[0067] In one embodiment, the state recognition result of the target object is an emotion recognition result, and the imaging module is specifically used to:

[0068] According to the target display imaging strategy, determining display particles corresponding to the emotion recognition result and target attribute parameters of the display particles; the display particles are used to mark the interaction trajectory of the target object;

[0069] identifying an interaction trajectory of the target object according to the image data of the target object;

[0070] Interactive imaging is performed on the target object according to the display particles, the target attribute parameters of the display particles, the interactive trajectory and the image data of the target object.

[0071] In one embodiment, the state recognition result of the target object is a concentration recognition result, and the imaging module is specifically used to:

[0072] According to the target display imaging strategy, when the concentration recognition result of the target object is high concentration, the light intensity is controlled to be greater than or equal to a preset light threshold, so as to reflect the image of the target object through the mirror surface of the plane glass to perform mirror interactive imaging;

[0073] When the concentration recognition result of the target object is low concentration, the light intensity is controlled to be less than a preset light threshold so as to display interactive imaging of the flat glass transmission display device.

[0074] In one embodiment, the device further comprises:

[0075] An acquisition module, used to acquire image data and depth data of the target object;

[0076] A processing module, used to determine the imaging position of the target object according to the depth data, and process the image data to obtain imaging content;

[0077] The third determination module is used to determine the display interactive imaging of the target object on the display device according to the emotion recognition result of the target object, the imaging position, the imaging content and the target display imaging strategy.

[0078] In one embodiment, the depth data includes the position coordinates of the target feature point of the target object; the position coordinates of the target feature point are coordinate values ​​in the coordinate system of the preset tilt angle of the depth camera; the processing module is specifically used to:

[0079] Performing a smoothing process on the position coordinates of the target feature point to obtain a processed first position coordinate of the target feature point;

[0080] According to the preset tilt angle, the first position coordinates are transformed to obtain second position coordinates in a vertical coordinate system;

[0081] The second position coordinates are reduced according to a preset ratio to obtain the imaging position of the target object.

[0082] In one embodiment, the device further comprises:

[0083] A fourth determining module, configured to determine updated position information of the display interactive imaging after displacement in response to a displacement operation on the display interactive imaging;

[0084] The output module is used to output the interactive imaging of the target object after displacement according to the updated position information.

[0085] In one embodiment, the state recognition result of the target object is a concentration recognition result, and the device further includes:

[0086] A first control module, configured to control a volume parameter of an audio device to meet a first preset audio condition when a concentration recognition result of the target object is high concentration;

[0087] The second control module is used to control the volume parameter of the audio device to meet a second preset audio condition when the concentration recognition result of the target object is low concentration.

[0088] In a fifth aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the first aspect are implemented.

[0089] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the first aspect are implemented.

[0090] In a seventh aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the first aspect are implemented.

[0091] The above-mentioned multimedia interactive imaging method, equipment, system, device, storage medium and computer program product obtain the EEG data of the target object, and determine the current state recognition result of the target object based on the EEG data; the state recognition result is used to characterize the characteristic attributes of the emotion and / or concentration of the target object; in response to the current state recognition result of the target object, the target display imaging strategy is determined in the correspondence between each state recognition result and the display imaging strategy; according to the target display imaging strategy, the target object is interactively imaged. Using this method, the brain-computer data of the target object is analyzed and identified to determine the state recognition result of the target object, and then, based on the state recognition result and the determined target display imaging strategy, the target object is interactively imaged, thereby realizing interactive control of the multimedia interactive imaging device through the state recognition result.

[0092] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention.

[0094] The present invention may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0095] Figure 1 A diagram of an application environment of a multimedia interactive imaging method in an embodiment;

[0096] Figure 2 A schematic diagram of a process of a multimedia interactive imaging method in one embodiment;

[0097] Figure 3 A schematic diagram of a flow chart of a step of determining a recognition result of a current state of a target object in one embodiment;

[0098] Figure 4 A schematic diagram of a flow chart of a step of determining an emotional state recognition result in one embodiment;

[0099] Figure 5 A schematic diagram of a flow chart of a step of determining a concentration recognition result in one embodiment;

[0100] Figure 6 It is a flowchart of the steps of displaying imaging corresponding to the emotional state recognition result in one embodiment;

[0101] Figure 7 A schematic diagram of a display strategy in emotion recognition results in one embodiment;

[0102] Figure 8 A schematic diagram of a flow chart of a step of adjusting an imaging position of interactive imaging in one embodiment;

[0103] Fig. 9 It is a schematic diagram of a flow chart of a display interactive imaging method for a display device in one embodiment;

[0104] Fig.10 A schematic diagram of a flow chart of a step of correcting a display interactive imaging position in an embodiment;

[0105] Fig.11 A schematic diagram of the depth camera coordinate system conversion in one embodiment;

[0106] Fig.12 A schematic diagram of imaging position correction in one embodiment;

[0107] Fig.13 A schematic diagram of a flow chart of a step of adjusting an imaging position of interactive imaging in one embodiment;

[0108] Fig.14 A schematic diagram of a process for controlling an audio device based on a concentration recognition result in one embodiment;

[0109] Fig.15 A flowchart of a method for controlling an audio device based on a concentration recognition result in one embodiment;

[0110] Fig.16 A schematic diagram of the structure of a multimedia interactive imaging device in one embodiment;

[0111] Fig.17 is a structural block diagram of a multimedia interactive imaging device in one embodiment;

[0112] Fig.18 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0113] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0114] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0115] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0116] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0117] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0118] Embodiments of the present invention may be applied to computer systems / servers that may operate with numerous other general or special computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer systems / servers include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0119] Computer systems / servers may be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. In general, program modules may include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

[0120] The multimedia interactive imaging method provided by the embodiment of the present invention can be applied to Figure 1The multimedia interactive imaging system 100 shown. Among them, the brain-computer device 102 communicates with the multimedia interactive imaging device 104 through the network. The data storage system can store the data that the multimedia interactive imaging device 104 needs to process. The data storage system can be integrated on the multimedia interactive imaging device 104, or it can be placed on the cloud or other network servers, which is not limited by the embodiment of the present invention. Specifically, the brain-computer device 102 obtains the EEG data of the target object. Then, the brain-computer device 102 determines the current state recognition result of the target object based on the EEG data. Among them, the current state recognition result is used to characterize the characteristic attributes of the emotion and / or concentration of the target object. Then, the multimedia interactive imaging device 104, in response to the current state recognition result of the target object, determines the target display imaging strategy in the corresponding relationship between each state recognition result and the display imaging strategy. Then, the multimedia interactive imaging device 104 performs interactive imaging on the target object according to the target display imaging strategy.

[0121] In one embodiment, Figure 2 As shown, a multimedia interactive imaging method is provided, which is applied to Figure 1 The multimedia interactive imaging system in the embodiment of the present invention is described, and includes the following steps:

[0122] Step 202, obtaining the EEG data of the target object, and determining the current state recognition result of the target object according to the EEG data.

[0123] The state recognition result is used to characterize the characteristic attributes of the target object's emotion and / or concentration.

[0124] In implementation, a multimedia interactive imaging system may include a brain-computer device and a multimedia interactive imaging device. The brain-computer device obtains the EEG data of the target object through brain-computer interface technology (BCI), and then extracts features from the EEG data of the target object according to a preset EEG feature extraction algorithm to obtain the EEG features of the target object. After obtaining the EEG features, the brain-computer device analyzes and processes the EEG features according to a preset feature recognition algorithm to obtain the current state recognition result of the target object. For example, the current state recognition result of the target object may include the emotion recognition result of the target object, wherein the emotion recognition result includes multiple types such as happiness, sadness, fear, anger, etc., which are used to reflect the current emotional characteristics of the target object.

[0125] Optionally, the embodiments of the present disclosure do not limit the above-mentioned preset EEG feature extraction algorithm. Specifically, a variety of feature extraction methods can be applied to the process of EEG data feature extraction to obtain the EEG features of the target object. Specifically, in the feature extraction process of EEG data, the EEG features extracted by the brain-computer device can be divided into three categories: time domain features, frequency domain features, and time-frequency domain features. Time domain features are mainly used to capture the temporal organization information of EEG signals. For example, Hjorth features, fractal dimension features, and high-order cross features are all time domain features. Frequency domain features mainly capture the emotional information of the target object from the perspective of the frequency domain. The time-frequency domain features capture both time domain information and frequency domain information, that is, extract frequency domain information from the unit time signal segmented by the sliding window. Since the sliding window segments the EEG data in the time dimension, the time domain information of the EEG data can also be obtained at the same time. For the above three characteristics, this application takes the extraction of frequency domain features as an example to illustrate that the brain-computer device can first decompose the original frequency band into several sub-frequency bands, and then extract the EEG characteristics of each frequency band respectively. The extraction methods that can be used include Fourier transform (RT), power spectral density (PSD), wavelet transform (WT) and differential entropy (DT).

[0126] Optionally, there are also a variety of recognition algorithms for EEG data, for example, supervised learning algorithms such as support vector machine (SVM), K-nearest neighbor (KNN) and naive Bayes (NB) in machine learning algorithms. The embodiment of the present invention does not limit the recognition algorithm for EEG data. The recognition algorithm for identifying the EEG features of the target object in this application will be specifically described later, and no further details will be given here.

[0127] Step 204 , in response to the current state recognition result of the target object, determine the target display imaging strategy in the corresponding relationship between each state recognition result and the display imaging strategy.

[0128] In implementation, a plurality of display imaging strategies are pre-stored in the multimedia interactive imaging device, and each display imaging strategy is used to control different interactive components in the multimedia interactive imaging device to execute corresponding interactive effects. In addition, a correspondence between each display imaging strategy and a state recognition result is pre-established, so that the multimedia interactive imaging device responds to the current state recognition result of the target object, and determines the target display imaging strategy in the correspondence of the current state recognition result among the plurality of correspondences, so that the multimedia interactive imaging device can execute the corresponding target display imaging strategy.

[0129] Step 206: interactively image the target object according to the target display imaging strategy.

[0130] In implementation, the target display imaging strategy determined based on the current state recognition result can be used to control the imaging content of the interactive imaging of the target object, and can also be used to control the light intensity of the light source device, etc. The embodiment of the present invention does not limit the specific content of the target display imaging strategy. Then, for the determined target display imaging strategy, the multimedia interactive imaging device can execute the target display imaging strategy to achieve interactive imaging of the target display effect of the target object.

[0131] In the above-mentioned multimedia interactive imaging method, the brain-computer device obtains the EEG data of the target object, and determines the current state recognition result of the target object based on the EEG data. Then, the multimedia interactive imaging device responds to the current state recognition result of the target object, and determines the target display imaging strategy in the correspondence between each state recognition result and the display imaging strategy. Furthermore, the multimedia interactive imaging device interactively images the target object according to the target display imaging strategy. Using this method, the brain-computer data of the target object is analyzed and identified to determine the state recognition result of the target object. Then, based on the state recognition result and the determined target display imaging strategy, the target object is interactively imaged, thereby realizing interactive control of the multimedia interactive imaging device through the state recognition result.

[0132] In one embodiment, Figure 3 As shown, the brain-computer device may be pre-stored with a program for realizing EEG data analysis and recognition, so that the EEG data can be analyzed and recognized by executing the program. An external computer device may also be connected, and the computer device stores a program for realizing EEG data analysis and recognition. After the brain-computer device collects the EEG data of the target object, it is transmitted to the computer device so that the computer device executes the program to analyze and process the EEG data. In the embodiment of the present invention, the program for realizing EEG data analysis and recognition stored in the brain-computer device is used as an example for explanation. In step 202, the current state recognition result of the target object is determined according to the EEG data. The specific implementation process is as follows:

[0133] Step 302: filter the EEG data to obtain processed target EEG data.

[0134] In implementation, for the original EEG data of the target object obtained, the brain-computer device first filters the original EEG data to obtain the processed target EEG data. Specifically, the brain-computer device contains characteristic attributes such as the target object's emotions and concentration (or attention), and different data processing strategies can be executed for different characteristics. For example, for the emotional state attribute of the target object, the brain-computer device can process the original EEG data according to a 45-55Hz bandpass filter and a 0.5-50Hz bandpass filter to obtain the processed target EEG data. The processed target EEG data is the EEG data in which the data of each channel is divided into 5 main frequency bands. For another example, for the attention state attribute of the target object, the brain-computer device can process the original EEG data based on a high-pass filter, a low-pass filter, a 50Hz band filter, and a 100Hz band filter to obtain the processed target EEG data.

[0135] Step 304: extracting EEG features from the target EEG data according to a preset feature extraction algorithm.

[0136] The EEG features include the state information of the target object, which includes the emotional state information and attention state information of the target object.

[0137] In the implementation, the brain-computer device extracts EEG features from the target EEG data according to a preset feature extraction algorithm. Specifically, the brain-computer device can extract features based on the emotional state of the target object's EEG data, mainly extracting time domain features from the EEG data, such as Hjorth features (the Hjorth features include the activity, mobility and complexity of the EEG data) and DE (differential entropy) features.

[0138] The brain-computer device performs feature extraction on the attention state in the EEG data of the target object according to the preset feature extraction algorithm. The specific extraction process is: the brain-computer device divides the filtered target EEG data into a window (also called a time window) with 125 data points, and then each data channel corresponds to 10 windows, and the EEG data corresponds to a total of 5 data channels. Then, the brain-computer device calculates the data standard deviation of these 10 windows in each channel. Then, the brain-computer device compares the data standard deviation of each window with the preset standard deviation threshold, and determines the window with a data standard deviation greater than the preset standard deviation threshold as a blink window. Furthermore, the brain-computer device counts the total number of blink windows of each data channel within 5 seconds (preset time period). The total number of blink windows is used as the feature extraction of the concentration state of the target object.

[0139] Step 306, determining the current state recognition result of the target object according to the EEG characteristics and a preset state recognition algorithm.

[0140] In implementation, the brain-computer device determines the current state recognition result of the target object based on the EEG features and the preset state recognition algorithm. Specifically, there are multiple preset state recognition algorithms. For different EEG features, different state recognition algorithms (or feature recognition algorithms) can be selected to determine the current state recognition result of the target object.

[0141] The state recognition algorithm may be a traditional machine learning algorithm, such as a support vector machine algorithm, a K-nearest neighbor algorithm, a naive Bayes algorithm, etc. Specifically, Figure 4 As shown, in the application process of the state recognition algorithm, when the present application performs feature recognition on the emotional state of the target object, the random forest algorithm can be used to classify and recognize the target EEG features of the target object to obtain an emotion recognition result for characterizing the target object's emotion category, such as "happy", "relaxed", "sad" and "fearful", etc. The embodiment of the present invention does not limit the type of emotion recognition results. Figure 5 As shown, in the present application, when the focus state of the target object is identified, the extracted focus feature of the target object can be identified based on the preset focus calculation method. The focus feature identification process is: the brain-computer device calculates the focus value of the target object based on the sum of the number of blink windows within a preset time period (for example, 5 seconds). Then, the focus identification result of the target object is determined based on the focus value and the preset focus threshold.

[0142] Optionally, the concentration value calculated by the sum of the number of blink windows may be smoothed to obtain a final concentration value. Then, the concentration recognition result (ie, current state recognition result) of the target object is determined according to the final concentration value and a preset concentration threshold.

[0143] Optionally, the concentration thresholds of the corresponding concentration recognition results in different target display imaging strategies are different. In one example, in the target display imaging strategy for controlling the light source device, the concentration threshold includes an upper threshold and a lower threshold, the upper threshold is 85, and the lower threshold is 60; in the target display imaging strategy for controlling the audio device, the concentration threshold includes an upper threshold and a lower threshold, the upper threshold is 80, and the lower threshold is 30. Different concentration levels can be divided for different concentration thresholds, which is not limited in the embodiments of the present invention.

[0144] In one embodiment, Figure 6As shown, by analyzing and processing the EEG data of the target object, the state recognition result of the target object can be obtained, wherein the state recognition result of the target object is an emotion recognition result, and the emotion recognition result represents an emotion category of the target object. Based on the emotion recognition result, the multimedia interactive imaging device can determine the corresponding target display imaging strategy. In step 206, according to the target display imaging strategy, the target object is interactively imaged, and the specific implementation process is as follows:

[0145] In step 602, display particles corresponding to the emotion recognition results and target attribute parameters of the display particles are determined according to the target display imaging strategy.

[0146] Among them, display particles are used to mark the interaction trajectory of the target object;

[0147] In implementation, since the emotion recognition results may include multiple types, such as happiness, sadness, fear, relaxation, etc., the multimedia interactive imaging device matches each emotion recognition result of the target object with a corresponding visual display particle in the display imaging strategy for each emotion recognition result. Furthermore, when the target display imaging strategy is determined based on the current emotion recognition result of the target object, the multimedia interactive imaging device can determine the display particles corresponding to the specific type of emotion recognition result and the target attribute parameters of the display particles. Among them, the target attribute parameters of the display particles may include, but are not limited to, color parameters, shape parameters, and brightness parameters of the display particles.

[0148] For example, when the current emotion recognition result of the target object is "happy", the target display imaging strategy corresponding to "happy" determines the display particles used to mark the interactive trajectory of the target object and the target attribute parameters of the display particles. The imaging effect of the display particles under the target attribute parameter configuration is: the display color of the display particles is pink, the display brightness is bright, and the particle shape is round, etc.

[0149] Optional, such as Figure 7 As shown, different target display imaging strategies determined by corresponding emotion recognition results can correspond to different display particle attribute parameters to distinguish different emotion types. Specifically, taking the color attribute parameters of the display particles as an example, when the current emotion recognition result of the target object is "happy", the color attribute parameters of the display particles correspond to pink, when the current emotion recognition result of the target object is "fear", the color attribute parameters of the display particles correspond to orange, when the current emotion recognition result of the target object is "sad", the color attribute parameters of the display particles correspond to blue, and when the current emotion recognition result of the target object is "relaxed", the color attribute parameters of the display particles correspond to green. The embodiment of the present invention does not limit the target display strategy for distinguishing different emotion recognition results.

[0150] In step 604 , the interaction trajectory of the target object is identified according to the image data of the target object.

[0151] In implementation, the multimedia interactive imaging device identifies the interactive trajectory of the target object based on the image data of the target object. Specifically, for each two adjacent image frames contained in the image data of the target object, the multimedia interactive imaging device can compare the adjacent image frames, identify the displacement change of the target part of the target object contained in the adjacent image frames, and calculate the displacement difference. Furthermore, the multimedia interactive imaging device can determine the change trajectory of the target part of the target object based on the displacement difference and the displacement change.

[0152] In step 606, interactive imaging is performed on the target object according to the display particles, the target attribute parameters of the display particles, the interactive trajectory, and the image data of the target object.

[0153] In implementation, the multimedia interactive imaging device performs interactive imaging of the target object according to the display particles configured in the target display strategy, the target attribute parameters of the display particles, the determined interactive trajectory, and the image data of the target object.

[0154] In one embodiment, Figure 8 As shown, the state recognition result of the target object can be obtained by analyzing and processing the EEG data of the target object, wherein the state recognition result of the target object is a concentration recognition result, and the display imaging strategy set for the state recognition result of the concentration type in the multimedia interactive imaging device includes a control instruction for controlling the light intensity. Furthermore, according to the concentration recognition result of the target object, the multimedia interactive imaging device can control the light intensity of the light source device. Since the concentration recognition result can include different concentration levels, for example, divided into two levels of high concentration and low concentration. Then, according to different concentration recognition results, the multimedia interactive imaging device executes the corresponding target display imaging strategy. In step 106, according to the target display imaging strategy, the specific implementation process of interactive imaging of the target object is as follows:

[0155] Step 802, according to the target display imaging strategy, when the concentration recognition result of the target object is low concentration, the light intensity is controlled to be greater than or equal to the preset light threshold, so as to reflect the image of the target object through the mirror surface of the flat glass to perform mirror interactive imaging.

[0156] In implementation, the multimedia interactive imaging device includes a display device, and the shell of the display device can encapsulate a depth camera and a display device, and the display device can also include a flat glass, which is located between the display device and the light-transmitting surface of the shell (that is, the display area surface of the multimedia interactive imaging device). The flat glass has the characteristics of having different light presentation states for different light intensities. Specifically, when the concentration recognition result of the target object is a high concentration, the display device executes the corresponding target display imaging strategy, and controls the light intensity of the light source device to be greater than or equal to the preset light threshold (that is, the light intensity is high intensity) according to the control instructions of the target display imaging strategy, so that the flat glass has the characteristics of strong reflection and weak transmission for high-intensity light. The flat glass appears as a mirror surface, and then the display device performs mirror interactive imaging of the target object through the mirror reflection of the mirror surface.

[0157] Step 804, when the concentration recognition result of the target object is low concentration, control the light intensity to be less than a preset light threshold, so as to display interactive imaging through the flat glass transmission display device.

[0158] In implementation, when the target object's concentration recognition result is high concentration, the display device in the display device controls the light intensity of the light source device to be less than the preset light threshold (i.e., the light intensity is low intensity) based on the control instructions in the target display imaging strategy, so that the flat glass has the characteristics of weak reflection and strong transmission for low-intensity light. The flat glass appears as a lens, and then the display interactive imaging is performed by transmitting the display device located behind the flat glass.

[0159] Optionally, the display device may be an all-in-one device having a display unit and a data processing unit, or a combined device including a display and a data processor. Therefore, the specific structure of the display device may be selected according to actual needs and is not limited in this application.

[0160] In a specific embodiment, the display device in the display device performs interactive imaging using flat glass as a medium according to the focus recognition result of the target object and the corresponding target display imaging strategy, and the specific process includes:

[0161] Step 1: When the target object's concentration recognition result is low concentration (i.e., the concentration is lower than the lower limit threshold of concentration (such as a value of 60)), the display interactive imaging in the display device is not displayed, and the target object only sees its own mirror image interactive imaging on the flat glass (mirror).

[0162] Step 2: When the target object's concentration recognition result is medium concentration (i.e., the concentration is higher than the lower concentration threshold (e.g., a value of 60)) but lower than the upper concentration threshold (e.g., a value of 100), the display interactive image on the display device slowly emerges, and the visibility changes with the concentration level;

[0163] Step three, when the concentration recognition result of the target object is high concentration (that is, the concentration is higher than the upper concentration threshold (such as a value of 100)), the display interactive imaging in the display device is completely visible.

[0164] Step 4: displaying an interactive image in a display device of the display apparatus, wherein the interactive image is a display color and a display style determined according to the emotion recognition result of the user.

[0165] In one embodiment, Fig. 9 As shown, after controlling the light intensity to be less than the preset light threshold in step 804, the target object can observe the display interactive imaging of the display device in the multimedia interactive imaging device, and further, the multimedia interactive imaging method further includes:

[0166] Step 902: Acquire image data and depth data of the target object.

[0167] In implementation, the display device includes a depth camera and a display device, and the depth camera is connected to the display device in the display device, wherein the depth camera is used to obtain image data and depth data of a target object.

[0168] Step 904 , determining the imaging position of the target object according to the depth data, and processing the image data to obtain imaging content.

[0169] In implementation, the display device is pre-installed with touchdesigner (multimedia special effects interactive software), and the display device (or specifically a data processor in the display device) runs the touchdesigner software to determine the imaging position of the target object in the display according to the depth data. In addition, the display device can also process the image data of the target object to determine the imaging content of the target object in the display.

[0170] Step 906 , determining the display interactive imaging of the target object on the display device according to the emotion recognition result, imaging position, imaging content and target display imaging strategy of the target object.

[0171] In implementation, the target object's display interactive imaging in the display device is determined based on the target object's emotion recognition result, the target display imaging strategy corresponding to the emotion recognition result, the image containing the target object (i.e., the imaging content), and the determined imaging position of the target object. The display interactive imaging includes the display special effects corresponding to the emotion recognition result, such as the special effects of adding display particles to the interactive track of the target object in the above step 602.

[0172] In one embodiment, Fig.10 As shown, due to the position difference between the human eye and the depth camera, the display device in the multimedia interactive imaging device often cannot combine the data directly collected by the depth camera with the reflection of the mirror surface (i.e., flat glass) of the target object when performing interactive imaging, which will cause visual imaging dislocation. Therefore, the display device needs to process the imaging position data (depth data) of the target object. The specific processing process of determining the imaging position of the target object according to the depth data in step 904 includes:

[0173] Step 1002 , smoothing the position coordinates of the target feature point to obtain the first position coordinates of the target feature point after processing.

[0174] In implementation, in the display device, the depth camera has a certain installation angle, which is the inclination angle of the depth camera relative to the horizontal direction. Therefore, the position coordinates of the depth data collected by the depth camera are the position coordinates in the coordinate system with an inclination angle to the horizontal plane. Therefore, in the display device, the display device needs to process the position coordinates of the target feature point in the coordinate system with the inclination angle to obtain the processed first position coordinates.

[0175] Optionally, the acquired depth data of the target object may be the depth position coordinates of the target feature point of the target object, where the target feature point is a point pre-marked in the image frame of the target object. Therefore, the display device may first determine the target point among the target feature points (for example, the target point is selected as the target feature point with the center position of the left and right eyes of the face among the target feature points), use the target point as the center point, and perform smoothing processing on the other target feature points, thereby obtaining the first position coordinates of the processed target feature point.

[0176] Step 1004: transform the coordinate value of the first position coordinate according to a preset tilt angle to obtain the second position coordinate in a vertical coordinate system.

[0177] In implementation, since the depth camera has a tilt angle, the display device transforms the first position coordinates of the target feature point into the second position coordinates in the vertical coordinate system according to the tilt angle. Fig.11As shown, the vertical coordinate system is a coordinate system parallel to the horizontal plane (no inclination angle). The second position coordinate of the target object in the vertical coordinate system is the position coordinate symmetrical to each target feature point of the mirror image of the target object, which means that the position difference between the depth camera imaging and the human eye visual imaging is eliminated.

[0178] Step 1006: Reduce the second position coordinates according to a preset ratio to obtain an imaging position of the target object.

[0179] In implementation, the display device reduces the determined second position coordinates according to a preset ratio to obtain the imaging position of the target object. The preset ratio can be determined based on the display interface of the display device, and the embodiment of the present invention does not limit this.

[0180] Specifically, in the above embodiment, the principle of coordinate transformation of the target feature point of the target object is as follows: Fig.12 As shown, the depth data of the target object is transformed into a coordinate system, and the transformed second position coordinates are based on the mirror image of the target object ( Fig.12 The imaging position of the image (the virtual image in the image) is symmetrical, and then the final imaging position in the display device is determined based on a preset reduction ratio, thereby eliminating the position difference between the depth camera imaging and the human eye visual imaging.

[0181] In one embodiment, Fig.13 As shown, after step 906, the multimedia interactive imaging method further includes:

[0182] Step 1302 : In response to a displacement operation on the display interactive imaging, determining updated position information of the display interactive imaging after the displacement.

[0183] In implementation, a control option for controlling the imaging position is provided in the display device, and the target object can trigger the control option to generate a displacement control instruction for the display interactive imaging, and adjust the imaging position of the display interactive imaging. Specifically, the target object uses its own image reflected by the plane glass as a reference to trigger the position adjustment of the display interactive imaging in the display device, and the display device responds to the displacement instruction for the display interactive imaging and determines the updated position information of the display interactive imaging after displacement. Optionally, the updated position information can be the updated position coordinates of each target feature point contained in the display interactive imaging.

[0184] Step 1304: outputting the interactive imaging of the target object after displacement according to the updated position information.

[0185] In implementation, the display device outputs the interactive display image of the target object after displacement according to the updated position information. The interactive display image can be more accurately aligned with the mirror image of the target object.

[0186] In one embodiment, Fig.14 As shown, when the state recognition result of the target object is a concentration recognition result, the multimedia interactive imaging device also includes a control strategy for controlling the audio volume, and the multimedia interactive imaging method also includes:

[0187] Step 1402: When the concentration recognition result of the target object is high concentration, control the volume parameter of the audio device to meet the first preset audio condition.

[0188] In implementation, when the target object's concentration recognition result is high concentration, the multimedia interactive imaging device controls the volume parameter of the audio device to meet the first preset audio condition based on the control instruction in the target control strategy. Fig.15 As shown, the first preset audio condition may be, but is not limited to, a noise volume index greater than or equal to a preset noise threshold, a main melody volume index less than a preset melody threshold, and a treble volume index value of 0.

[0189] Step 1404 , when the concentration recognition result of the target object is low concentration, control the volume parameter of the audio device to meet the second preset audio condition.

[0190] In implementation, when the concentration recognition result of the target object is low concentration, the multimedia interactive imaging device controls the volume parameter of the audio device to meet the first preset audio condition based on the control instruction in the target control strategy. Specifically, the second preset audio condition can be, but is not limited to, the noise volume index is 0, the main melody volume index is less than the preset melody threshold, and the treble volume index is greater than or equal to the preset treble threshold.

[0191] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0192] In one embodiment, Fig.16 As shown, a multimedia interactive imaging device 1600 is provided, and the multimedia interactive imaging device includes: a supporting component 1610 and a display device 1620;

[0193] A supporting member 1610, used for supporting a display device;

[0194] The display device 1620 includes a housing, a depth camera and a display device. The display device and the depth camera are fixed inside the housing; and the light-transmitting surface of the housing is parallel to the display interface of the display device.

[0195] The display device is used to respond to the current state recognition result of the target object, determine the target display imaging strategy in the corresponding relationship between each state recognition result and the display imaging strategy; and perform interactive imaging on the target object according to the target display imaging strategy.

[0196] In one embodiment, Fig.16 As shown, the multimedia interactive imaging device 1600 also includes:

[0197] Light source equipment, used to provide display light source.

[0198] The controller is used to control the light intensity of the light source device in response to the focus recognition result of the target object and according to the target display imaging strategy.

[0199] Optionally, the controller can also be integrated into the light source device to control the light intensity of the light emitting components in the light source device. In addition, the light source device can be fixed on the supporting component of the multimedia interactive imaging device to achieve the integration of the multimedia interactive imaging device.

[0200] In one embodiment, the multimedia interactive imaging device 1600 further includes:

[0201] The audio device includes a control unit and a speaker, wherein the control power supply is used to control the volume parameter of the speaker according to the state recognition result of the target object;

[0202] Speaker, used to play audio according to volume parameters.

[0203] Optionally, the audio device may be disposed in a housing of the display device, specifically, may be connected to the display by being fixed on a bracket of the display. The embodiment of the present invention does not limit the position of the audio device in the display device.

[0204] In one embodiment, the display device 1620 further includes:

[0205] The flat glass is arranged in the housing in parallel with the display device and is located between the display device and the light-transmitting surface of the housing, and is used to display the interactive imaging of the target object according to a preset second display imaging strategy and the intensity of the display light source.

[0206] In one embodiment, the depth camera is used to obtain image data and depth data of the target object;

[0207] The display device is used to determine the imaging position of the target object according to the depth data, and process the image data to obtain the imaging content; according to the state recognition result, imaging position, imaging content and preset second display imaging strategy of the target object, the digital interactive imaging of the target object is determined.

[0208] In implementation, the display device processes the image data and depth data of the target object as described in steps 902 to 906 of the above method embodiment, which will not be described in detail in this embodiment.

[0209] In one embodiment, the depth data includes the position coordinates of a target feature point of the target object; the position coordinates of the target feature point are coordinate values ​​in a coordinate system of a depth camera at a preset tilt angle;

[0210] The display device is also used to perform smoothing processing on the position coordinates of the target feature point to obtain the first position coordinates of the processed target feature point, wherein the first position coordinates are symmetrical to the position coordinates of the mirror image of the target object.

[0211] It is also used to transform the coordinate value of the first position coordinate according to a preset tilt angle to obtain the second position coordinate in the vertical coordinate system; and to reduce the second position coordinate according to a preset ratio to obtain the imaging position of the target object.

[0212] In one embodiment, the display device is further used to provide an imaging position control button, wherein the imaging position control button is used to adjust the imaging position of the display interactive imaging of the target object in response to a trigger operation of the user.

[0213] In one embodiment, the support component 1610 includes a support rod and a base; a movable snap-in is provided on the support rod for connecting the display device, and the height of the display device is adjusted by moving the snap-in;

[0214] The base is provided with a moving part for moving the position of the supporting rod and the display device.

[0215] In this embodiment, equipment integration is achieved by integrating hardware such as depth cameras, light source equipment, and audio equipment, and interactive imaging under different state recognition results of the target object is achieved based on devices such as display devices included in the multimedia interactive imaging equipment, thereby overcoming the limitation of the larger layout space required to achieve display imaging.

[0216] Based on the same inventive concept, as mentioned above Figure 1As shown, the embodiment of the present invention further provides a multimedia interactive imaging system 100 for implementing the multimedia interactive imaging method involved above. The implementation scheme for solving the problem provided by the system 100 is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multimedia interactive imaging system embodiments provided below can refer to the limitations of the multimedia interactive imaging method above, and will not be repeated here.

[0217] In one embodiment, a multimedia interactive imaging system 100 is provided, the system 100 comprising a brain-computer device and the above-mentioned multimedia interactive imaging device: wherein:

[0218] The brain-computer device is used to collect EEG data of the target object, determine the current state recognition result of the target object according to the EEG data; and send the current state recognition result to the multimedia interactive imaging device; the current state recognition result is used to characterize the characteristic attributes of the emotion and / or concentration of the target object;

[0219] The multimedia interactive imaging device is used to respond to the current state recognition result of the target object, determine the target display imaging strategy in the corresponding relationship between each state recognition result and the display imaging strategy; and perform interactive imaging on the target object according to the target display imaging strategy.

[0220] In an embodiment of the present invention, the multimedia interactive imaging system can collect EEG data of the target object through a brain-computer device, analyze and identify the brain-computer data of the target object, determine a state recognition result of the target object, and send the current state recognition result to the multimedia interactive imaging device. Then, the multimedia interactive imaging device performs interactive imaging of the target object based on the state recognition result and the determined target display imaging strategy, thereby realizing interactive control of the multimedia interactive imaging device through the state recognition result.

[0221] Based on the same inventive concept, an embodiment of the present invention further provides a multimedia interactive imaging device for implementing the multimedia interactive imaging method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more multimedia interactive imaging device embodiments provided below can refer to the limitations of the multimedia interactive imaging method above, and will not be repeated here.

[0222] In one embodiment, Fig.17 As shown, a multimedia interactive imaging device is provided, including: a first determination module 1710, a second determination module 1720 and an imaging module 1730, wherein:

[0223] The first determination module 1710 is used to obtain EEG data of the target object and determine the current state recognition result of the target object according to the EEG data; the state recognition result is used to characterize the characteristic attributes of the emotion and / or concentration of the target object;

[0224] A second determination module 1720 is used to determine a target display imaging strategy in the correspondence between each state recognition result and the display imaging strategy in response to the current state recognition result of the target object;

[0225] The imaging module 1730 is used to perform interactive imaging of the target object according to the target display imaging strategy.

[0226] In one embodiment, the first determining module 1710 is specifically configured to:

[0227] Performing filtering on the EEG data to obtain processed target EEG data;

[0228] Extracting EEG features from target EEG data according to a preset feature extraction algorithm; the EEG features contain state information of the target object;

[0229] According to the EEG characteristics and the preset state recognition algorithm, the current state recognition result of the target object is determined.

[0230] In one embodiment, the state recognition result of the target object is an emotion recognition result, and the imaging module 1730 is specifically used to:

[0231] According to the target display imaging strategy, display particles corresponding to the emotion recognition results and target attribute parameters of the display particles are determined; the display particles are used to mark the interaction trajectory of the target object;

[0232] Identify the interaction trajectory of the target object based on the image data of the target object;

[0233] According to the display particles, target attribute parameters of the display particles, the interactive trajectory and the image data of the target object, the target object is interactively imaged.

[0234] In one embodiment, the state recognition result of the target object is a concentration recognition result, and the imaging module 1730 is specifically used to:

[0235] According to the target display imaging strategy, when the concentration recognition result of the target object is high concentration, the light intensity is controlled to be greater than or equal to the preset light threshold, so as to reflect the image of the target object through the mirror surface of the plane glass to perform mirror interactive imaging;

[0236] When the concentration recognition result of the target object is low concentration, the light intensity is controlled to be less than a preset light threshold so as to display interactive imaging of the display device through the flat glass transmission.

[0237] In one embodiment, the apparatus 1700 further includes:

[0238] An acquisition module, used to acquire image data and depth data of a target object;

[0239] A processing module is used to determine the imaging position of the target object according to the depth data, and process the image data to obtain the imaging content;

[0240] The third determination module is used to determine the display interactive imaging of the target object on the display device according to the emotion recognition result, imaging position, imaging content and target display imaging strategy of the target object.

[0241] In one embodiment, the depth data includes the position coordinates of the target feature points of the target object; the position coordinates of the target feature points are coordinate values ​​in a coordinate system with a preset tilt angle of the depth camera; the processing module is specifically used to:

[0242] Smoothing the position coordinates of the target feature point to obtain the first position coordinates of the target feature point after processing;

[0243] According to a preset tilt angle, coordinate values ​​of the first position coordinates are transformed to obtain second position coordinates in a vertical coordinate system;

[0244] The second position coordinate is reduced according to a preset ratio to obtain the imaging position of the target object.

[0245] In one embodiment, the apparatus 1700 further includes:

[0246] A fourth determination module, configured to determine updated position information of the display interactive imaging after displacement in response to a displacement operation on the display interactive imaging;

[0247] The output module is used to output the interactive imaging of the displacement of the target object according to the updated position information.

[0248] In one embodiment, the state recognition result of the target object is a concentration recognition result, and the device further includes:

[0249] A first control module, configured to control a volume parameter of an audio device to meet a first preset audio condition when a concentration recognition result of the target object is high concentration;

[0250] The second control module is used to control the volume parameter of the audio device to meet the second preset audio condition when the concentration recognition result of the target object is low concentration.

[0251] In an embodiment of the present invention, the multimedia interactive imaging device can analyze and identify the brain-computer data of the target object, determine the state recognition result of the target object, and then, based on the state recognition result and the determined target display imaging strategy, perform interactive imaging of the target object, thereby realizing interactive control of the multimedia interactive imaging device through the state recognition result.

[0252] Each module in the above-mentioned multimedia interactive imaging device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0253] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.18 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a multimedia interactive imaging method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0254] Those skilled in the art will understand that Fig.18 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0255] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned multimedia interactive imaging method are implemented.

[0256] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the above-mentioned multimedia interactive imaging method when executed by a processor.

[0257] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0258] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0259] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0260] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0261] The description of the present invention is given for the purpose of illustration and description, and is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for particular uses.

Claims

1. A multimedia interactive imaging method, characterized in that: include: Acquire EEG data of a target object, and determine a current state recognition result of the target object according to the EEG data; The current state recognition result is used to characterize the characteristic attributes of the emotion and concentration of the target object; In response to the current state recognition result of the target object, determining a target display imaging strategy in the corresponding relationship between each state recognition result and the display imaging strategy; Performing interactive imaging on the target object according to the target display imaging strategy; The state recognition result of the target object is a concentration recognition result, and the interactive imaging of the target object according to the target display imaging strategy includes: According to the target display imaging strategy, when the concentration recognition result of the target object is low concentration, the light intensity is controlled to be greater than or equal to a preset light threshold, so as to reflect the image of the target object through the mirror surface of the plane glass to perform mirror interactive imaging; When the concentration recognition result of the target object is high concentration, controlling the light intensity to be less than a preset light threshold, so as to display interactive imaging of the display device through the flat glass transmission; The method further comprises: Acquire image data and depth data of the target object; Determining the imaging position of the target object according to the depth data, and processing the image data to obtain imaging content; The display interactive imaging of the target object on the display device is determined according to the emotion recognition result of the target object, the imaging position, the imaging content and the target display imaging strategy.

2. The method according to claim 1, characterized in that Determining the current state recognition result of the target object according to the EEG data includes: Performing filtering processing on the EEG data to obtain processed target EEG data; Extracting EEG features from the target EEG data according to a preset feature extraction algorithm; the EEG features contain state information of the target object; According to the EEG characteristics and a preset state recognition algorithm, a current state recognition result of the target object is determined.

3. The method according to claim 1, characterized in that The step of determining the display interactive imaging of the target object on the display device according to the emotion recognition result of the target object, the imaging position, the imaging content, and the target display imaging strategy includes: Determine the display interactive imaging of the target object in the display device according to the target display imaging strategy corresponding to the emotion recognition result, the image containing the target object, and the determined imaging position of the target object; Among them, the display interactive imaging includes display effects corresponding to the emotion recognition results; the display effects are to determine the display particles corresponding to the emotion recognition results and the target attribute parameters of the display particles according to the target display imaging strategy; the display particles are used to mark the interactive trajectory of the target object; the interactive trajectory of the target object is identified according to the image data of the target object; and the target object is interactively imaged according to the display particles, the target attribute parameters of the display particles, the interactive trajectory and the image data of the target object.

4. The method according to claim 1, characterized in that: The depth data includes the position coordinates of the target feature points of the target object; the position coordinates of the target feature points are coordinate values ​​in a coordinate system with a preset tilt angle of the depth camera; Determining the imaging position of the target object according to the depth data includes: Performing a smoothing process on the position coordinates of the target feature point to obtain a processed first position coordinate of the target feature point; According to the preset tilt angle, the first position coordinates are transformed to obtain second position coordinates in a vertical coordinate system; The second position coordinates are reduced according to a preset ratio to obtain the imaging position of the target object.

5. The method according to claim 1, characterized in that After determining, based on the emotion recognition result of the target object, the imaging position, the imaging content, and the target display imaging strategy, that the target object is displayed on the display device for interactive imaging, the method further includes: In response to a displacement operation on the display interactive imaging, determining updated position information of the display interactive imaging after the displacement; Output the interactive imaging of the target object after displacement according to the updated position information.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: When the concentration recognition result of the target object is high concentration, controlling the volume parameter of the audio device to meet the first preset audio condition; When the concentration recognition result of the target object is low concentration, the volume parameter of the audio device is controlled to meet a second preset audio condition.

7. A multimedia interactive imaging device, characterized in that: The device comprises: A first determination module is used to obtain EEG data of a target object, and determine a current state recognition result of the target object according to the EEG data; the state recognition result is used to characterize characteristic attributes of the emotion and concentration of the target object; A second determination module, configured to determine a target display imaging strategy in a correspondence between each state recognition result and a display imaging strategy in response to a current state recognition result of the target object; An imaging module, used for interactively imaging the target object according to the target display imaging strategy; The imaging module is specifically used to: according to the target display imaging strategy, when the concentration recognition result of the target object is high concentration, control the light intensity to be greater than or equal to the preset light threshold, so as to reflect the image of the target object through the mirror surface of the plane glass to perform mirror interactive imaging; When the concentration recognition result of the target object is low concentration, the light intensity is controlled to be less than a preset light threshold, so as to display interactive imaging of the display device through the flat glass transmission; The device also includes: An acquisition module, used to acquire image data and depth data of a target object; A processing module, used to determine the imaging position of the target object according to the depth data, and process the image data to obtain imaging content; The third determination module is used to determine the display interactive imaging of the target object on the display device according to the emotion recognition result, imaging position, imaging content and target display imaging strategy of the target object.

8. The device according to claim 7, characterized in that The first determination module is specifically used to filter the EEG data to obtain processed target EEG data; Extracting EEG features from target EEG data according to a preset feature extraction algorithm; The EEG features contain the state information of the target object; According to the EEG characteristics and the preset state recognition algorithm, the current state recognition result of the target object is determined.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images