Data information processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data information processing method and device, a storage medium and electronic equipment. Relates to the technical field of artificial intelligence. The method comprises the steps that an image data set of a target object is acquired, and the image data set comprises a plurality of images of the target object in a target time period; based on the image data set, an emotion curve of the target object is drawn, a target curve graph is obtained, the emotion curve of the target object at least comprises a titer curve of the target object and a wake-up curve of the target object, and the titer curve of the target object is used for recording the emotion pleasure degree of the target object. The wake-up curve is used for recording the activated degree of the body of the target object; and inputting the target curve graph into a classification model to obtain an output result, and determining an emotion type corresponding to the target object based on the output result. The technical problem that the recognition result is inaccurate when the emotion of the driver is recognized by adopting a recognition method in a discrete form in the related technology is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data information processing method, device, storage medium and electronic device. Background Art

[0002] At present, in actual emotion recognition applications, it is usually based on machine learning or deep learning models, using training data sets of face images with category labels to learn the feature representations of different emotion categories. However, this discrete emotion recognition method has inconsistent classification standards and granularity. In addition, it is affected by AI (Artificial Intelligence) visual cameras and different algorithms, and the accuracy and recall rate are also different. In addition, discrete emotion recognition is difficult to capture more subtle emotional fluctuations and gradual changes in emotional states. Therefore, the emotion recognition methods in related technologies often have inaccurate recognition results.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide a data information processing method, device, storage medium and electronic device to at least solve the technical problem that a discrete recognition method is used in the related art to recognize the driver's emotions, resulting in inaccurate recognition results.

[0005] According to one aspect of an embodiment of the present invention, a method for processing data information is provided, comprising: acquiring an image data set of a target object, wherein the target object is located in a target vehicle, and the image data set includes: multiple images of the target object within a target time period; based on the image data set, drawing an emotion curve of the target object to obtain a target curve graph, wherein the emotion curve of the target object includes at least: a valence curve of the target object and an arousal curve of the target object, the valence curve of the target object is used to record the degree of pleasure of the target object's emotions, and the arousal curve is used to record the degree to which the target object's body is activated; inputting the target curve graph into a classification model to obtain an output result, and determining the emotion type corresponding to the target object based on the output result, wherein the classification model includes: a model obtained after training a neural network model using emotion curves of multiple objects and the emotion type of the emotion curve of each object as training samples.

[0006] Furthermore, the target curve graph is input into a classification model to obtain an output result, including: inputting the target curve graph into the classification model, using the classification model to extract features of the emotion curve in the target curve graph to obtain curve features, wherein the curve features include at least one of the following: the curve amplitude of the target curve graph, the peak value of the target curve graph, the valley value of the target curve graph, the correlation between the valence curve of the target object and the arousal curve of the target object; based on the curve features, using the classification model to identify the emotion type corresponding to the target object to obtain the output result.

[0007] Furthermore, based on the image data set, the emotional curve of the target object is drawn to obtain a target curve graph, including: establishing a coordinate system, and based on the multiple images and the acquisition time of each image, drawing the valence curve and the arousal curve of the target object in the coordinate system to obtain an initial curve graph, wherein the horizontal axis of the coordinate system represents time, and the vertical axis of the coordinate system represents the degree of emotional pleasure of the target object and the degree of physical activation of the target object; preprocessing the initial curve graph to obtain the target curve graph, wherein the preprocessing method includes at least one of the following: denoising processing, filtering processing and smoothing processing.

[0008] Furthermore, the target object is located in a target vehicle, and obtaining an image data set of the target object includes: using an image acquisition device deployed in the target vehicle to acquire a video related to the target object within the target time period to obtain a target video, and extracting multiple frames of images in the target video to obtain the image data set; or, using an image acquisition device deployed in the target vehicle to acquire images of the target object at preset time intervals within the target time period to obtain the image data set, wherein the duration of the preset time interval is less than the duration of the target time period.

[0009] Furthermore, the classification model is obtained in the following manner: acquiring images of the multiple objects, and drawing emotion curves of the multiple objects based on the images of the multiple objects; determining the training samples based on the emotion curves of the multiple objects and the emotion type corresponding to the emotion curve of each object; based on the training samples, taking the emotion type corresponding to the emotion curve of each object as a training label, performing model training on the neural network model to obtain the classification model.

[0010] Furthermore, after determining the emotion type corresponding to the target object based on the output result, it also includes: obtaining a preset mapping relationship, wherein the preset mapping relationship includes: a mapping relationship between different emotion types and control instructions of the target vehicle, triggering a target instruction according to the emotion type corresponding to the target object and the preset mapping relationship, wherein the target instruction includes: a control instruction that has a mapping relationship with the emotion type corresponding to the target object; executing the target instruction to control the target vehicle to perform a target operation to adjust the emotion of the target object.

[0011] Furthermore, the target instruction includes one of the following: an inquiry instruction, a dialing instruction and a control instruction, the inquiry instruction is used to request a voice to inquire about the physical condition of the target object, the dialing instruction is used to request a target phone, and the target phone is a phone of an object other than the target object. The control instruction is used to request to play music or to request to have a conversation with the target object. According to the emotion type corresponding to the target object and the preset mapping relationship, the target instruction is triggered, including: when the emotion type is a first type of emotion, triggering the inquiry instruction or the dialing instruction, wherein the first type of emotion represents an emotion type with a sadness degree greater than a preset sadness degree; when the emotion type is a second type of emotion, triggering the control instruction, wherein the second type of emotion represents an emotion type other than the first type of emotion.

[0012] According to another aspect of an embodiment of the present invention, a data information processing device is also provided, including: an acquisition unit, used to acquire an image data set of a target object, wherein the target object is located in a target vehicle, and the image data set includes: multiple images of the target object within a target time period; a drawing unit, used to draw an emotion curve of the target object based on the image data set to obtain a target curve graph, wherein the emotion curve of the target object at least includes: a valence curve of the target object and an arousal curve of the target object, the valence curve of the target object is used to record the degree of pleasure of the target object's emotions, and the arousal curve is used to record the degree to which the target object's body is activated; a processing unit, used to input the target curve graph into a classification model to obtain an output result, and determine the emotion type corresponding to the target object based on the output result, wherein the classification model includes: a model obtained after training a neural network model using the emotion curves of multiple objects and the emotion type of the emotion curve of each object as training samples.

[0013] Furthermore, the processing unit, the extraction subunit, is used to input the target curve graph into the classification model, and use the classification model to extract features of the emotion curve in the target curve graph to obtain curve features, wherein the curve features include at least one of the following: the curve amplitude of the target curve graph, the peak value of the target curve graph, the valley value of the target curve graph, the correlation between the valence curve of the target object and the arousal curve of the target object; the identification subunit is used to identify the emotion type corresponding to the target object based on the curve features and use the classification model to obtain the output result.

[0014] Furthermore, the drawing unit includes: a drawing subunit, used to establish a coordinate system, and based on the multiple images and the acquisition time of each image, draw the valence curve and the arousal curve of the target object in the coordinate system to obtain an initial curve graph, wherein the horizontal axis of the coordinate system represents time, and the vertical axis of the coordinate system represents the degree of emotional pleasure of the target object and the degree of physical activation of the target object; a first processing subunit, used to preprocess the initial curve graph to obtain the target curve graph, wherein the preprocessing processing method includes at least one of the following: denoising processing, filtering processing and smoothing processing.

[0015] Furthermore, the target object is located in a target vehicle, and the acquisition unit includes: a second processing subunit, used to use the image acquisition device deployed in the target vehicle to acquire a video related to the target object within the target time period to obtain the target video, and extract multiple frames of images in the target video to obtain the image data set; or, an acquisition subunit, used to use the image acquisition device deployed in the target vehicle to acquire images of the target object at preset time intervals within the target time period to obtain the image data set, wherein the duration of the preset time interval is less than the duration of the target time period.

[0016] Furthermore, the classification model is obtained through the following sub-units: an acquisition sub-unit, used to acquire images of the multiple objects and draw emotion curves of the multiple objects based on the images of the multiple objects; a determination sub-unit, used to determine the training samples based on the emotion curves of the multiple objects and the emotion type corresponding to the emotion curve of each object; a training sub-unit, used to perform model training on the neural network model based on the training samples and with the emotion type corresponding to the emotion curve of each object as a training label to obtain the classification model.

[0017] Furthermore, the data information processing device also includes: a relationship acquisition unit, which is used to acquire a preset mapping relationship after determining the emotion type corresponding to the target object based on the output result, wherein the preset mapping relationship includes: a mapping relationship between different emotion types and control instructions of the target vehicle; a determination unit, which is used to trigger a target instruction according to the emotion type corresponding to the target object and the preset mapping relationship, wherein the target instruction includes: a control instruction that has a mapping relationship with the emotion type corresponding to the target object; and an execution unit, which is used to execute the target instruction to control the target vehicle to perform a target operation to adjust the emotion of the target object.

[0018] Furthermore, the target instruction includes one of the following: an inquiry instruction, a dialing instruction and a control instruction, the inquiry instruction is used to request a voice to inquire about the physical condition of the target object, the dialing instruction is used to request to dial a target phone, and the target phone is a phone of another object other than the target object. The control instruction is used to request to play music or to request to have a conversation with the target object. The trigger unit includes: a first trigger subunit, used to trigger the inquiry instruction or the dialing instruction when the emotion type is a first type of emotion, wherein the first type of emotion represents an emotion type with a sadness degree greater than a preset sadness degree; a second trigger subunit, used to trigger the control instruction when the emotion type is a second type of emotion, wherein the second type of emotion represents an emotion type other than the first type of emotion.

[0019] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned data information processing methods by executing the executable instructions.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned data information processing methods.

[0021] In the present invention, an image data set of a target object is obtained, wherein the target object is located in a target vehicle, and the image data set includes: multiple images of the target object in a target time period; based on the image data set, an emotion curve of the target object is drawn to obtain a target curve graph, wherein the emotion curve of the target object at least includes: a valence curve of the target object and an arousal curve of the target object, wherein the valence curve of the target object is used to record the degree of pleasure of the target object's emotion, and the arousal curve is used to record the degree of activation of the target object's body; the target curve graph is input into a classification model to obtain an output result, and the emotion type corresponding to the target object is determined based on the output result, wherein the classification model includes: a model obtained after training a neural network model using the emotion curves of multiple objects and the emotion type of the emotion curve of each object as training samples. Thus, the technical problem of inaccurate recognition results of the driver's emotions using a discrete recognition method in the related art is solved. In the present invention, an emotion curve is drawn according to the image of the target object, and the emotion type is identified using the trained classification model according to the emotion curve, so as to achieve the purpose of continuous emotional state recognition, thereby achieving the technical effect of improving the accuracy of the emotion recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0023] Figure 1 is a flow chart of an optional method for processing data information according to an embodiment of the present invention;

[0024] Figure 2 is a schematic diagram of an optional two-dimensional corresponding relationship of emotions according to an embodiment of the present invention;

[0025] Figure 3 is a schematic diagram of an optional model training according to an embodiment of the present invention;

[0026] Figure 4 is a flow chart of an optional classification model according to an embodiment of the present invention;

[0027] Figure 5 is a schematic diagram of an optional data information processing device according to an embodiment of the present invention;

[0028] Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.), collected information and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0032] For the convenience of description, some terms or names involved in the present invention are explained below.

[0033] Valence: refers to the degree of pleasure or unpleasantness of an emotion. It is a continuum from very negative to very positive. For example, happiness and satisfaction are considered to have positive valence, while sadness and anger are considered to have negative valence. Valence can help understand how people feel about different stimuli or situations.

[0034] Arousal: refers to the degree to which the body is activated, or the change in the level of mental and physiological activity. It can range from a state of complete relaxation to a state of extreme excitement or tension. High arousal may be accompanied by physiological reactions such as a rapid heartbeat and sweating, while low arousal may mean calmness or sleepiness. Certain emotions such as fear or anger are generally associated with high arousal, while feelings of tranquility or peace are associated with lower arousal levels.

[0035] Embodiment 1

[0036] According to an embodiment of the present invention, a method embodiment of an optional method for processing data information is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] Figure 1 is a flowchart of an optional data information processing method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0038] Step S101 : acquiring an image data set of a target object, wherein the image data set includes: a plurality of images of the target object within a target time period.

[0039] The above-mentioned target object may be an object to be subjected to emotion recognition, and may include but is not limited to a driver or passenger in a vehicle. The above-mentioned image data set may include multiple images obtained by continuously photographing the target object through an image acquisition device within a target time period (for example, multiple images obtained by photographing once every preset time interval within the target time period), or multiple frames of images in a video obtained by photographing the target object within the target time period, wherein the target time period may be a period of time before the moment when emotion recognition is required, for example, a period of time before the moment when the emotion recognition request is initiated, and the multiple frames of images may be spaced apart by a preset time length.

[0040] Step S102, based on the image data set, draw the emotion curve of the target object to obtain a target curve graph, wherein the emotion curve at least includes: the emotion curve of the target object at least includes: the valence curve of the target object and the arousal curve of the target object, the valence curve of the target object is used to record the degree of pleasure of the target object's emotions, and the arousal curve is used to record the degree of activation of the target object's body.

[0041] The above-mentioned emotion curve may be a VA curve, namely (Valence curve and Arousal curve).

[0042] In this embodiment, based on the target image set, the VA model can be used to draw the emotion curve of the target object to obtain a target curve graph, wherein the VA model can be a model for drawing a Valence curve and an Arousal curve.

[0043] In this embodiment, the emotional characteristics of the target object may be continuously recorded through the emotional curve, so as to improve the accuracy of the emotion recognition result.

[0044] Step S103, input the target curve graph into the classification model to obtain an output result, and determine the emotion type corresponding to the target object based on the output result, wherein the classification model includes: using the emotion curves of multiple objects and the emotion type of the emotion curve of each object as training samples, and the model obtained after the neural network model is trained.

[0045] The input of the above-mentioned classification model can be the emotion curve of the object to be emotion recognized, for example, the target curve graph, and the output of the above-mentioned classification model can be the probability that the target object is in a certain emotion type, that is, the above-mentioned output result can include the emotion probability of the target object being in a certain emotion type. When the probability of being in a certain emotion type is higher than a preset threshold, it can be determined that the target object belongs to this emotion type. Through the classification model, emotion recognition is performed according to the target curve graph, and the target curve graph is used to record the changes in the continuous emotional characteristics of the target object, thereby achieving the goal of identifying the emotion type of the target object based on continuous emotions, thereby improving the accuracy of the emotion recognition results.

[0046] Figure 2 is a schematic diagram of an optional two-dimensional correspondence relationship of emotions according to an embodiment of the present invention, such as Figure 2 As shown, a two-dimensional spatial correspondence between arousal and pleasure levels and emotions is demonstrated.

[0047] It should be noted that the emotion types in this embodiment may include but are not limited to the following classification methods:

[0048] (1) Four basic emotions: fear, grief, love, and rage.

[0049] (2) Six basic emotions: anger, disgust, fear, happiness, sadness, and surprise.

[0050] (3) 15 emotions: aesthetic experience, anger, anxiety, compassion, depression, envy, fright, gratitude, guilt, happiness, hope, jealousy, love, pride, relief, sadness, and shame.

[0051] Through the above steps, in this embodiment, an emotion curve is drawn according to the image of the target object, and the emotion type is identified according to the emotion curve using the trained classification model, thereby achieving the purpose of continuous emotion state recognition, thereby achieving the technical effect of improving the accuracy of emotion recognition results. This solves the technical problem of inaccurate recognition results when the driver's emotions are recognized using a discrete recognition method in the related art.

[0052] Optionally, the target curve graph is input into a classification model, and the classification model is used to extract features of the emotion curve in the target curve graph to obtain curve features, wherein the curve features include at least one of the following: the curve amplitude of the target curve graph, the peak value of the target curve graph, the valley value of the target curve graph, the correlation between the valence curve of the target object and the arousal curve of the target object; based on the curve features, the classification model is used to identify the emotion type corresponding to the target object to obtain an output result.

[0053] The target curves mentioned above refer to curves plotted based on the changes in Valence and Arousal values ​​over time. The following features can be extracted from these curves:

[0054] Curve amplitude: Changes in the amplitude of the curve can reflect fluctuations in the intensity of emotions. For example, large fluctuations in the emotion curve may correspond to drastic changes in emotional state, while smaller amplitude changes may indicate relatively stable emotions.

[0055] Peaks and valleys: Peak and valley detection can help identify the peaks and valleys of emotional states, which correspond to the strongest expressions and calmest states of emotions. For example, the peak of the Valence curve may correspond to extreme happiness, while the valley may correspond to extreme sadness.

[0056] Correlation between the valence curve and the arousal curve: By calculating the correlation between the valence curve and the arousal curve (such as the Pearson correlation coefficient), we can understand the relationship between these two dimensions in emotion recognition, so as to facilitate the recognition of complex emotions, because different types of emotions may show different correlation patterns on the two dimensions. For example, some emotions (e.g., excitement) may show high values ​​on both dimensions, while other emotions (e.g., calmness) may show high values ​​on valence but low values ​​on arousal.

[0057] By extracting the above-mentioned multiple features, it is possible to facilitate the classification model to perform emotion recognition, and by performing emotion recognition based on multiple features of the target curve graph, the accuracy of the emotion recognition result can be improved.

[0058] After feature extraction is completed, a classification model can be used to identify the type of emotion. The model type of the classification model can be a deep learning model (such as convolutional neural network CNN, recurrent neural network RNN, long short-term memory network LSTM, etc.), or it can be a traditional machine learning model (for example, support vector machine SVM, decision tree, random forest, etc.). Since the classification model has been trained and learned how to map the above-extracted features to specific emotion categories. Therefore, the classification model can output one or more possible emotion types and their confidence (that is, the probability of a certain emotion type) based on the characteristics of the emotion curve. Since the classification model can establish a mapping from continuous emotional state features to discrete emotion categories, it avoids the situation in which the emotion classification model in the related technology only recognizes emotions based on a face image, and the recognition results are inaccurate only through this discrete emotion recognition. Therefore, the classification model can accurately identify the emotion type.

[0059] Optionally, based on the image data set, an emotional curve of the target object is drawn to obtain a target curve graph, including: establishing a coordinate system, and based on multiple images and the acquisition time of each image, drawing the valence curve of the target object and the arousal curve of the target object in the coordinate system to obtain an initial curve graph, wherein the horizontal axis of the coordinate system represents time, and the vertical axis of the coordinate system represents the degree of emotional pleasure of the target object and the degree of physical activation of the target object; preprocessing the initial curve graph to obtain a target curve graph, wherein the preprocessing processing method includes at least one of the following: denoising processing, filtering processing and smoothing processing.

[0060] In this embodiment, deep learning or computer vision technology can be used to extract emotional features related to Valence and Arousal (e.g., arousal, pleasure) from each image in the image data set using a VA model. An initial emotional curve graph is drawn based on the extracted emotional features and the time points associated therewith. In the initial emotional curve graph, the horizontal axis can represent the time axis, and the vertical axis can be divided into two parts, which are used to represent the values ​​of Valence and Arousal respectively. By drawing the points where Valence and Arousal change over time on the time axis, two curves can be formed, namely, the valence curve and the arousal curve of the target object.

[0061] In order to improve the quality of the curve graph and the accuracy of the classification results of the subsequent classification model, the initial curve graph can also be preprocessed. The preprocessing methods may include: Denoising: removing random noise in the curve graph to improve the clarity and accuracy of the data; Filtering: applying filters, such as low-pass filters, high-pass filters, or band-pass filters, to highlight or suppress fluctuations of specific frequencies and further purify the data; Smoothing: through smoothing techniques such as moving average or Gaussian smoothing, the curve is made smoother, the sharp peaks and valleys are reduced, and the curve is more continuous and easier to analyze. Preprocessing the initial curve graph can help reduce outliers and irregular fluctuations in the data, make the curve clearer and more stable, and thus provide higher quality data for the classification model.

[0062] Optionally, the target object is located in a target vehicle, and obtaining an image data set of the target object includes: using an image acquisition device deployed in the target vehicle to acquire a video related to the target object within a target time period to obtain a target video, and extracting multiple frames of images in the target video to obtain an image data set; or, using an image acquisition device deployed in the target vehicle to acquire images of the target object at preset time intervals within the target time period to obtain an image data set, wherein the duration of the preset time interval is less than the duration of the target time period.

[0063] In this embodiment, the target video can be captured by an image acquisition device. For example, a camera or other image acquisition device installed in a vehicle can be used to continuously record the behavior and facial expressions of the target object within a target time period to obtain a target video. The target time period can be any predefined period, for example, the entire process of driving, or a specific activity period. From the target video, a series of image frames in the video can be extracted to form an image data set. Each frame of image can be an image of the target object at a specific time point.

[0064] In an optional example, in this embodiment, images of the target object can also be captured at preset time intervals. For example, within the target time period, the image acquisition device can automatically capture images of the target object at preset time intervals (e.g., every second, every 5 seconds, etc.). The selection of the preset time interval can depend on the required level of refinement of the emotional state change and the efficiency of data processing. Generally, the shorter the time interval is set, the more detailed the emotional changes that can be captured. By capturing images at time intervals, an image data set can be directly constructed, in which each image is associated with a specific time point. The image data set can be used for subsequent emotion analysis and curve drawing to identify the emotion type through a classification model, thereby achieving a technical effect of improving the accuracy of the emotion recognition results.

[0065] Optionally, the classification model is obtained in the following manner: acquiring images of multiple objects, and based on the images of the multiple objects, drawing emotion curves of the multiple objects; determining training samples based on the emotion curves of the multiple objects and the emotion type corresponding to the emotion curve of each object; based on the training samples, using the emotion type corresponding to the emotion curve of each object as a training label, performing model training on a neural network model to obtain a classification model.

[0066] In this embodiment, images of multiple different objects (such as drivers, passengers, etc.) in different situations and different emotional states can be collected, and these images can come from video clips, continuous snapshots or real-time monitoring. The information that can be contained in the image, such as facial expressions, body movements, etc., will be used for subsequent emotional analysis. Then, based on the collected images, a pre-trained deep learning model or algorithm (for example, a VA model) can be used to extract emotional features related to Valence and Arousal to form a numerical sequence that changes over time, corresponding to the two dimensions of Valence and Arousal. Then, based on these numerical sequences, emotional curves for multiple objects are obtained, where the horizontal axis represents time and the vertical axis represents the values ​​of Valence and Arousal. The emotional curve of each object will reflect the continuous change of its emotional state over time. In order to improve the quality of training samples and improve the accuracy of model training, the emotional curves in the training samples can also be denoised, filtered, and smoothed to obtain a relatively flat waveform, and then a data set that can be used as a training picture is obtained through image processing.

[0067] In this embodiment, training samples may be constructed based on the emotion curves of multiple objects and the known emotion type corresponding to the emotion curve of each object (discrete emotion labels, such as “happy”, “angry”, and “sad”, etc.).

[0068] In this embodiment, the emotional curve of the target object can be annotated into trainable data from the two dimensions of Valence and Arousal, and then the VA value is output through reasoning, and then drawn into a VA curve in the form of a time axis to indicate the continuous changes in the emotions of the observed person. The user needs to pay attention to the changing curve at all times, and there are many inconveniences in product application.

[0069] Specifically, each training sample may include an emotion curve image at one or more time points, and a corresponding discrete emotion type label. For example, an emotion curve image showing high Valence and high Arousal may be labeled as "excited", while a curve with low Valence and low Arousal may be labeled as "calm". For example, the Valence (curve waveform of valence-pleasure degree) and Arousal (curve waveform of arousal-excitement degree) of the calm class curve graph may both be in a smaller positive numerical range. In the anger class curve graph, Valence (curve waveform of valence-pleasure degree) may be in a smaller negative numerical range, and Arousal (curve waveform of arousal-excitement degree) may be in a smaller positive numerical range. In the surprise class curve graph, Valence (curve waveform of valence-pleasure degree) may be in a larger negative numerical range, and Arousal (curve waveform of arousal-excitement degree) may be in a larger positive numerical range. In the joy curve graph, Valence (the waveform of the valence-pleasure degree curve) can be in a large positive numerical range; Arousal (the waveform of the arousal-excitement degree curve) can be in a large positive numerical range. It should be noted that the division between the smaller range and the larger range can be based on whether it exceeds a preset threshold. For example, the values ​​in the smaller numerical range can all be less than the preset threshold, and the values ​​in the larger numerical range can all be greater than or equal to the preset threshold.

[0070] In this embodiment, the emotion curve in the training sample can be used to train the neural network model with the emotion type corresponding to the emotion curve as the training label. The neural network model can involve a deep learning framework, wherein the neural network model can be a CNN (convolutional neural network) for identifying local features in an image, or an LSTM (long short-term memory network) for processing time dependencies in sequence data. The purpose of model training is to enable the model to learn the ability to predict discrete emotion types from the emotion curve image. By repeatedly adjusting the model parameters to minimize the prediction error, the training process will enable the model to gradually accurately map the emotion curve to a specific emotion category. The classification model can also be continuously optimized and updated over time. By collecting new data and feedback, the training model can be continuously optimized and updated to adapt to different situations and individual differences, thereby improving the accuracy and generalization ability of the model.

[0071] It should be noted that once the model training is completed, the obtained classification model can be applied to a variety of scenarios, including but not limited to: Optimization of the cockpit experience in smart cars: The model can analyze the driver's emotional state in real time and adjust the in-car environment (such as music, temperature, lighting) according to the identified emotion type to improve driving safety and comfort.

[0072] By building a classification model that can identify discrete emotion types from continuous emotion curve images, it provides new tools for understanding and responding to individual emotional states, which helps to realize more humane and intelligent products and services.

[0073] Optionally, after determining the emotion type corresponding to the target object based on the output result, it also includes: obtaining a preset mapping relationship, wherein the preset mapping relationship includes: a mapping relationship between different emotion types and control instructions of the target vehicle; triggering a target instruction according to the emotion type corresponding to the target object and the preset mapping relationship, wherein the target instruction includes: a control instruction that has a mapping relationship with the emotion type corresponding to the target object; executing the target instruction to control the target vehicle to perform a target operation to adjust the emotion of the target object.

[0074] The above-mentioned preset mapping relationship may include but is not limited to: the mapping relationship between different emotion types and the control instructions of the target vehicle, for example, the mapping relationship between the emotion of joy and the control instruction of controlling the target vehicle to play cheering songs; the mapping relationship between the emotion of surprise and the control instruction of controlling the target vehicle to resolve its doubts through the large voice model; the mapping relationship between the emotion of anger and the control instruction of controlling the voice assistant of the target vehicle to actively initiate a chat to soothe the emotion; the emotion of sadness and pain and the control instruction of controlling the target vehicle to ask about the physical condition or actively call family members or emergency numbers.

[0075] In order to provide a more personalized interactive experience in the vehicle, in this embodiment, a more personalized and emotion-sensitive interactive experience can be provided according to the emotional type of the target object. For example, based on a preset mapping relationship, a control instruction that has a mapping relationship with the emotional type of the target object is determined, and the target instruction is triggered to control the target vehicle. For example, the target instruction can be executed to control the vehicle to perform a target operation to adjust the target object's emotions. For example, the target instruction can be used to control the target vehicle to perform target operations such as playing music, calling emergency contacts, and answering questions, so as to adjust the target object's emotions and improve the safety of vehicle driving.

[0076] Optionally, the target instruction includes one of the following: an inquiry instruction, a dialing instruction and a control instruction, the inquiry instruction is used to request a voice to inquire about the physical condition of the target object, the dialing instruction is used to request to dial a target phone, and the target phone is a phone of an object other than the target object. The control instruction is used to request to play music or to request to have a conversation with the target object. According to the emotion type corresponding to the target object and the preset mapping relationship, the target instruction is triggered, including: when the emotion type is a first type of emotion, triggering an inquiry instruction or a dialing instruction, wherein the first type of emotion represents an emotion type with a degree of sadness greater than a preset degree of sadness; when the emotion type is a second type of emotion, triggering a control instruction, wherein the second type of emotion represents an emotion type other than the first type of emotion.

[0077] In this embodiment, the above-mentioned inquiry instruction is used to request a voice to inquire about the physical condition of the target object, the above-mentioned dialing instruction is used to request to dial the target phone, the target phone is the phone of another object other than the target object (such as the phone of the emergency contact of the target object), and the control instruction is used to request to play music or to request to have a conversation with the target object, to answer questions for the target object, etc. Specifically, when the emotion type is the first type of emotion type, for example, when the joy of the driver in the cabin is recognized, it can be recommended to play cheering songs; when he is surprised, his doubts can be analyzed through the large voice model; when he is angry, you can actively initiate a chat to soothe his emotions; when the emotion type is the first type of emotion type, for example, when he is sad and painful, you can ask about his physical condition or actively call his family or emergency number, etc.

[0078] Provide personalized services based on the target object's emotional type, for example, provide comforting conversations when the user is depressed and provide more positive responses when the user is excited. Monitoring and analysis of mental health: By continuously monitoring the emotional state of an individual, potential mental health problems can be identified, such as long-term depression or anxiety, providing data support for mental health intervention. Customized recommendations for entertainment products can recommend corresponding entertainment content (such as movies, music) based on the identified emotion type to meet the user's immediate emotional needs, achieving the technical effect of improving the intelligence of the vehicle.

[0079] Figure 3 is a schematic diagram of an optional model training according to an embodiment of the present invention, such as Figure 3 As shown, in this embodiment, a training data set of a VA emotion curve can be used, and after preprocessing and feature extraction of the training data set, an excellent classifier (corresponding to a classification model) can be trained to establish a mapping relationship between continuous emotions and discrete emotions. The obtained discrete emotion label threshold is used as the basis for emotion judgment of car cockpit and other emotion recognition, which can eventually be applied to persuasive products.

[0080] Figure 4 is a flow chart of an optional classification model according to an embodiment of the present invention, such as Figure 4 As shown, by constructing a classification model that can perform emotion recognition based on the VA emotion curve and establishing a mapping relationship between continuous emotions and discrete emotions, it is finally possible to generate an emotion curve based on the original image using the emotion analysis VA model, and use the classification model to determine the corresponding emotion type according to the mapping table, thereby achieving the technical effect of efficient and accurate emotion recognition.

[0081] Embodiment 2

[0082] Embodiment 2 of the present invention provides an optional data information processing device, and each implementation unit in the processing device corresponds to each implementation step in Embodiment 1.

[0083] Figure 5 is a schematic diagram of an optional data information processing device according to an embodiment of the present invention, such as Figure 5 As shown, the processing device includes: an acquisition unit 51, a drawing unit 52 and a processing unit 53.

[0084] The acquisition unit 51 is used to acquire an image data set of a target object, wherein the target object is located in a target vehicle, and the image data set includes: a plurality of images of the target object within a target time period;

[0085] A drawing unit 52 is used to draw an emotion curve of the target object based on the image data set to obtain a target curve graph, wherein the emotion curve of the target object at least includes: a valence curve of the target object and an arousal curve of the target object, the valence curve of the target object is used to record the degree of pleasure of the target object's emotion, and the arousal curve is used to record the degree of physical activation of the target object;

[0086] The processing unit 53 is used to input the target curve graph into the classification model to obtain an output result, and determine the emotion type corresponding to the target object based on the output result, wherein the classification model includes: using the emotion curves of multiple objects and the emotion type of the emotion curve of each object as training samples, and the model obtained after the neural network model is trained.

[0087] In the data information processing device provided in the second embodiment of the present invention, an image data set of a target object can be obtained by an acquisition unit 51, wherein the target object is located in a target vehicle, and the image data set includes: multiple images of the target object in a target time period, and an emotional curve of the target object is drawn by a drawing unit 52 based on the image data set to obtain a target curve graph, wherein the emotional curve of the target object at least includes: a valence curve of the target object and an arousal curve of the target object, wherein the valence curve of the target object is used to record the degree of pleasure of the target object's emotions, and the arousal curve is used to record the degree of activation of the target object's body, and the target curve graph is input into a classification model by a processing unit 53 to obtain an output result, and the emotional type corresponding to the target object is determined based on the output result, wherein the classification model includes: a model obtained after training a neural network model using the emotional curves of multiple objects and the emotional type of the emotional curve of each object as training samples. This solves the technical problem of inaccurate recognition results of using a discrete form recognition method to recognize the driver's emotions in the related art. In this embodiment, an emotion curve is drawn according to the image of the target object, and based on the emotion curve, the emotion type is identified using a trained classification model, thereby achieving the purpose of continuous emotion state recognition, thereby achieving the technical effect of improving the accuracy of emotion recognition results.

[0088] Optionally, in the data information processing device provided in Embodiment 2 of the present invention, the processing unit, the extraction subunit, is used to input the target curve graph into the classification model, and use the classification model to perform feature extraction on the emotion curve in the target curve graph to obtain curve features, wherein the curve features include at least one of the following: the curve amplitude of the target curve graph, the peak value of the target curve graph, the valley value of the target curve graph, the correlation between the valence curve of the target object and the arousal curve of the target object; the identification subunit is used to identify the emotion type corresponding to the target object based on the curve features and use the classification model to obtain an output result.

[0089] Optionally, in the data information processing device provided in the second embodiment of the present invention, the drawing unit 52 includes: a drawing subunit, used to establish a coordinate system, and based on multiple images and the acquisition time of each image, draw the valence curve and the arousal curve of the target object in the coordinate system to obtain an initial curve graph, wherein the horizontal axis of the coordinate system represents time, and the vertical axis of the coordinate system represents the degree of emotional pleasure of the target object and the degree of physical activation of the target object; a first processing subunit, used to pre-process the initial curve graph to obtain a target curve graph, wherein the pre-processing processing method includes at least one of the following: denoising processing, filtering processing, and smoothing processing.

[0090] Optionally, in the data information processing device provided in the second embodiment of the present invention, the target object is located in the target vehicle, and the acquisition unit 51 includes: a second processing subunit, used to use the image acquisition device deployed in the target vehicle to acquire a video related to the target object within a target time period to obtain a target video, and extract multiple frames of images in the target video to obtain an image data set; or, an acquisition subunit, used to use the image acquisition device deployed in the target vehicle to acquire images of the target object at preset time intervals within the target time period to obtain an image data set, wherein the duration of the preset time interval is less than the duration of the target time period.

[0091] Optionally, in the data information processing device provided in Example 2 of the present invention, the classification model is obtained through the following sub-units: an acquisition sub-unit, used to acquire images of multiple objects, and based on the images of the multiple objects, draw emotion curves of the multiple objects; a determination sub-unit, used to determine training samples based on the emotion curves of the multiple objects and the emotion type corresponding to the emotion curve of each object; a training sub-unit, used to perform model training on the neural network model based on the training samples and with the emotion type corresponding to the emotion curve of each object as a training label to obtain a classification model.

[0092] Optionally, in the data information processing device provided in Embodiment 2 of the present invention, the data information processing device further includes: a relationship acquisition unit, used to acquire a preset mapping relationship after determining the emotion type corresponding to the target object based on the output result, wherein the preset mapping relationship includes: a mapping relationship between different emotion types and control instructions of the target vehicle; a determination unit, used to trigger the target instruction according to the emotion type corresponding to the target object and the preset mapping relationship, wherein the target instruction includes: a control instruction that has a mapping relationship with the emotion type corresponding to the target object; and an execution unit, used to execute the target instruction to control the target vehicle to perform a target operation to adjust the emotion of the target object.

[0093] Optionally, in the data information processing device provided in Example 2 of the present invention, the target instruction includes one of the following: an inquiry instruction, a dialing instruction, and a control instruction, the inquiry instruction is used to request a voice to inquire about the physical condition of the target object, the dialing instruction is used to request to dial a target phone, and the target phone is a phone of an object other than the target object, and the control instruction is used to request to play music or to request to have a conversation with the target object, and the trigger unit includes: a first trigger sub-unit, used to trigger the inquiry instruction or the dialing instruction when the emotion type is a first type of emotion, wherein the first type of emotion represents an emotion type with a degree of sadness greater than a preset degree of sadness; a second trigger sub-unit, used to trigger the control instruction when the emotion type is a second type of emotion, wherein the second type of emotion represents an emotion type other than the first type of emotion.

[0094] The above-mentioned data information processing device may also include a processor and a memory. The above-mentioned acquisition unit 51, drawing unit 52 and processing unit 53 are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0095] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the kernel parameters are adjusted to draw an emotion curve according to the image of the target object, and the emotion type is identified according to the emotion curve using the trained classification model, so as to achieve the purpose of continuous emotion state recognition, thereby achieving the technical effect of improving the accuracy of emotion recognition results.

[0096] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one storage chip.

[0097] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned data information processing methods by executing the executable instructions.

[0098] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned data information processing methods.

[0099] Figure 6 is a schematic diagram of an electronic device according to an embodiment of the present invention, such as Figure 6 As shown, an embodiment of the present invention provides an electronic device 60, which includes a processor, a memory, and a program stored in the memory and executable on the processor, and implements any one of the above-mentioned data information processing methods when the processor executes the program.

[0100] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0101] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0104] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0106] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for processing data information, characterized in that: include: Acquire an image data set of a target object, wherein the image data set includes: a plurality of images of the target object within a target time period; Based on the image data set, an emotion curve of the target object is drawn to obtain a target curve graph, wherein the emotion curve of the target object at least includes: a valence curve of the target object and an arousal curve of the target object, the valence curve of the target object is used to record the degree of pleasure of the target object's emotion, and the arousal curve is used to record the degree to which the target object's body is activated; The target curve graph is input into a classification model to obtain an output result, and the emotion type corresponding to the target object is determined based on the output result, wherein the classification model includes: a model obtained after training a neural network model using the emotion curves of multiple objects and the emotion type of the emotion curve of each object as training samples.

2. The processing method according to claim 1, characterized in that: The target curve graph is input into the classification model to obtain output results, including: Inputting the target curve graph into a classification model, and using the classification model to extract features of the emotion curve in the target curve graph to obtain curve features, wherein the curve features include at least one of the following: a curve amplitude of the target curve graph, a peak value of the target curve graph, a valley value of the target curve graph, and a correlation between a valence curve of the target object and an arousal curve of the target object; Based on the curve features, the classification model is used to identify the emotion type corresponding to the target object to obtain the output result.

3. The processing method according to claim 1, characterized in that: Based on the image data set, drawing the emotion curve of the target object to obtain a target curve graph includes: Establishing a coordinate system, and drawing the valence curve and the arousal curve of the target object in the coordinate system based on the multiple images and the acquisition time of each image, to obtain an initial curve graph, wherein the horizontal axis of the coordinate system represents time, and the vertical axis of the coordinate system represents the degree of emotional pleasure of the target object and the degree of physical activation of the target object; The initial curve graph is preprocessed to obtain the target curve graph, wherein the preprocessing method includes at least one of the following: denoising processing, filtering processing and smoothing processing.

4. The processing method according to claim 1, characterized in that: The target object is located in a target vehicle, and an image data set of the target object is obtained, including: Using an image acquisition device deployed in the target vehicle, within the target time period, a video related to the target object is acquired to obtain a target video, and multiple frames of images in the target video are extracted to obtain the image data set; or, By using the image acquisition device deployed in the target vehicle, images of the target object are acquired at preset time intervals within the target time period to obtain the image data set, wherein the duration of the preset time interval is less than the duration of the target time period.

5. The processing method according to claim 1, characterized in that: The classification model is obtained in the following way: Acquire images of the plurality of objects, and draw emotion curves of the plurality of objects based on the images of the plurality of objects; Determining the training sample based on the emotion curves of the multiple objects and the emotion type corresponding to the emotion curve of each object; Based on the training samples, the neural network model is trained by taking the emotion type corresponding to the emotion curve of each object as a training label to obtain the classification model.

6. The processing method according to claim 1, characterized in that: After determining the emotion type corresponding to the target object based on the output result, the method further includes: Acquire a preset mapping relationship, wherein the preset mapping relationship includes: a mapping relationship between different emotion types and control instructions of a target vehicle; According to the emotion type corresponding to the target object and the preset mapping relationship, triggering a target instruction, wherein the target instruction includes: a control instruction having a mapping relationship with the emotion type corresponding to the target object; The target instruction is executed to control the target vehicle to perform a target operation to adjust the emotion of the target object.

7. The processing method according to claim 6, characterized in that: The target instruction includes one of the following: an inquiry instruction, a dialing instruction, and a control instruction, wherein the inquiry instruction is used to request a voice to inquire about the physical condition of the target object, the dialing instruction is used to request to dial a target phone, and the target phone is a phone of another object other than the target object, and the control instruction is used to request to play music or to request to have a conversation with the target object. According to the emotion type corresponding to the target object and the preset mapping relationship, the target instruction is triggered, including: In the case where the emotion type is a first type of emotion, triggering the inquiry instruction or the dialing instruction, wherein the first type of emotion represents an emotion type with a sadness degree greater than a preset sadness degree; In the case where the emotion type is a second type of emotion, the control instruction is triggered, wherein the second type of emotion represents an emotion type other than the first type of emotion.

8. A data information processing device, characterized in that: include: An acquisition unit, configured to acquire an image data set of a target object, wherein the target object is located in a target vehicle, and the image data set includes: a plurality of images of the target object within a target time period; a drawing unit, configured to draw an emotion curve of the target object based on the image data set to obtain a target curve graph, wherein the emotion curve of the target object at least includes: a valence curve of the target object and an arousal curve of the target object, wherein the valence curve of the target object is used to record the degree of pleasure of the target object's emotion, and the arousal curve is used to record the degree to which the target object's body is activated; A processing unit is used to input the target curve graph into a classification model to obtain an output result, and determine the emotion type corresponding to the target object based on the output result, wherein the classification model includes: a model obtained after training a neural network model using the emotion curves of multiple objects and the emotion type of the emotion curve of each object as training samples.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for processing data information according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data information processing method described in any one of claims 1 to 7.

Images