Dynamic video emotion and attention tracking analysis method, device, equipment and medium

Abstract

The present invention relates to a dynamic video emotion and attention tracking analysis method, device, equipment and medium, which are applied to the field of video analysis technology, and the method includes: obtaining a dynamic video to be analyzed; collecting video viewing data of a target tester on the dynamic video to be analyzed, and generating a test viewing video; determining the eye movement data and facial expression data of the target tester based on the test viewing video; generating a multimodal data matching result based on the eye movement data, the facial expression data and the dynamic video to be analyzed; and generating a tracking analysis result based on the multimodal data matching result. The present application has the effect of objectively and accurately analyzing dynamic videos.

Description

Technical Field

[0001] The present application relates to the technical field of video analysis, and in particular to a method, device, equipment and medium for tracking and analyzing dynamic video emotion and attention. Background Art

[0002] Dynamic video refers to video content that contains moving images, usually with a synchronized audio track, such as movies, TV shows, live sports events, animations, short videos, etc. With the development of digital media and Internet technology, dynamic video has become an important part of current visual materials. Its importance continues to grow with the development of technology and changes in user habits. It is very meaningful to predict the audience's emotions and visual attention when watching videos.

[0003] For film and television companies, if they can know in advance which parts of the film screen are likely to interest the audience and trigger higher positive emotions before the work is released, they can extract and integrate this information for use in pre-promotion such as film posters and trailers, bringing considerable economic benefits to the company; for short video creators, if they can know which parts of the video screen are of interest to netizens before the work is released, they can evaluate the popularity of the video among netizens based on the distribution of gaze points and emotional fluctuations while watching, thereby improving the quality of video creation; for video advertising companies, through the analysis of gaze points and emotional focus points in the video, they can set up advertising products more reasonably in the video and obtain better publicity effects.

[0004] At present, the main way to determine the focus and emotional changes is for film and television companies and advertising companies to gather a small number of viewers to hold advance viewings before the video is released, and ask participants to record events, actions, characters, scenery, plots, and emotional changes during viewing by filling out questionnaires and face-to-face Q&A. Film and television companies and advertising companies then analyze the audience's points of interest based on this information. However, this method has some limitations. The main reason is that the collected data cannot be accurate to the pixel level, the obtained perceptual points of interest are inaccurate, and it is impossible to accurately obtain the audience's objective and intuitive feelings during the viewing process. There is no evaluation of dynamic video effects based on the combination of emotional changes and attention changes based on time series. Summary of the invention

[0005] In order to objectively and accurately analyze dynamic videos, the present application provides a dynamic video emotion and attention tracking analysis method, device, equipment and medium.

[0006] In the first aspect, the present application provides a dynamic video emotion and attention tracking analysis method, which adopts the following technical solutions:

[0007] A dynamic video emotion and attention tracking analysis method, comprising:

[0008] Get the dynamic video to be analyzed;

[0009] Preprocessing the dynamic video to be analyzed to generate a target dynamic video;

[0010] Collecting the video viewing data of the target tester on the dynamic video to be analyzed, and generating a test viewing video;

[0011] Determining eye movement data and facial expression data of the target tester based on the test viewing video;

[0012] Generate a multimodal data matching result based on the eye movement data, the facial expression data and the dynamic video to be analyzed;

[0013] A tracking analysis result is generated based on the multimodal data matching result.

[0014] By adopting the above technical solutions and combining eye movement data and facial expression data, we can more comprehensively understand the target tester's attention distribution, emotional response and possible cognitive process when watching the video, and better reveal the tester's real reaction and experience. With the advancement of technology, eye tracking and facial expression recognition technology has become more mature and automated, and a large amount of data can be collected and analyzed in a shorter period of time, improving the efficiency and accuracy of research, so as to objectively and accurately analyze dynamic videos.

[0015] Optionally, before generating a multimodal data matching result based on the eye movement data, the facial expression data and the dynamic video to be analyzed, the method further includes:

[0016] Acquire the target object and the target object information in the dynamic video to be analyzed;

[0017] Determine an object marking frame of the target object based on the target object information;

[0018] Acquire key events and scene change information in the dynamic video to be analyzed;

[0019] generating a key event marker based on the key event;

[0020] generating a scene change point based on the scene change information;

[0021] The object marking frame, the key event marking and the scene change point are added to the dynamic video to be analyzed to generate a target dynamic video.

[0022] Optionally, determining the eye movement data and facial expression data of the target tester based on the test viewing video includes:

[0023] Obtaining the video frame rate of the dynamic video to be analyzed and the viewing frame rate of the test viewing video;

[0024] Determining whether the viewing frame rate is consistent with the video frame rate;

[0025] If the viewing frame rate is consistent with the video frame rate, extracting the video viewing data of the target tester frame by frame based on the viewing frame rate to generate eye movement data and facial expression data;

[0026] If the viewing frame rate is inconsistent with the video frame rate, adjusting the viewing frame rate based on the video frame rate to generate an adjusted viewing frame rate;

[0027] The video viewing data of the target tester is extracted frame by frame based on the adjusted viewing frame rate to generate eye movement data and facial expression data.

[0028] Optionally, extracting the video viewing data of the target tester frame by frame based on the viewing frame rate to generate eye movement data and facial expression data includes:

[0029] Recognize the facial emotion of each frame of the target tester and determine the emotion type and emotion value of each frame;

[0030] Generate facial expression data based on the emotion type and the emotion value of all frames;

[0031] Determine eye feature points of the target tester based on the video viewing data;

[0032] Determining the stay position of the eye feature point of each frame to determine the sight line coordinates of each frame;

[0033] Eye movement data is generated based on the gaze coordinates of all frames.

[0034] Optionally, generating a multimodal data matching result based on the eye movement data, the facial expression data and the dynamic video to be analyzed includes:

[0035] Matching the eye movement data with the target dynamic video and the dynamic video to be analyzed based on the video frame rate to generate a first matching result;

[0036] Matching the facial expression data with the target dynamic video and the dynamic video to be analyzed based on the video frame rate to generate a second matching result;

[0037] The target dynamic video is bound to the first matching result and the second matching result to generate a multimodal data matching result.

[0038] Optionally, generating a tracking analysis result based on the multimodal data matching result includes:

[0039] Determine key gaze information of the target tester based on the first matching result and the eye movement data;

[0040] Determining the attention-response relationship of the target tester based on the key gaze information and the second matching result;

[0041] Determine key emotional information of the target tester based on the second matching result and the facial expression data;

[0042] A tracking analysis result is generated based on the key gaze information, the attention-response relationship and the key emotion information.

[0043] Optionally, after generating the tracking analysis result based on the multimodal data matching result, the method further includes:

[0044] Formulate a video optimization strategy based on the tracking and analysis results;

[0045] The video is optimized based on the video optimization strategy.

[0046] In the second aspect, the present application provides a dynamic video emotion and attention tracking and analysis device, which adopts the following technical solution:

[0047] A dynamic video emotion and attention tracking and analysis device, comprising:

[0048] A dynamic video acquisition module is used to acquire the dynamic video to be analyzed;

[0049] A test video generation module, used to collect the video viewing data of the target tester on the dynamic video to be analyzed, and generate a test viewing video;

[0050] A relevant data determination module, used to determine the eye movement data and facial expression data of the target tester based on the test viewing video;

[0051] A matching result generating module, used for generating a multimodal data matching result based on the eye movement data, the facial expression data and the dynamic video to be analyzed;

[0052] The analysis result generating module is used to generate the tracking analysis result based on the multimodal data matching result.

[0053] By adopting the above technical solutions and combining eye movement data and facial expression data, we can more comprehensively understand the target tester's attention distribution, emotional response and possible cognitive process when watching the video, and better reveal the tester's real reaction and experience. With the advancement of technology, eye tracking and facial expression recognition technology has become more mature and automated, and a large amount of data can be collected and analyzed in a shorter period of time, improving the efficiency and accuracy of research, so as to objectively and accurately analyze dynamic videos.

[0054] In a third aspect, the present application provides an electronic device, which adopts the following technical solution:

[0055] An electronic device comprises a processor, wherein the processor is coupled to a memory;

[0056] The processor is used to execute the computer program stored in the memory, so that the electronic device executes the computer program of the dynamic video emotion and attention tracking and analysis method described in any one of the first aspects.

[0057] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0058] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute the dynamic video emotion and attention tracking and analysis method described in any one of the first aspects.

[0059] In summary, this application includes the following beneficial technical effects:

[0060] Combining eye movement data and facial expression data can provide a more comprehensive understanding of the target test subject's attention distribution, emotional response, and possible cognitive process when watching the video, and can better reveal the test subject's true reaction and experience. With the advancement of technology, eye tracking and facial expression recognition technology has become more mature and automated, and can collect and analyze large amounts of data in a shorter period of time, improving the efficiency and accuracy of research, thereby objectively and accurately analyzing dynamic videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flow chart of a dynamic video emotion and attention tracking and analysis method provided in an embodiment of the present application.

[0062] Figure 2 It is a structural block diagram of a dynamic video emotion and attention tracking and analysis device provided in an embodiment of the present application.

[0063] Figure 3 It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The present application is further described in detail below in conjunction with the accompanying drawings.

[0065] The embodiment of the present application provides a dynamic video emotion and attention tracking analysis method, which can be executed by an electronic device, which can be a server or a terminal device, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smart phone, a tablet computer, a desktop computer, etc., but is not limited thereto.

[0066] Figure 1 A flowchart of a dynamic video emotion and attention tracking and analysis method provided in an embodiment of the present application.

[0067] like Figure 1 As shown, the main process of the method is described as follows (steps S101 to S105):

[0068] Step S101, obtaining a dynamic video to be analyzed.

[0069] In this embodiment, the dynamic video to be analyzed can be an advertisement, a trailer, a short video, or a clipped video, etc., and there is no restriction on the length of time. However, the video content of the dynamic video to be analyzed cannot be single and repetitive, such as a still ball or a ball that bounces at a certain frequency.

[0070] Step S102 , collecting the target tester's video viewing data of the dynamic video to be analyzed, and generating a test viewing video.

[0071] In this embodiment, when the dynamic video to be analyzed is played, the video viewing data of the target tester is synchronously collected, and the collection is stopped after the dynamic video to be analyzed is played, so as to obtain a test viewing video. The video start of the test viewing video is consistent with the start of the dynamic video to be analyzed, and the video end of the test viewing video is consistent with the end of the dynamic video to be analyzed.

[0072] Step S103, determining the eye movement data and facial expression data of the target tester based on the test viewing video.

[0073] For step S103, obtain the video frame rate of the dynamic video to be analyzed and the viewing frame rate of the test viewing video; determine whether the viewing frame rate is consistent with the video frame rate; if the viewing frame rate is consistent with the video frame rate, extract the video viewing data of the target tester frame by frame based on the viewing frame rate to generate eye movement data and facial expression data; if the viewing frame rate is inconsistent with the video frame rate, adjust the viewing frame rate based on the video frame rate to generate an adjusted viewing frame rate; extract the video viewing data of the target tester frame by frame based on the adjusted viewing frame rate to generate eye movement data and facial expression data.

[0074] In this embodiment, when determining the eye movement data and facial expression data of the target tester, it is necessary to make the frame rate of the test viewing video consistent with the frame rate of the dynamic video to be analyzed, so as to ensure the accuracy of subsequent data analysis. Similarly, in order to ensure the accuracy of data analysis, it is necessary to make the frame rate of the test viewing video consistent with the video frame rate of the dynamic video to be analyzed.

[0075] Determine whether the viewing frame rate is consistent with the video frame rate. If the viewing frame rate is consistent with the video frame rate, there is no need to adjust the viewing frame rate. The viewing frame rate can be used to extract the video viewing data. If the viewing frame rate is inconsistent with the video frame rate, it is necessary to adjust the viewing video frame rate, that is, based on the video frame rate, adjust the viewing frame rate to a state consistent with the video frame rate, use the adjusted viewing video frame rate as the adjusted viewing frame rate, and use the adjusted viewing frame rate to extract the video viewing data. Eye movement data and facial expression data will be extracted at both frame rates.

[0076] Furthermore, when determining the facial expression data and eye movement data based on the viewing frame rate, the facial emotions of the target tester in each frame are identified to determine the emotion type and emotion value of each frame; facial expression data are generated based on the emotion types and emotion values ​​of all frames; the eye feature points of the target tester are determined based on the video viewing data; the stay positions of the eye feature points of each frame are determined to determine the sight line coordinates of each frame; and eye movement data are generated based on the sight line coordinates of all frames.

[0077] The facial emotions of the target tester in each frame are identified through the corresponding facial micro-expression software to obtain the emotion type corresponding to the current frame and the emotion value used for the emotion type. In addition to the seven basic emotions of pleasure, curiosity, immersion, anger, fear, disgust, and contempt, the emotion type can also include complex emotions triggered in different scenarios. Complex emotions include but are not limited to concentration, joy, relief, happiness, romance, and love. The emotion type and emotion value of each frame are arranged in time series to generate an emotion line graph. The line graph can be used to obtain all the emotion types and emotion values ​​of the target tester in each frame. The generated emotion line graph is the final facial expression data.

[0078] When determining the eye movement data, we must first determine the eye feature points of the target tester. The eye feature points can be the entire eye, the eyeball part, or the pupil part. Determine the dwell position of the eye feature points in each frame to obtain the line of sight coordinates. The dwell position and line of sight coordinate system here are based on the playback device of the target dynamic video. Arrange the line of sight coordinates of all frames in time series to obtain the final eye movement data. The eye movement data is the line of sight trajectory of the target tester when watching the target dynamic video.

[0079] The screen used to play the target dynamic video is taken as a rectangular figure, and the vertex of the upper left corner of the rectangle is taken as the origin of the coordinate system. The x-axis is extended to the right and the y-axis is extended downward. The gaze point is estimated by using the eye movement model such as the line of sight projection model and the position of the eye feature point, and the pixel coordinates of the gaze point are converted into screen coordinates or coordinates related to the screen content. Since each image of each frame in the captured test viewing video is composed of pixels, each pixel in this image has its corresponding pixel coordinate point. The eye coordinate points of the eye feature points in each frame are identified according to the eye tracking algorithm. When the eye coordinate points are associated with the target dynamic video, the eye coordinate points are converted into corresponding pixel coordinate points, thereby determining the area watched by the target tester, that is, the line of sight coordinates. In general, since the line of sight of a person when watching a video has a certain range, not a pixel point, the line of sight coordinates are not the coordinate points corresponding to a pixel, but the coordinate points corresponding to the center point of an area.

[0080] It should be noted that the method for determining the facial expression data and the eye movement data based on adjusting the viewing frame rate is the same as that for determining the facial expression data and the eye movement data based on the viewing frame rate, and will not be elaborated here.

[0081] In this embodiment, before step S104, there is also a step of preprocessing the dynamic video to be analyzed to generate a target dynamic video, which includes obtaining the target object and target object information in the dynamic video to be analyzed; determining the object marking frame of the target object based on the target object information; obtaining key events and scene change information in the dynamic video to be analyzed; generating key event markings based on key events; generating scene change points based on scene change information; adding object marking frames, key event markings and scene change points to the dynamic video to be analyzed to generate the target dynamic video.

[0082] In this embodiment, after the dynamic video to be analyzed is obtained, the dynamic video to be analyzed is pre-processed to generate a target dynamic video with a mark, so as to facilitate the analysis of the dynamic video according to the video viewing data of the target tester.

[0083] In the dynamic video to be analyzed, the target objects include but are not limited to people, objects, brand trademarks and buildings. The target object can be one or more. The target object information is the size and shape of the target object. The target object is selected by the target detection model and the target object information, and a bounding box is generated to mark the target object. The generated bounding box is the object marking box of the target object. When the position, size and shape of the target object changes, the object marking box also changes accordingly, so that the target object is always located in the object marking box. YOLO, SSD, Faster R-CNN and other target detection models can be used to detect and determine the target object, and the intelligent bounding box (IBBs) technology is used to generate the object marking box, so as to realize automatic adjustment of the object marking box to adapt to the size and shape changes of the target object in the video.

[0084] When generating the object marking frame, the dynamic video to be analyzed is decomposed and analyzed, and the dynamic video to be analyzed is decomposed into frames according to the time series. The target object is determined and the object marking frame is generated frame by frame. In addition to determining the target object and generating the object marking frame for selecting and tracking the target object according to the target object information, the dynamic video to be analyzed is also analyzed frame by frame to determine the key events and scene change information in the dynamic video to be analyzed, and the key events are marked separately to determine the time when the key events occur and the frames where they are located. The key event tags are generated according to the time when the key events occur and the frames where they are located. When the dynamic video to be analyzed is played to the key events, the key events can be quickly identified and determined according to the key event tags. Key events are used to facilitate subsequent analysis. Similarly, when scene changes occur, they also need to be marked. The dynamic video to be analyzed is analyzed frame by frame to obtain the scene change information of the dynamic video to be analyzed. The scene change information includes scene change frames and change times corresponding to scene change frames. Scene change points are generated according to scene change frames and change times, and the scene change points are marked. When the dynamic video to be analyzed is played to the scene change points, the scene change information can be quickly identified and determined according to the key event markers, which is convenient for subsequent analysis. The object marker box, key event markers and scene change points are added to the dynamic video to be analyzed to obtain the target dynamic video.

[0085] Step S104, generating a multimodal data matching result based on the eye movement data, the facial expression data and the dynamic video to be analyzed.

[0086] For step S104, the eye movement data is matched with the target dynamic video and the dynamic video to be analyzed based on the video frame rate to generate a first matching result; the facial expression data is matched with the target dynamic video and the dynamic video to be analyzed based on the video frame rate to generate a second matching result; the target dynamic video is bound to the first matching result and the second matching result to generate a multimodal data matching result.

[0087] In this embodiment, the eye movement data is matched with the target dynamic video and the dynamic video to be analyzed through the time generated by each frame, that is, the time series, to obtain a first matching result. The relationship between the line of sight trajectory and the picture content can be known through the first matching result. The facial expression data is also matched with the target dynamic video and the dynamic video to be analyzed through the time series to generate a second matching result. The relationship between the expression change and the picture content can be known through the second matching result. Thereafter, the two matching results are bound to generate a multimodal data matching result, so as to determine the line of sight and emotional conditions of the target tester under different picture contents according to the multimodal data matching result.

[0088] Step S105: Generate a tracking analysis result based on the multimodal data matching result.

[0089] For step S105, the key gaze information of the target tester is determined based on the first matching result and the eye movement data; the attention-response relationship of the target tester is determined based on the key gaze information and the second matching result; the key emotion information of the target tester is determined based on the second matching result and the facial expression data; and the tracking analysis results are generated based on the key gaze information, the attention-response relationship and the key emotion information.

[0090] In this embodiment, the key gaze information includes key gaze content, gaze duration of the key gaze content, visual path, etc., and then the facial expression data corresponding to the key gaze information is obtained, the relationship between the specific attention pattern and the emotional response is analyzed, and the analysis result is used as the attention-response relationship. Then, the second matching result is compared and analyzed with the facial expression data to obtain the key emotional information of the target tester. The key emotional information is the main emotional response and change of the target tester when watching the video, the duration of the emotion and the picture content during the duration of the emotion, etc. The key gaze information, attention-response relationship and key emotional information are integrated and analyzed to obtain the final tracking analysis results. The tracking analysis results can reflect the trajectory of emotional changes, the correlation between emotions and focus points, etc., and also include whether the target tester focuses on certain specific parts of the video due to specific emotional responses, the attention and emotional response of the target tester when seeing key events and scene changes, etc.

[0091] In this embodiment, after the tracking analysis result is generated based on the multimodal data matching result, a video optimization strategy is formulated based on the tracking analysis result; and the video is optimized based on the video optimization strategy.

[0092] Based on the tracking and analysis results, we can determine the relationship between the advertising content and the audience's emotional attention, or the visual attention and emotional response when watching the video, etc., and identify the parts that need to be highlighted or improved. We can use these contents as video optimization strategies to optimize the video, increase the attractiveness and persuasiveness of the advertisement, and improve the attractiveness of the video and audience satisfaction.

[0093] Figure 2 A structural block diagram of a dynamic video emotion and attention tracking and analysis device 200 provided in an application embodiment.

[0094] like Figure 2 As shown, the dynamic video emotion and attention tracking and analysis device 200 mainly includes:

[0095] A dynamic video acquisition module 201 is used to acquire the dynamic video to be analyzed;

[0096] A test video generation module 202 is used to collect the video viewing data of the target tester on the dynamic video to be analyzed and generate a test viewing video;

[0097] A relevant data determination module 203, for determining the eye movement data and facial expression data of the target tester based on the test viewing video;

[0098] A matching result generating module 204, for generating a multimodal data matching result based on the eye movement data, the facial expression data and the dynamic video to be analyzed;

[0099] The analysis result generating module 205 is used to generate a tracking analysis result based on the multimodal data matching result.

[0100] As an optional implementation of this embodiment, the dynamic video emotion and attention tracking and analysis device 200 further includes:

[0101] A target information acquisition module is used to acquire the target object and target object information in the dynamic video to be analyzed;

[0102] An object mark determination module, used to determine an object mark frame of a target object based on target object information;

[0103] A key change acquisition module is used to obtain key events and scene change information in the dynamic video to be analyzed;

[0104] A key tag generation module, used for generating key event tags based on key events;

[0105] A scene change generation module, used for generating scene change points based on scene change information;

[0106] The target video generation module is used to add object marking frames, key event markings and scene change points to the dynamic video to be analyzed to generate the target dynamic video.

[0107] As an optional implementation of this embodiment, the relevant data determination module 203 includes:

[0108] A video frame rate acquisition module is used to acquire the video frame rate of the dynamic video to be analyzed and to acquire the viewing frame rate of the test viewing video;

[0109] A video frame rate determination module is used to determine whether the viewing frame rate is consistent with the video frame rate;

[0110] A first data generating module is used to extract the video viewing data of the target tester frame by frame based on the viewing frame rate to generate eye movement data and facial expression data;

[0111] An adjustment frame rate generation module, used to adjust the viewing frame rate based on the video frame rate to generate an adjusted viewing frame rate;

[0112] The second data generating module is used to extract the video viewing data of the target tester frame by frame based on adjusting the viewing frame rate, and generate eye movement data and facial expression data.

[0113] In this optional embodiment, the first data generation module is specifically used to identify the facial emotions of the target tester in each frame, determine the emotion type and emotion value of each frame; generate facial expression data based on the emotion type and emotion value of all frames; determine the eye feature points of the target tester based on the video viewing data; determine the stay position of the eye feature points of each frame, and determine the line of sight coordinates of each frame; generate eye movement data based on the line of sight coordinates of all frames.

[0114] As an optional implementation of this embodiment, the matching result generation module 204 is specifically used to match the eye movement data with the target dynamic video and the dynamic video to be analyzed based on the video frame rate to generate a first matching result; match the facial expression data with the target dynamic video and the dynamic video to be analyzed based on the video frame rate to generate a second matching result; bind the target dynamic video with the first matching result and the second matching result to generate a multimodal data matching result.

[0115] As an optional implementation of this embodiment, the analysis result generation module 205 is specifically used to determine the key gaze information of the target tester based on the first matching result and the eye movement data; determine the attention-response relationship of the target tester based on the key gaze information and the second matching result; determine the key emotion information of the target tester based on the second matching result and the facial expression data; and generate tracking analysis results based on the key gaze information, the attention-response relationship and the key emotion information.

[0116] As an optional implementation of this embodiment, the dynamic video emotion and attention tracking and analysis device 200 further includes:

[0117] An optimization strategy formulation module is used to formulate video optimization strategies based on tracking and analysis results;

[0118] The video optimization processing module is used to optimize the video based on the video optimization strategy.

[0119] In one example, the module in any of the above devices can be one or more integrated circuits configured to implement the above methods, such as: one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0120] For another example, when the modules in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0122] Figure 3 This is a structural block diagram of an electronic device 300 provided in an embodiment of the present application.

[0123] like Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302 , and may further include an information input / information output (I / O) interface 303 , one or more of a communication component 304 , and a communication bus 305 .

[0124] The processor 301 is used to control the overall operation of the electronic device 300 to complete all or part of the steps of the above-mentioned dynamic video emotion and attention tracking analysis method; the memory 302 is used to store various types of data to support the operation of the electronic device 300, and these data may include, for example, instructions for any application or method used to operate on the electronic device 300, and application-related data. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), programmable read-only memory (Programmable Read-Only Memory, PROM), read-only memory (Read-Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0125] The I / O interface 303 provides an interface between the processor 301 and other interface modules, and the above-mentioned other interface modules can be keyboards, mice, buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 304 is used for wired or wireless communication between the electronic device 300 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 104 can include: Wi-Fi components, Bluetooth components, NFC components.

[0126] The electronic device 300 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the dynamic video emotion and attention tracking and analysis method given in the above embodiment.

[0127] The communication bus 305 may include a path to transmit information between the above components. The communication bus 305 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 305 may be divided into an address bus, a data bus, a control bus, etc.

[0128] The electronic device 300 may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc., and may also be servers, etc.

[0129] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned dynamic video emotion and attention tracking and analysis method are implemented.

[0130] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0131] The terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, article, or apparatus.

[0132] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A dynamic video emotion and attention tracking analysis method, characterized in that: include: Get the dynamic video to be analyzed; Collecting the video viewing data of the target tester on the dynamic video to be analyzed, and generating a test viewing video; Determining eye movement data and facial expression data of the target tester based on the test viewing video; Generate a multimodal data matching result based on the eye movement data, the facial expression data and the dynamic video to be analyzed; A tracking analysis result is generated based on the multimodal data matching result.

2. The method according to claim 1, characterized in that Before generating a multimodal data matching result based on the eye movement data, the facial expression data and the dynamic video to be analyzed, the method further includes: Acquire the target object and the target object information in the dynamic video to be analyzed; Determine an object marking frame of the target object based on the target object information; Acquire key events and scene change information in the dynamic video to be analyzed; generating a key event marker based on the key event; generating a scene change point based on the scene change information; The object marking frame, the key event marking and the scene change point are added to the dynamic video to be analyzed to generate a target dynamic video.

3. The method according to claim 2, characterized in that The step of determining the eye movement data and facial expression data of the target tester based on the test viewing video comprises: Obtaining the video frame rate of the dynamic video to be analyzed and the viewing frame rate of the test viewing video; Determining whether the viewing frame rate is consistent with the video frame rate; If the viewing frame rate is consistent with the video frame rate, extracting the video viewing data of the target tester frame by frame based on the viewing frame rate to generate eye movement data and facial expression data; If the viewing frame rate is inconsistent with the video frame rate, adjusting the viewing frame rate based on the video frame rate to generate an adjusted viewing frame rate; The video viewing data of the target tester is extracted frame by frame based on the adjusted viewing frame rate to generate eye movement data and facial expression data.

4. The method according to claim 3, characterized in that The extracting the video viewing data of the target tester frame by frame based on the viewing frame rate to generate eye movement data and facial expression data comprises: Recognize the facial emotion of each frame of the target tester and determine the emotion type and emotion value of each frame; Generate facial expression data based on the emotion type and the emotion value of all frames; Determine eye feature points of the target tester based on the video viewing data; Determining the stay position of the eye feature point of each frame to determine the sight line coordinates of each frame; Eye movement data is generated based on the gaze coordinates of all frames.

5. The method according to claim 3, characterized in that: The generating of the multimodal data matching result based on the eye movement data, the facial expression data and the dynamic video to be analyzed comprises: Matching the eye movement data with the target dynamic video and the dynamic video to be analyzed based on the video frame rate to generate a first matching result; Matching the facial expression data with the target dynamic video and the dynamic video to be analyzed based on the video frame rate to generate a second matching result; The target dynamic video is bound to the first matching result and the second matching result to generate a multimodal data matching result.

6. The method according to claim 5, characterized in that Generating the tracking analysis result based on the multimodal data matching result includes: Determine key gaze information of the target tester based on the first matching result and the eye movement data; Determining the attention-response relationship of the target tester based on the key gaze information and the second matching result; Determine key emotional information of the target tester based on the second matching result and the facial expression data; A tracking analysis result is generated based on the key gaze information, the attention-response relationship and the key emotion information.

7. The method according to claim 1, characterized in that After generating the tracking analysis result based on the multimodal data matching result, the method further includes: Formulate a video optimization strategy based on the tracking and analysis results; The video is optimized based on the video optimization strategy.

8. A dynamic video emotion and attention tracking and analysis device, characterized in that: include: A dynamic video acquisition module is used to acquire the dynamic video to be analyzed; A test video generation module, used to collect the video viewing data of the target tester on the dynamic video to be analyzed, and generate a test viewing video; A relevant data determination module, used to determine the eye movement data and facial expression data of the target tester based on the test viewing video; A matching result generating module, used for generating a multimodal data matching result based on the eye movement data, the facial expression data and the dynamic video to be analyzed; The analysis result generating module is used to generate the tracking analysis result based on the multimodal data matching result.

9. An electronic device, characterized in that: comprising a processor coupled to a memory; The processor is configured to execute a computer program stored in the memory, so that the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The method comprises a computer program or an instruction, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 7.

Images