A multimodal image processing method and system

Abstract

The present invention discloses a multimodal image processing method and system, the method includes obtaining video data to be analyzed, establishing a time coordinate axis; extracting sound wave change images in the video data to be analyzed; extracting the time coordinate interval corresponding to the image segment whose change amplitude exceeds a preset range in the sound wave change image, extracting the corresponding screen image from the screen stream of the video data to be analyzed as a comparison image set according to the time coordinate interval; comparing each screen image in the comparison image set with a standard image to extract a target image containing the standard image, and obtaining the time coordinate corresponding to the target image; locating the first key segment in the video data to be analyzed according to the time coordinate corresponding to the target image; and displaying a segment switching touch button according to the first key segment along the progress bar of the video data to be analyzed. The technical solution of the present invention aims to quickly guide users to switch to key video segments.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and in particular to a multimodal image processing method and a multimodal image processing system. Background Art

[0002] Video data is a stream of images that changes over time, and contains richer information and content that cannot be expressed by other media. Transmitting information in the form of video can express the real world intuitively, vividly, realistically, and efficiently, and the amount of information transmitted is very rich, far greater than text or static images.

[0003] Synchronous sound data is often attached to video data. Since the picture brings a stronger intuitive feeling than the sound, people often pay more attention to the picture in a video data with audio, but ignore the sound.

[0004] When users watch audio videos (such as videos or TV series), they always like to pull the progress bar in the hope of skipping the screens they are not interested in and directly switch to the screens they are more interested in. When choosing to watch a video segment by skipping the screens, users often need to pull the progress bar to stay on a certain video segment for a short time to know whether the current video segment is the one they are interested in. If it is not the one they are interested in, they need to pull the progress bar again to skip several segments to switch to the next video segment and watch it again for a short time.

[0005] The efficiency of the above video switching selection is not high, and the selection of video clips is relatively blind, resulting in a weak user experience. Summary of the invention

[0006] The main purpose of the present invention is to provide a multimodal image processing method, aiming to quickly guide users to switch to key video clips.

[0007] To achieve the above object, the present invention provides a multimodal image processing method, comprising the following steps:

[0008] Acquire the video data to be analyzed, and acquire the start time and end time of the video data to be analyzed to establish a time coordinate axis;

[0009] Extracting the sound wave variation image that changes along the time coordinate axis in the video data to be analyzed;

[0010] Extracting the time coordinate interval corresponding to the image segment whose change amplitude exceeds the preset range in the sound wave change image, and extracting the corresponding screen image from the screen stream of the video data to be analyzed as a comparison image set according to the time coordinate interval;

[0011] By using an image recognition unit, each frame image in the comparison image set is compared with the standard image to extract a target image containing the standard image, and obtain a time coordinate corresponding to the target image;

[0012] Locating a first key segment in the video data to be analyzed according to the time coordinates corresponding to the target image;

[0013] Along the progress bar of the video data to be analyzed, a segment switching touch button is displayed according to the first key segment.

[0014] Preferably, the multimodal image processing method further includes:

[0015] Locating a second key segment from the video data to be analyzed according to the change of the sound wave change image along the time coordinate axis;

[0016] Along the progress bar of the video data to be analyzed, a segment switching touch button is displayed according to the second key segment.

[0017] Preferably, the step of extracting the sound wave variation image that changes with the time coordinate axis in the video data to be analyzed includes:

[0018] Extracting human voice sound data and environmental sound data from the video data to be analyzed;

[0019] Acquire a human voice sound wave change image established according to the human voice sound data and the time coordinate axis;

[0020] Acquire an environmental sound wave change image established according to the environmental sound data and the time coordinate axis;

[0021] Filtering a first target segment from the video data to be analyzed by using the human voice sound wave change image;

[0022] A second target segment is screened from the video data to be analyzed using the environmental sound wave change image.

[0023] Preferably, the step of selecting a first target segment from the video data to be analyzed by using the human voice sound wave change image comprises:

[0024] Obtaining the maximum value of the ordinate in the human voice sound wave change image, the minimum value of the ordinate, and the start time and end time of the human voice sound wave change image;

[0025] According to the maximum value of the ordinate in the human voice sound wave change image, the minimum value of the ordinate, and the start time and end time of the human voice sound wave change image, the width and height of the human voice sound wave scanning window are determined;

[0026] Scanning the human voice sound wave change image along the time coordinate axis using the human voice sound wave scanning window;

[0027] Calculate the first ordinate mean of each scanned human voice sound wave segment in the human voice sound wave scanning window;

[0028] Screening out a human voice sound wave segment whose first ordinate mean value exceeds a first preset value as a first target segment;

[0029] Among them, the height of the human voice sound wave scanning window is H a , width W a ;

[0030] H a =MAX(y a1 ,y a2 ......,y an )-MIN(y a1 ,y a2 ......,y an );

[0031] Among them, y a1 ,y a2 ......,y an , respectively represent the ordinate values ​​of the first sound wave coordinate point, the second sound wave coordinate point, ..., until the nth sound wave coordinate point in the sound wave change image of human voice; MAX(y a1 ,y a2 ......,y an ) represents the maximum value of the ordinate of each sound wave coordinate point in the sound wave change image of human voice; MIN(y a1 ,y a2 ......,y an ) represents the minimum value of the ordinate of each sound wave coordinate point in the sound wave change image of human voice;

[0032]

[0033] Among them, t 1 represents the horizontal coordinate value of the first sound wave coordinate point in the human voice sound wave change image, t n represents the horizontal coordinate value of the nth sound wave coordinate point in the human voice sound wave change image, t Wa The reference time width of the sound wave change image of human voice, t Ba The time width of the standard sound wave scanning window that represents the sound wave change image of human voice.

[0034] Preferably, the step of selecting a second target segment from the video data to be analyzed by using the environmental sound wave change image comprises:

[0035] Obtaining the maximum value of the ordinate and the minimum value of the ordinate in the environmental sound wave change image, as well as the start time and end time of the environmental sound wave change image;

[0036] According to the maximum value of the ordinate in the environmental sound wave change image, the minimum value of the ordinate, and the start time and end time of the environmental sound wave change image, the width and height of the environmental sound wave scanning window are determined;

[0037] Scanning the environmental sound wave change image along the time coordinate axis using the environmental sound wave scanning window;

[0038] Calculate the second ordinate mean of each scanned ambient sound wave waveform segment in the ambient sound wave scanning window;

[0039] Filter out the environmental sound wave waveform segments whose second ordinate mean value exceeds a second preset value as second target segments;

[0040] Among them, the height of the ambient sound wave scanning window is H b , width W b ;

[0041] H b =MAX(y b1 ,y b2 ......,y bn )-MIN(y b1 ,y b2 ......,y bn );

[0042] Among them, y b1 ,y b2 ......,y bn , respectively represent the ordinate values ​​of the first sound wave coordinate point, the second sound wave coordinate point, ..., until the nth sound wave coordinate point in the environmental sound wave change image; MAX(y b1 ,y b2 ......,y bn ) represents the maximum value of the ordinate of each sound wave coordinate point in the environmental sound wave change image; MIN(y b1 ,y b2 ......,y bn ) represents the minimum value of the ordinate of each sound wave coordinate point in the environmental sound wave change image;

[0043]

[0044] Among them, t 1 represents the horizontal coordinate value of the first sound wave coordinate point in the environmental sound wave change image, t nrepresents the horizontal coordinate value of the nth sound wave coordinate point in the environmental sound wave change image, t Wb Indicates the reference time width of the environmental sound wave change image, t Bb The time width of the standard sound wave scanning window that represents the image of environmental sound wave changes.

[0045] Preferably, the step of locating the second key segment from the video data to be analyzed according to the change of the sound wave change image along the time coordinate axis includes:

[0046] Obtaining a first time coordinate interval corresponding to each first target segment;

[0047] Obtaining the second time coordinate interval corresponding to each second target segment;

[0048] Determine whether there is temporal continuity between the first time coordinate interval and the second time coordinate interval;

[0049] Integrate the first time coordinate interval and the second time coordinate interval that have temporal continuity into one time coordinate interval;

[0050] Determine a corresponding progress bar interval according to the first time coordinate interval and the second time coordinate interval that do not have time continuity, and a time coordinate interval formed by integrating the first time coordinate interval and the second time coordinate interval, so as to form a key progress bar interval;

[0051] According to the key progress bar interval, a second key segment is located from the video data to be analyzed.

[0052] Preferably, the step of integrating the first time coordinate interval and the second time coordinate interval having temporal continuity into one time coordinate interval comprises:

[0053] Determine whether each first time coordinate interval and each second time coordinate interval having time continuity have an intersection;

[0054] If so, performing a union operation on the first time coordinate intervals and the second time coordinate intervals that have intersections to obtain a third time coordinate interval, until each first time coordinate interval and each second time coordinate interval have no intersections;

[0055] If not, obtain the time coordinate starting value and ending value in the first time coordinate interval and the second time coordinate interval that have time continuity and no intersection, and integrate the starting value and the ending value to obtain a fourth time coordinate interval.

[0056] Preferably, the step of displaying a segment switching touch key according to the second key segment along the progress bar of the video data to be analyzed includes:

[0057] Determine a progress bar interval corresponding to a time coordinate interval of the second key segment;

[0058] Acquire a sound wave change image display banner preset on the playback interface, wherein the sound wave change image display banner is located above or below the progress bar and extends along the progress bar;

[0059] Displaying the sound wave change image in the sound wave change image display banner along the progress bar;

[0060] Marking the progress bar interval on the progress bar;

[0061] Acquire the starting time coordinates corresponding to each of the second key segments, and retrieve the starting screen image corresponding to each of the second key segments according to the starting coordinates;

[0062] The starting screen image of the second key segment is displayed along the progress bar, and each starting screen image of the second key segment is associated with a segment switching touch key.

[0063] Preferably, the step of obtaining a human voice sound wave change image established according to the human voice sound data and the time coordinate axis comprises:

[0064] Acquire short-time energy, fundamental frequency data, speech rate data and formant data that vary with time in the human voice sound data;

[0065] Determine the human voice sound wave effect value at the corresponding time point according to the short-time energy, the fundamental frequency data, the speech rate data and the formant data at each time point;

[0066] Using the time value as the horizontal coordinate and the vertical effect value of the human voice sound wave as the vertical coordinate, a human voice sound wave change image is established;

[0067] and / or

[0068] The step of obtaining an environmental sound wave change image established according to the environmental sound data and the time coordinate axis comprises:

[0069] Acquire amplitude data of the environmental sound data that changes with time;

[0070] Using the time value as the abscissa and the amplitude data as the ordinate, an environmental sound wave change image is established;

[0071] The calculation formula of the sound wave effect value of human voice is:

[0072]

[0073] y aiIndicates the vocal sound wave effect value (vertical coordinate value) corresponding to the i-th sound wave coordinate point in the vocal sound wave change image, 0 <i≤n;

[0074] r 1 、r 2 、r 3 and r 4 are the weights corresponding to short-time energy, fundamental frequency data, speech rate data and formant data, and r 1 、r 2 、r 3 and r 4 The value range of is (0,1). 1 +r 2 +r 3 +r 4 =1;

[0075] x ai 、z ai 、e ai 、f ai Respectively represent the short-time energy, fundamental frequency data, speech rate data and formant data corresponding to the i-th sound wave coordinate point in the human voice sound wave change image;

[0076] a, b, c, and d represent the standard values ​​of short-time energy, fundamental frequency data, speech rate data, and formant data, respectively;

[0077] Among them, in the environmental sound wave change image, y bj represents the ambient sound wave amplitude value (ordinate value) corresponding to the j-th sound wave coordinate point in the ambient sound wave change image, 0 <j≤n。

[0078] In addition, to achieve the above object, the present invention further provides a multimodal image processing system for executing the steps of any one of the above methods, the multimodal image processing system comprising:

[0079] An acquisition module, used for acquiring the video data to be analyzed, and acquiring the start time and end time of the video data to be analyzed, so as to establish a time coordinate axis;

[0080] A sound wave variation image extraction module, used for extracting the sound wave variation image that varies with the time coordinate axis in the video data to be analyzed;

[0081] A comparison image extraction module, used to extract the time coordinate interval corresponding to the image segment whose change amplitude exceeds the preset range in the sound wave change image, and extract the corresponding screen image from the screen stream of the video data to be analyzed as a comparison image set according to the time coordinate interval;

[0082] A recognition module, used to compare each picture image in the comparison image set with the standard image through an image recognition unit, so as to extract a target image containing the standard image, and obtain a time coordinate corresponding to the target image;

[0083] A positioning module, used for locating a first key segment in the video data to be analyzed according to a time coordinate corresponding to the target image;

[0084] A touch selection module is used to display a segment switching touch button according to the first key segment along the progress bar of the video data to be analyzed.

[0085] In the technical solution of the present invention, there is no need to perform any semantic analysis or speech recognition on the video data to be analyzed. Instead, a multimodal image determined by a picture image and a sound wave change image in the video data to be analyzed is used to locate the key segments of the video data to be analyzed, so that the extraction is accurate and efficient. Specifically, an image analysis is performed on the sound wave change image to find out the time coordinate interval corresponding to the interval with significant change amplitude in the sound wave change image, and further extract the corresponding comparison image set from the picture stream of the video data to be analyzed through the time coordinate interval, and obtain the target image containing the standard image from the comparison image set, so as to locate the key segment (i.e., the first key segment) of the video to be analyzed according to the time coordinate corresponding to the target image, so as to achieve accurate positioning, and further, along the progress bar of the video data to be analyzed, a segment switching touch button is displayed according to the first key segment, and the segment switching touch button of the first key segment can further use the starting screen image of the first key segment as the background image, which can not only prompt the user where the first key segment is located on the progress bar, but also play the effect of segment content prompting through the starting screen image in the segment switching touch button, so that the user can quickly locate the video segment that he wants to focus on from each first key segment. Therefore, the technical solution of the present invention is conducive to quickly guiding the user to switch to the key video segment. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 FIG. 4 is a flow chart of a first embodiment of a multimodal image processing method according to the present invention.

[0087] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0088] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0089] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0090] See also Figure 1 To achieve the above object, the present invention provides a multimodal image processing method, comprising the following steps:

[0091] Step S10, obtaining the video data to be analyzed, and obtaining the start time and end time of the video data to be analyzed to establish a time coordinate axis;

[0092] Step S20, extracting the sound wave variation image that changes with the time coordinate axis in the video data to be analyzed;

[0093] Step S30, extracting the time coordinate interval corresponding to the image segment whose change amplitude exceeds the preset range in the sound wave change image, and extracting the corresponding screen image from the screen stream of the video data to be analyzed as a comparison image set according to the time coordinate interval;

[0094] Step S40, comparing each frame image in the comparison image set with the standard image through the image recognition unit to extract the target image containing the standard image, and obtaining the time coordinate corresponding to the target image;

[0095] Step S50, locating a first key segment in the video data to be analyzed according to the time coordinate corresponding to the target image;

[0096] Step S60: displaying a segment switching touch button according to the first key segment along the progress bar of the video data to be analyzed.

[0097] In the technical solution of the present invention, there is no need to perform any semantic analysis or speech recognition on the video data to be analyzed. Instead, a multimodal image determined by a picture image and a sound wave change image in the video data to be analyzed is used to locate the key segments of the video data to be analyzed, so that the extraction is accurate and efficient. Specifically, an image analysis is performed on the sound wave change image to find out the time coordinate interval corresponding to the interval with significant change amplitude in the sound wave change image, and further extract the corresponding comparison image set from the picture stream of the video data to be analyzed through the time coordinate interval, and obtain the target image containing the standard image from the comparison image set, so as to locate the key segment (i.e., the first key segment) of the video to be analyzed according to the time coordinate corresponding to the target image, so as to achieve accurate positioning, and further, along the progress bar of the video data to be analyzed, a segment switching touch button is displayed according to the first key segment, and the segment switching touch button of the first key segment can further use the starting screen image of the first key segment as the background image, which can not only prompt the user where the first key segment is located on the progress bar, but also play the effect of segment content prompting through the starting screen image in the segment switching touch button, so that the user can quickly locate the video segment that he wants to focus on from each first key segment. Therefore, the technical solution of the present invention is conducive to quickly guiding the user to switch to the key video segment.

[0098] Specifically, the video data to be analyzed may be a variety of video images, such as TV series, movies or videos. The start time and end time of the video data to be analyzed may have a variety of embodiments, for example, the start time of the video data to be analyzed is 00:00, and the end time is the total duration of the video data; the start time of the video data to be analyzed is the start time of the video recording, and the end time is the stop time when the video recording ends.

[0099] The video data to be analyzed is composed of a picture stream, and the picture stream is formed by switching each picture image along the time coordinate axis.

[0100] In the present invention, the sound wave change image is used to characterize the dynamic change of the sound wave along the time axis. For audio video data, the change of the sound wave change image can indicate the change of the plot. For example, the sound change in the movie video indicates the evolution of the plot. When the plot is intense, the sound often has significant changes. However, the silent picture data requires the user to watch a series of pictures and perform a deep analysis of the pictures in the mind to roughly understand the plot expressed by the silent picture data. In addition, since the silent picture data has no sound prompt, the user only relies on the picture content to guess the meaning of the video, which is easy to produce a large understanding deviation and is not conducive to the positioning of the key segments of the video. The segment with a relatively strong plot change can be quickly located by the change amplitude of the sound wave change image.

[0101] Screening out the time coordinate intervals with strong plot changes through the sound wave change image is the first step of video screening. In the second step, it is necessary to further screen the video clips corresponding to the time coordinate intervals with strong plot changes. In the present invention, the standard image refers to the image that the user is more likely to pay attention to. For example, the standard image can be a specified face image (for example, the face image of person A), a specified object image, etc. Screening out the target image from the comparison image set through the standard image is conducive to locating a more accurate video clip, such as locating a video clip where relevant characters appear.

[0102] Furthermore, in step S40, a preset standard image can be used as an image option for the user to select, so as to filter out the key segments that the user is interested in according to the standard image selected by the user. Alternatively, the user may not be allowed to participate in the selection, and the image processing staff may directly upload several standard images to directly complete the comparison.

[0103] Further, in step S50, the time coordinate corresponding to the target image may be a time point rather than a time coordinate interval. At this time, the time coordinate point corresponding to the target image is moved forward by a preset time and moved backward by a preset time. Based on the time point value corresponding to the preset time moved forward and the time point value corresponding to the preset time moved backward, the time coordinate interval corresponding to the first key segment is determined, thereby determining the first key segment.

[0104] Based on the first embodiment of the multimodal image processing method of the present invention, in a second embodiment of the multimodal image processing method of the present invention, the multimodal image processing method further includes:

[0105] Step S70, locating a second key segment from the video data to be analyzed according to the change of the sound wave change image along the time coordinate axis;

[0106] Step S80: displaying a segment switching touch button according to the second key segment along the progress bar of the video data to be analyzed.

[0107] The second key segment is more related to the sound wave change image, but less related to the video change image. By screening the second key segment, the video segment corresponding to the time coordinate interval in which the change amplitude of the sound wave change image exceeds the preset range can be quickly located as the second key segment.

[0108] Furthermore, along the progress bar of the video data to be analyzed, the segment switching touch button of the first key segment and the segment switching touch button of the second key segment can be displayed differently (for example, color difference, text difference or flashing difference) so that the user can obtain richer reference information when selecting key segments and can select more types of key segments.

[0109] Based on the second embodiment of the multimodal image processing method of the present invention, in a third embodiment of the multimodal image processing method of the present invention, step S20 includes:

[0110] Step S21, extracting human voice sound data and environmental sound data from the video data to be analyzed;

[0111] Step S22, obtaining a human voice sound wave change image established according to the human voice sound data and the time coordinate axis;

[0112] Step S23, obtaining an environmental sound wave change image established according to the environmental sound data and the time coordinate axis;

[0113] Step S24, screening a first target segment from the video data to be analyzed by using the human voice sound wave change image;

[0114] Step S25: screening a second target segment from the video data to be analyzed by using the environmental sound wave change image.

[0115] In the usual sound processing process, only the human voice is focused on, and the environmental sound is filtered out as noise to enhance the expression of the human voice. In the present invention, not only the human voice but also the environmental sound is focused on, and the importance of both is equal. Therefore, in the present invention, it is necessary to extract the human voice sound data and the environmental sound data in the video data to be analyzed.

[0116] By acquiring the image mutation area (the area where the change amplitude exceeds the preset range) in the human voice sound wave change image, and the image mutation area (the area where the change amplitude exceeds the preset range) in the environmental sound wave change image, the sudden, tense or intense plot can be determined, and thus, the first target segment and the second target segment can be screened out from the video data to be analyzed only by identifying the sudden change area of ​​the image.

[0117] Specifically, in the video data to be analyzed, both the human voice data and the environmental sound data can reflect the criticality of each video clip through the volume. Not only that, the human voice data also has the value of emotional analysis to determine the current emotional state of the human voice. The emotional state can also express the criticality of the video clip. For example, the human voice data corresponding to the sad emotion often has a low volume, but corresponds to the strong ups and downs of the plot.

[0118] Furthermore, human voice data also includes many types. The human voice in the present invention is defined as the sound made by a person, including: speaking, laughing, crying, roaring, shouting, breathing, wailing and breathing. Among human voices, speaking requires emotional analysis or semantic recognition to infer whether the corresponding video clip is critical, while other types of human voices other than speaking are direct expressions of emotions and plots. By identifying which clips have which types of human voices, the second key clip in the video data to be analyzed can also be quickly located.

[0119] In the present invention, a neural network model for analyzing the types of human voice sound data is established, and the data on the changes of the characteristic quantity data of various types of human voice sound data over time is used as input data, and the type of human voice sound data is brought into the neural network model as output data for training; the data on the changes of the characteristic quantity data of the human voice sound data corresponding to each analysis segment over time is input into the trained neural network model to identify the type of human voice sound data corresponding to the analysis segment. Among them, the characteristic quantity of human voice sound data includes frequency data, amplitude data and waveform data, among which frequency data determines pitch, amplitude data determines loudness, and waveform data determines timbre, and the three types of data are used to jointly determine the sound type.

[0120] Based on the third embodiment of the multimodal image processing method of the present invention, in a fourth embodiment of the multimodal image processing method of the present invention, step S24 includes:

[0121] Step S241, obtaining the maximum value of the ordinate in the human voice sound wave change image, the minimum value of the ordinate, and the start time and end time of the human voice sound wave change image;

[0122] Step S242, determining the width and height of the human voice sound wave scanning window according to the maximum value of the ordinate in the human voice sound wave change image, the minimum value of the ordinate, and the start time and end time of the human voice sound wave change image;

[0123] Step S243, scanning the human voice sound wave change image along the time coordinate axis using the human voice sound wave scanning window;

[0124] Step S244, calculating the first ordinate mean of each scanned human voice wave segment in the human voice wave scanning window;

[0125] Step S245, selecting a human voice sound wave segment whose first ordinate mean exceeds a first preset value as a first target segment;

[0126] Among them, the height of the human voice sound wave scanning window is H a , width W a ;

[0127] H a =MAX(y a1 ,y a2 ......,y an )-MIN(y a1 ,y a2 ......,y an );

[0128] Among them, y a1 ,y a2 ......,y an , respectively represent the ordinate values ​​of the first sound wave coordinate point, the second sound wave coordinate point, ..., until the nth sound wave coordinate point in the sound wave change image of human voice; MAX(y a1 ,y a2 ......,y an ) represents the maximum value of the ordinate of each sound wave coordinate point in the sound wave change image of human voice; MIN(y a1 ,y a2 ......,y an ) represents the minimum value of the ordinate of each sound wave coordinate point in the sound wave change image of human voice;

[0129]

[0130] Among them, t 1 represents the horizontal coordinate value of the first sound wave coordinate point in the human voice sound wave change image, t n represents the horizontal coordinate value of the nth sound wave coordinate point in the human voice sound wave change image, t Wa The reference time width of the sound wave change image of human voice, t Ba The time width of the standard sound wave scanning window that represents the sound wave change image of human voice.

[0131] Among them, the vertical coordinate in the human voice sound wave change image can be one of short-time energy, fundamental frequency data, speaking rate data and resonance peak data, or a human voice sound wave effect value determined based on short-time energy, fundamental frequency data, speaking rate data and resonance peak data.

[0132] The horizontal axes in the human voice sound wave change image and the environmental sound wave change image are both time.

[0133] In the present invention, a human voice sound wave scanning window with adjustable height and width is used to scan the human voice sound wave change image, wherein the height of the human voice sound wave scanning window is determined according to the maximum value and the minimum value of the ordinate in the human voice sound wave change image, so as to avoid the scanning window height being too large, for example, the interval between the ordinate value of 0 and the minimum value does not need to be scanned, thereby reducing the data scanned by the human voice sound wave scanning window. In addition, the width of the human voice sound wave scanning window can be adaptively adjusted according to the total duration of the video data to be analyzed.

[0134] Based on the third embodiment or the fourth embodiment of the multimodal image processing method of the present invention, in a fifth embodiment of the multimodal image processing method of the present invention, the step S25 further includes:

[0135] Step S251, obtaining the maximum value of the ordinate in the environmental sound wave change image, the minimum value of the ordinate, and the start time and end time of the environmental sound wave change image;

[0136] Step S252, determining the width and height of the ambient sound wave scanning window according to the maximum value of the ordinate in the ambient sound wave change image, the minimum value of the ordinate, and the start time and end time of the ambient sound wave change image;

[0137] Step S253, scanning the environmental sound wave change image along the time coordinate axis using the environmental sound wave scanning window;

[0138] Step S254, calculating the second ordinate mean of each scanned ambient sound wave waveform segment in the ambient sound wave scanning window;

[0139] Step S255, selecting the environmental sound wave waveform segments whose second ordinate mean value exceeds a second preset value as second target segments;

[0140] Among them, the height of the ambient sound wave scanning window is H b , width W b ;

[0141] H b =MAX(y b1 ,y b2 ......,y bn )-MIN(y b1 ,y b2 ......,y bn );

[0142] Among them, y b1 ,y b2 ......,y bn , respectively represent the ordinate values ​​of the first sound wave coordinate point, the second sound wave coordinate point, ..., until the nth sound wave coordinate point in the environmental sound wave change image; MAX(y b1 ,y b2 ......,y bn ) represents the maximum value of the ordinate of each sound wave coordinate point in the environmental sound wave change image; MIN(y b1 ,y b2 ......,y bn ) represents the minimum value of the ordinate of each sound wave coordinate point in the environmental sound wave change image;

[0143]

[0144] Among them, t 1 represents the horizontal coordinate value of the first sound wave coordinate point in the environmental sound wave change image, t n represents the horizontal coordinate value of the nth sound wave coordinate point in the environmental sound wave change image, t Wb Indicates the reference time width of the environmental sound wave change image, t Bb The time width of the standard sound wave scanning window that represents the image of environmental sound wave changes.

[0145] The vertical coordinate in the environmental sound wave change image may be determined according to one of the frequency and the amplitude, or may be an effect value determined according to both the frequency and the amplitude.

[0146] In the present invention, an ambient sound wave scanning window with adjustable height and width is used to scan the ambient sound wave change image, wherein the height of the ambient sound wave scanning window is determined according to the maximum value and the minimum value of the ordinate in the ambient sound wave change image, so as to avoid the scanning window height being too large. For example, the interval between the ordinate value of 0 and the minimum value does not need to be scanned, thereby reducing the data scanned by the ambient sound wave scanning window. In addition, the width of the ambient sound wave scanning window can be adaptively adjusted according to the total duration of the video data to be analyzed.

[0147] Based on the third to fifth embodiments of the multimodal image processing method of the present invention, in a sixth embodiment of the multimodal image processing method of the present invention, step S70 includes:

[0148] Step S71, obtaining the first time coordinate interval corresponding to each first target segment;

[0149] Step S72, obtaining the second time coordinate interval corresponding to each second target segment;

[0150] Step S73, determining whether there is temporal continuity between the first time coordinate interval and the second time coordinate interval;

[0151] Step S74, integrating the first time coordinate interval and the second time coordinate interval that are temporally continuous into one time coordinate interval;

[0152] Step S75, determining a corresponding progress bar interval according to the first time coordinate interval and the second time coordinate interval that do not have time continuity, and the time coordinate interval formed by integrating the first time coordinate interval and the second time coordinate interval, so as to form a key progress bar interval;

[0153] Step S76: locating a second key segment from the video data to be analyzed according to the key progress bar interval.

[0154] Among them, judging whether there is time continuity between the first time coordinate interval and the second time coordinate interval includes: judging whether there is an intersection between adjacent first time coordinate intervals, adjacent first time coordinate intervals and second time coordinate intervals, and adjacent second time coordinate intervals. If so, it is determined that there is time continuity. If not, the time coordinate interval values ​​of adjacent first time coordinate intervals without intersection, adjacent first time coordinate intervals without intersection, and adjacent second time coordinate intervals without intersection are obtained. If the time coordinate interval value does not exceed the preset interval value, it is determined that there is time continuity. If the time coordinate interval value exceeds the preset interval value, it is determined that there is no time continuity. In this way, adjacent first time coordinate intervals and second time coordinate intervals with short time intervals are integrated into one time coordinate interval to avoid too many time coordinate intervals.

[0155] Based on the sixth embodiment of the multimodal image processing method of the present invention, in a seventh embodiment of the multimodal image processing method of the present invention, step S75 includes:

[0156] Step S751, determining whether each first time coordinate interval with time continuity and each second time coordinate interval have an intersection; specifically, determining whether the first time coordinate interval with time continuity, the first time coordinate interval with time continuity and the second time coordinate interval with time continuity, and the second time coordinate interval with time continuity have an intersection;

[0157] If so, execute step S752: perform a union operation on the first time coordinate intervals and the second time coordinate intervals that have an intersection to integrate and obtain a third time coordinate interval, until each first time coordinate interval and each second time coordinate interval has no intersection; specifically, integrate the first time coordinate intervals that have an intersection, the first time coordinate intervals and the second time coordinate intervals that have an intersection, and the second time coordinate intervals that have an intersection to obtain respective third time coordinate intervals, until each first time coordinate interval, each first time coordinate interval and the second time coordinate interval, and each second time coordinate interval have no intersection.

[0158] If not, execute step S753: obtain the starting value and ending value of the time coordinates in the first time coordinate interval and the second time coordinate interval that have time continuity and no intersection, and integrate to obtain a fourth time coordinate interval according to the starting value and the ending value; specifically, obtain the starting value and ending value of the time coordinates in each first time coordinate interval that has time continuity and no intersection, to obtain the fourth time coordinate interval; obtain the starting value and ending value of the time coordinates in each first time coordinate interval and the second time coordinate interval that has time continuity and no intersection, to obtain the fourth time coordinate interval; obtain the starting value and ending value of the time coordinates in each second time coordinate interval that has time continuity and no intersection, to obtain the fourth time coordinate interval.

[0159] In this embodiment, the third time coordinate interval and the fourth time coordinate interval are both integrated time coordinate intervals.

[0160] Based on the second to seventh embodiments of the multimodal image processing method of the present invention, in an eighth embodiment of the multimodal image processing method of the present invention, step S80 includes:

[0161] Step S81, determining the progress bar interval corresponding to the time coordinate interval of the second key segment;

[0162] Step S82, obtaining a sound wave change image display banner preset on the playback interface, wherein the sound wave change image display banner is located above or below the progress bar and extends along the progress bar;

[0163] Step S83, displaying the sound wave change image along the progress bar in the sound wave change image display banner;

[0164] Step S84, marking the progress bar interval on the progress bar;

[0165] Step S85, obtaining the starting time coordinates corresponding to each of the second key segments, and retrieving the starting screen image corresponding to each of the second key segments according to the starting coordinates;

[0166] Step S86: displaying the starting screen image of the second key segment along the progress bar, and each starting screen image of the second key segment is associated with a segment switching touch key.

[0167] This embodiment provides a specific solution for displaying a segment switching touch button according to the second key segment along the progress bar of the video data to be analyzed.

[0168] Further, regarding the specific solution of displaying the segment switching touch button along the progress bar of the video data to be analyzed according to the first key segment, it can be performed in the following manner:

[0169] Step S61, determining the progress bar interval corresponding to the time coordinate interval of the first key segment;

[0170] Step S62, obtaining a target image display banner preset on the playback interface, wherein the target image display banner is located above or below the progress bar and extends along the progress bar;

[0171] Step S63, displaying the target image in the target image display banner along the progress bar;

[0172] Step S64, obtaining the starting time coordinates corresponding to each of the first key segments, and retrieving the starting screen image corresponding to each of the first key segments according to the starting coordinates;

[0173] Step S65: displaying the starting screen image of the first key segment along the progress bar, and each starting screen image of the first key segment is associated with a segment switching touch key.

[0174] Furthermore, one of the sound wave change image display banner and the target image display banner is located above the progress bar, and the other is located below the progress bar.

[0175] Based on the third to seventh embodiments of the multimodal image processing method of the present invention, in a ninth embodiment of the multimodal image processing method of the present invention, step S22 includes:

[0176] Step S221, obtaining short-time energy, fundamental frequency data, speech rate data and formant data that vary with time in the human voice sound data;

[0177] Step S222, determining the human voice sound wave effect value at the corresponding time point according to the short-time energy, the fundamental frequency data, the speech rate data and the formant data at each time point;

[0178] Step S223, using the time value as the horizontal coordinate and the vertical effect value of the human voice sound wave as the vertical coordinate to establish a human voice sound wave change image.

[0179] and / or

[0180] The step S23 comprises:

[0181] Step S231, obtaining amplitude data of the ambient sound data that changes with time;

[0182] Step S232, using the time value as the horizontal coordinate and the amplitude data as the vertical coordinate to create an environmental sound wave change image.

[0183] In a video, in addition to video clips with loud volume that can correspond to key video clips, clips with ups and downs in character emotions can also locate key video clips. Therefore, in this embodiment, the emotional type of the human voice is determined by the sound wave effect value of the human voice, so that the key video clip is located according to the emotional type.

[0184] Specifically, the formant feature can reflect the trend of emotional characteristics to a certain extent and is suitable for speech emotion recognition; short-term energy represents the volume level and is also an important parameter reflecting emotions; speech rate reflects excitement or calmness; and the fundamental frequency varies widely under different emotions. Therefore, short-term energy, fundamental frequency data, speech rate data, and formant data can jointly determine the emotion type corresponding to the human voice.

[0185] The calculation formula of the sound wave effect value of human voice is:

[0186]

[0187] y ai Indicates the vocal sound wave effect value (vertical coordinate value) corresponding to the i-th sound wave coordinate point in the vocal sound wave change image, 0 <i≤n。

[0188] r 1 、r 2 、r 3 and r 4 are the weights corresponding to short-time energy, fundamental frequency data, speech rate data and formant data, and their value ranges are all (0,1). At the same time, r 1 +r 2 +r 3 +r 4 =1.

[0189] x ai 、z ai 、e ai 、f ai Respectively represent the short-time energy, fundamental frequency data, speech rate data and formant data corresponding to the i-th sound wave coordinate point in the human voice sound wave change image;

[0190] a, b, c, and d represent the standard value of short-time energy, the standard value of fundamental frequency data, the standard value of speech rate data, and the standard value of formant data, respectively.

[0191] Among them, in the environmental sound wave change image, y bj represents the ambient sound wave amplitude value (ordinate value) corresponding to the j-th sound wave coordinate point in the ambient sound wave change image, 0 <j≤n。

[0192] To achieve the above object, the present invention further provides a multimodal image processing system for executing the steps of any of the above methods, the multimodal image processing system comprising:

[0193] An acquisition module, used for acquiring the video data to be analyzed, and acquiring the start time and end time of the video data to be analyzed, so as to establish a time coordinate axis;

[0194] A sound wave variation image extraction module, used for extracting the sound wave variation image that varies with the time coordinate axis in the video data to be analyzed;

[0195] A comparison image extraction module, used to extract the time coordinate interval corresponding to the image segment whose change amplitude exceeds the preset range in the sound wave change image, and extract the corresponding screen image from the screen stream of the video data to be analyzed as a comparison image set according to the time coordinate interval;

[0196] A recognition module, used to compare each picture image in the comparison image set with the standard image through an image recognition unit, so as to extract a target image containing the standard image, and obtain a time coordinate corresponding to the target image;

[0197] A positioning module, used for locating a first key segment in the video data to be analyzed according to a time coordinate corresponding to the target image;

[0198] A touch selection module is used to display a segment switching touch button according to the first key segment along the progress bar of the video data to be analyzed.

[0199] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device to enter the method described in each embodiment of the present invention.

[0200] In the description of this specification, the description with reference to the terms "an embodiment", "another embodiment", "other embodiments", or "first embodiment to Xth embodiment" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, method steps or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0201] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0202] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0203] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A multimodal image processing method, characterized in that: The steps include: Acquire the video data to be analyzed, and acquire the start time and end time of the video data to be analyzed to establish a time coordinate axis; Extracting the sound wave change image that changes with the time coordinate axis in the video data to be analyzed, including: extracting human voice sound data and environmental sound data in the video data to be analyzed; obtaining a human voice sound wave change image established according to the human voice sound data and the time coordinate axis; obtaining an environmental sound wave change image established according to the environmental sound data and the time coordinate axis; screening a first target segment from the video data to be analyzed by using the human voice sound wave change image; screening a second target segment from the video data to be analyzed by using the environmental sound wave change image; Extracting the time coordinate interval corresponding to the image segment whose change amplitude exceeds the preset range in the sound wave change image, and extracting the corresponding screen image from the screen stream of the video data to be analyzed as a comparison image set according to the time coordinate interval; By using an image recognition unit, each frame image in the comparison image set is compared with the standard image to extract a target image containing the standard image, and obtain a time coordinate corresponding to the target image; Locating a first key segment in the video data to be analyzed according to the time coordinates corresponding to the target image; Displaying a segment switching touch button according to the first key segment along the progress bar of the video data to be analyzed; According to the change of the sound wave change image along the time coordinate axis, the second key segment is located from the video data to be analyzed, including: obtaining the first time coordinate interval corresponding to each first target segment; obtaining the second time coordinate interval corresponding to each second target segment; judging whether there is time continuity between the first time coordinate interval and the second time coordinate interval; integrating the first time coordinate interval and the second time coordinate interval with time continuity into one time coordinate interval; determining the corresponding progress bar interval according to the first time coordinate interval and the second time coordinate interval without time continuity, and according to the time coordinate interval integrated from the first time coordinate interval and the second time coordinate interval, so as to form a key progress bar interval; according to the key progress bar interval, the second key segment is located from the video data to be analyzed; Along the progress bar of the video data to be analyzed, a segment switching touch button is displayed according to the second key segment.

2. The multimodal image processing method according to claim 1, characterized in that: The step of selecting a first target segment from the video data to be analyzed by using the human voice sound wave change image comprises: Obtaining the maximum value of the ordinate in the human voice sound wave change image, the minimum value of the ordinate, and the start time and end time of the human voice sound wave change image; According to the maximum value of the ordinate in the human voice sound wave change image, the minimum value of the ordinate, and the start time and end time of the human voice sound wave change image, the width and height of the human voice sound wave scanning window are determined; Scanning the human voice sound wave change image along the time coordinate axis using the human voice sound wave scanning window; Calculate the first ordinate mean of each scanned human voice sound wave segment in the human voice sound wave scanning window; Screening out a human voice sound wave segment whose first ordinate mean value exceeds a first preset value as a first target segment; Among them, the height of the human voice sound wave scanning window is H a , width W a ; H a =MAX(and a1 ,and a2 ......,and an )-MIN(and a1 ,and a2 ......,and an ); Among them, y a1 ,y a2 ......,y an , respectively represent the ordinate values ​​of the first sound wave coordinate point, the second sound wave coordinate point, ..., until the nth sound wave coordinate point in the sound wave change image of human voice; MAX(y a1 ,y a2 ......,y an ) represents the maximum value of the ordinate of each sound wave coordinate point in the sound wave change image of human voice; MIN(y a1 ,y a2 ......,y an ) represents the minimum value of the ordinate of each sound wave coordinate point in the sound wave change image of human voice; Wherein, t1 represents the horizontal coordinate value of the first sound wave coordinate point in the human voice sound wave change image, t n represents the horizontal coordinate value of the nth sound wave coordinate point in the human voice sound wave change image, t Wa The reference time width of the sound wave change image of human voice, t Ba The time width of the standard sound wave scanning window that represents the sound wave change image of human voice.

3. The multimodal image processing method according to claim 1, characterized in that: The step of screening a second target segment from the video data to be analyzed by using the environmental sound wave change image comprises: Obtaining the maximum value of the ordinate and the minimum value of the ordinate in the environmental sound wave change image, as well as the start time and end time of the environmental sound wave change image; According to the maximum value of the ordinate in the environmental sound wave change image, the minimum value of the ordinate, and the start time and end time of the environmental sound wave change image, the width and height of the environmental sound wave scanning window are determined; Scanning the environmental sound wave change image along the time coordinate axis using the environmental sound wave scanning window; Calculate the second ordinate mean of each scanned ambient sound wave waveform segment in the ambient sound wave scanning window; Screen out the environmental sound wave waveform segments whose second ordinate mean value exceeds the second preset value as the second target segments; Among them, the height of the ambient sound wave scanning window is H b , width W b ; H b =MAX(and b1 ,and b2 ......,and bn )-MIN(and b1 ,and b2 ......,and bn ); Among them, y b1 ,y b2 ......,y bn , respectively represent the ordinate values ​​of the first sound wave coordinate point, the second sound wave coordinate point, ..., until the nth sound wave coordinate point in the environmental sound wave change image; MAX(y b1 ,y b2 ......,y bn ) represents the maximum value of the ordinate of each sound wave coordinate point in the environmental sound wave change image; MIN(y b1 ,y b2 ......,y bn ) represents the minimum value of the ordinate of each sound wave coordinate point in the environmental sound wave change image; Wherein, t1 represents the horizontal coordinate value of the first sound wave coordinate point in the environmental sound wave change image, t n represents the horizontal coordinate value of the nth sound wave coordinate point in the environmental sound wave change image, t Wb Indicates the reference time width of the environmental sound wave change image, t Bb The time width of the standard sound wave scanning window that represents the image of environmental sound wave changes.

4. The multimodal image processing method according to claim 1, characterized in that: The step of integrating the first time coordinate interval and the second time coordinate interval having temporal continuity into one time coordinate interval comprises: Determine whether each first time coordinate interval and each second time coordinate interval having time continuity have an intersection; If so, performing a union operation on the first time coordinate intervals and the second time coordinate intervals that have intersections to obtain a third time coordinate interval, until each first time coordinate interval and each second time coordinate interval have no intersections; If not, obtain the time coordinate starting value and ending value in the first time coordinate interval and the second time coordinate interval that have time continuity and no intersection, and integrate the starting value and the ending value to obtain a fourth time coordinate interval.

5. The multimodal image processing method according to any one of claims 1 to 4, characterized in that: The step of displaying a segment switching touch key according to the second key segment along the progress bar of the video data to be analyzed comprises: Determine a progress bar interval corresponding to a time coordinate interval of the second key segment; Acquire a sound wave change image display banner preset on the playback interface, wherein the sound wave change image display banner is located above or below the progress bar and extends along the progress bar; Displaying the sound wave change image in the sound wave change image display banner along the progress bar; Marking the progress bar interval on the progress bar; Acquire the starting time coordinates corresponding to each of the second key segments, and retrieve the starting screen image corresponding to each of the second key segments according to the starting coordinates; The starting screen image of the second key segment is displayed along the progress bar, and each starting screen image of the second key segment is associated with a segment switching touch key.

6. The multimodal image processing method according to any one of claims 1 to 4, characterized in that: The step of obtaining a human voice sound wave change image established according to the human voice sound data and the time coordinate axis comprises: Acquire short-time energy, fundamental frequency data, speech rate data and formant data that vary with time in the human voice sound data; Determine the human voice sound wave effect value at the corresponding time point according to the short-time energy, the fundamental frequency data, the speech rate data and the formant data at each time point; Using the time value as the horizontal coordinate and the vertical effect value of the human voice sound wave as the vertical coordinate, a human voice sound wave change image is established; and / or The step of obtaining an environmental sound wave change image established according to the environmental sound data and the time coordinate axis comprises: Acquire amplitude data of the environmental sound data that changes with time; Using the time value as the abscissa and the amplitude data as the ordinate, an environmental sound wave change image is established; The calculation formula of the sound wave effect value of human voice is: y ai represents the vocal sound wave effect value corresponding to the i-th sound wave coordinate point in the vocal sound wave change image, 0 <i≤n; r1, r2, r3 and r4 are the weights corresponding to the short-time energy, fundamental frequency data, speech rate data and formant data respectively, and the value ranges of r1, r2, r3 and r4 are all (0,1), and at the same time, r1+r2+r3+r4=1; x ai 、z ai 、e ai 、f ai Respectively represent the short-time energy, fundamental frequency data, speech rate data and formant data corresponding to the i-th sound wave coordinate point in the human voice sound wave change image; a, b, c, and d represent the standard values ​​of short-time energy, fundamental frequency data, speech rate data, and formant data, respectively; Among them, in the environmental sound wave change image, y bj represents the ambient sound wave amplitude value corresponding to the j-th sound wave coordinate point in the ambient sound wave change image, 0 <j≤n。 7. A multimodal image processing system, characterized in that: For performing the steps of the method according to any one of claims 1 to 6, the multimodal image processing system comprises: An acquisition module, used for acquiring the video data to be analyzed, and acquiring the start time and end time of the video data to be analyzed, so as to establish a time coordinate axis; A sound wave variation image extraction module, used for extracting the sound wave variation image that varies with the time coordinate axis in the video data to be analyzed; A comparison image extraction module, used to extract the time coordinate interval corresponding to the image segment whose change amplitude exceeds the preset range in the sound wave change image, and extract the corresponding screen image from the screen stream of the video data to be analyzed as a comparison image set according to the time coordinate interval; A recognition module, used to compare each picture image in the comparison image set with the standard image through an image recognition unit, so as to extract a target image containing the standard image, and obtain a time coordinate corresponding to the target image; A positioning module, used for locating a first key segment in the video data to be analyzed according to a time coordinate corresponding to the target image; A touch selection module is used to display a segment switching touch button according to the first key segment along the progress bar of the video data to be analyzed.

Images