Method and device for measuring synchronization between speech and image

Abstract

The present application provides a method and device for measuring the synchronization of speech and image, the method comprising: obtaining speech segments and image segments in a video, wherein the speech segments and image segments have a corresponding relationship in the video; converting the speech segments into specific signals, wherein the specific signals are independent of the individual characteristics of the speaker in the speech segments; obtaining the speech features of the specific signals through a speech neural network; obtaining the visual features of the image segments through a visual neural network; and determining whether the speech segments and image segments are synchronized based on the speech features and the visual features. Before inputting the speech segments into the speech neural network, the speech segments are first processed to remove the features related to the individual characters in the speech segments, and then the speech data obtained after processing the speech segments are input into the speech neural network. In this way, the speech features obtained through the speech neural network no longer carry the features of the speaker himself, thereby improving the accuracy of the measurement of the synchronization of speech and image.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a method and device for measuring the synchronization of speech and image. Background Art

[0002] A video usually contains images and voices. When a person in the video speaks, the mouth movement of the person in the image should be synchronized with the voice of the person.

[0003] In order to measure whether the mouth movement of the person in the video is synchronized with the speech he / she makes, SyncNet-type technology is generally used. The so-called SyncNet-type technology can be referred to in the literature Chung, Joon Son, and Andrew Zisserman. "Out of time: automated lip sync in the wild." Asian conference on computer vision. Springer, Cham, 2016. Specifically, the speech clips in the video are input into a neural network to obtain speech features. Then the image clips in the video are input into another neural network to obtain visual features. Finally, by comparing the speech features with the visual features, it is determined whether the mouth movement of the person in the video is synchronized with the speech he / she makes.

[0004] However, the accuracy of using SyncNet-like technology to measure whether the mouth movements of people in the video are synchronized with the speech they produce is still low. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method and device for measuring the synchronization between speech and image, so as to improve the accuracy of the measurement of the synchronization between speech and image.

[0006] In order to solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0007] In a first aspect, the present application provides a method for measuring the synchronization between speech and image, the method comprising: obtaining a speech segment and an image segment in a video, the speech segment and the image segment having a corresponding relationship in the video; converting the speech segment into a specific signal, the specific signal being independent of the individual characteristics of the speaker in the speech segment; obtaining speech features of the specific signal through a speech neural network; obtaining visual features of the image segment through a visual neural network; and determining whether the speech segment and the image segment are synchronized based on the speech features and the visual features, the synchronization being used to characterize that the sound in the speech segment matches the action of the target person in the image segment (specifically, the movement of the lower half of the face).

[0008] According to a second aspect of the present application, there is provided a device for measuring the synchronization between speech and image, the device comprising: a receiving module for acquiring speech segments and image segments in a video, wherein the speech segments and the image segments have a corresponding relationship in the video; a data processing module for converting the speech segments into specific signals, wherein the specific signals are independent of the individual characteristics of the speaker in the speech segments; a feature extraction module for obtaining speech features of the specific signals through a speech neural network; the feature extraction module for obtaining visual features of the image segments through a visual neural network; and a synchronization measurement module for determining whether the speech segments and the image segments are synchronized based on the speech features and the visual features, wherein the synchronization is used to characterize that the sound in the speech segments matches the action of the target person in the image segments.

[0009] The third aspect of the present application provides an electronic device, comprising: a processor, a memory, and a bus; wherein the processor and the memory communicate with each other through the bus; and the processor is used to call program instructions in the memory to execute the method in the first aspect.

[0010] A fourth aspect of the present application provides a computer-readable storage medium, comprising: a stored program; wherein, when the program is executed, the device where the storage medium is located is controlled to execute the method in the first aspect.

[0011] Compared with the prior art, the method for measuring the synchronization of speech and image provided in the first aspect of the present application, after acquiring the speech segment and image segment in the video, first converts the speech segment into a specific signal that is unrelated to the individual characteristics of the speaker therein, then obtains the speech features in the specific signal through the speech neural network, and obtains the visual features of the image segment through the visual neural network, and finally, determines whether the speech segment and the image segment are synchronized based on the speech features and the visual features. That is to say, before the speech segment is input into the speech neural network, the speech segment is first processed to remove the features related to the individual characters in the speech segment, and then the speech data obtained after the speech segment is processed is input into the speech neural network. In this way, the speech features obtained by the speech neural network no longer carry the characteristics of the speaker himself, thereby improving the accuracy of the measurement of the synchronization of speech and image.

[0012] The device for measuring the synchronization of speech and image provided in the second aspect, the electronic device provided in the third aspect, and the computer-readable storage medium provided in the fourth aspect of the present application have the same or similar beneficial effects as the method for measuring the synchronization of speech and image provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0014] Figure 1 The process diagram of the method for measuring the synchronization of speech and image in the embodiment of the present application is as follows: Figure 1 ;

[0015] Figure 2 The process diagram of the method for measuring the synchronization of speech and image in the embodiment of the present application is as follows: Figure 2 ;

[0016] Figure 3 A schematic diagram of a process for processing a voice segment in an embodiment of the present application;

[0017] Figure 4 This is a schematic diagram of the range of the lower half of the face in an embodiment of the present application;

[0018] Figure 5 A schematic diagram of a process for processing image fragments in an embodiment of the present application;

[0019] Figure 6 This is a schematic diagram of an architecture for measuring the synchronization of speech and image in an embodiment of the present application;

[0020] Figure 7 This is a schematic diagram of the architecture of the speech neural network in the embodiment of the present application;

[0021] Figure 8 A schematic diagram of a process for generating speech features in an embodiment of the present application;

[0022] Fig. 9 A schematic diagram of a process for generating visual features in an embodiment of the present application;

[0023] Fig.10 A schematic diagram of the process of training a neural network in an embodiment of the present application;

[0024] Fig.11 A complete flow chart of the method for measuring the synchronization of speech and image in an embodiment of the present application;

[0025] Fig.12 The structure of the device for measuring the synchronization of speech and image in the embodiment of the present application is shown in FIG. Figure 1 ;

[0026] Fig.13 The structure of the device for measuring the synchronization of speech and image in the embodiment of the present application is shown in FIG. Figure 2 ;

[0027] Fig.14 Schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0029] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

[0030] In the prior art, SyncNet-type technology is used to measure whether the mouth movement of a person in a video is synchronized with the speech he or she makes, but the accuracy is low.

[0031] After careful research, the inventors found that the reason why SyncNet-type technologies have low accuracy in measuring whether mouth movements and speech are synchronized is that two neural networks are needed in SyncNet-type technologies. One is a speech neural network, which is used to extract speech features. The other is a visual neural network, which is used to extract visual features. Whether it is a speech neural network or a visual neural network, it is impossible to be independent of the speaker's own characteristics during training. In other words, when using samples for training, the samples carry the speaker's own characteristics, and the trained network also learns the characteristics of the speaker in the sample. For speakers not covered in the sample, the accuracy of the speech features and visual features obtained by the speech neural network and the visual neural network will decrease.

[0032] In view of this, the embodiment of the present application provides a method for measuring the synchronization of speech and image. In this method, before inputting the speech segment into the speech neural network, the speech segment is first processed to remove the features related to the individual person in the speech segment, and then the speech data obtained after the processing of the speech segment is input into the speech neural network. In this way, the speech features obtained by the speech neural network no longer carry the characteristics of the speaker himself, thereby improving the accuracy of the measurement of the synchronization of speech and image.

[0033] Figure 1 The process diagram of the method for measuring the synchronization of speech and image in the embodiment of the present application is as follows: Figure 1 , see Figure 1 As shown, the method may include:

[0034] S101: Acquire voice segments and image segments in a video.

[0035] The video is a video in which it is necessary to determine whether the image and the voice are synchronized. The synchronization here is used to indicate that the sound in the voice segment matches the action of the target person in the image segment.

[0036] Matching means that in a video, the sound produced by the action of the target person in the image segment (specifically, the action of the lower face) is identical to the sound in the voice segment in terms of semantics and time.

[0037] For example, if the mouth of the target person in the image segment makes the shape of a mouth that makes the sound "apple", and the sound in the voice segment is also "apple", then it can be considered that the image segment and the voice segment are synchronized. In addition, if the mouth of the target person in the image segment makes the shape of a mouth that makes the sound "apple", and the sound in the voice segment is "banana", then it can be considered that the image segment and the voice segment are not synchronized.

[0038] Generally speaking, we do not directly put all the images and all the voices in the video together for judgment, but put part of the images and the corresponding voices in the video together for judgment. The selected part of the image is the image segment in the video, and correspondingly, the selected voice is the voice segment in the video. The selected voice segment and the image segment have a corresponding relationship in the video.

[0039] The so-called corresponding relationship means that the selected voice segment and image segment have the same starting time and ending time in the video or have a certain time misalignment (the misalignment is acceptable within the visual range of the human eye).

[0040] For example, the image and voice corresponding to the 1st frame to the 10th frame in the video are obtained. The images of the 1st frame to the 10th frame in the video constitute an image segment, and the voices of the 1st frame to the 10th frame in the video constitute a voice segment. Here, the 1st frame to the 10th frame is a specific position. The specific position for obtaining the image segment and the voice segment can be set according to the actual situation, and no specific limitation is made here.

[0041] Of course, the image segment may also be a certain frame of image, and the corresponding voice segment may also be the voice of the frame and the voices of several frames before and after the frame.

[0042] After determining whether a portion of the images in the video are synchronized with the corresponding voice, determine whether another portion of the images in the video are synchronized with the corresponding voice, until the synchronization determination of all the images in the video and the corresponding voice is completed.

[0043] S102: Convert the speech segment into a specific signal.

[0044] Among them, the specific signal is independent of the individual characteristics of the speaker in the speech segment.

[0045] Before inputting the speech fragment into the speech neural network to obtain speech features, the speech fragment needs to be processed first to delete the individual features of the characters in the speech fragment, that is, to extract semantic features that are irrelevant to the individual features of the characters in the speech from the speech fragment.

[0046] For example, different people have different timbre, intonation, etc. Some people have a thick and solid timbre, while others have a bright and clear timbre. Some people have a soft tone, while others have a high-pitched tone. If a voice segment is directly input into a speech neural network, the obtained speech features will contain the individual characteristics of each person, which will reduce the accuracy of the judgment of speech and speech synchronization. Moreover, if a speech segment containing individual characteristics is input into a speech network for training, the trained network will not be able to accurately obtain the speech characteristics of the person not included in the training sample, thereby reducing the accuracy of the subsequent judgment of speech and speech synchronization. Therefore, before inputting the speech segment into the speech neural network, the speech segment is first converted into a specific signal, and only the semantically related features in the speech segment are extracted, avoiding the extraction of the person's own features, for example: only the speech content itself in the speech segment is extracted, and the timbre is not extracted. In this way, the individual features in the speech segment are deleted, that is, converted into a specific signal, and then the specific signal is input into the speech neural network, and the individual characteristics of the person can be avoided in the obtained speech features, thereby improving the accuracy of the judgment of speech and speech synchronization.

[0047] S103: Acquire speech features of a specific signal through a speech neural network.

[0048] The speech fragments are input into the speech neural network, and the speech fragments are processed by the speech neural network. The output of the speech neural network is the speech feature.

[0049] The speech neural network here can be any neural network that can obtain speech features in a speech segment. The specific type of the speech neural network is not specifically limited here.

[0050] S104: Obtain visual features of the image segment through a visual neural network.

[0051] The contour map obtained after processing the image fragment is input into the visual neural network, and the contour map is processed by the visual neural network. The output of the visual neural network is the visual feature.

[0052] The visual neural network here can be any neural network that can obtain visual features in an image segment. The specific type of the visual neural network is not specifically limited here.

[0053] S105: Determine whether the voice segment and the image segment are synchronized according to the voice features and the visual features.

[0054] After the speech neural network outputs speech features and the visual neural network outputs visual features, the speech features are compared with the visual features through an algorithm with a comparison function. Based on the comparison results, it is possible to determine whether the speech segment and the image segment are synchronized. The synchronization here is used to characterize that the sound in the speech segment matches the action of the target person in the image segment. In other words, based on the comparison results, it is determined whether the meaning of the sound in the speech segment and the action of the target person in the image segment are the same. It can also be understood that the sound produced by the action of the target person in the image segment is the same as the sound in the speech segment in terms of semantics and time. The action of the target person here generally refers to the action of the lower half of the face of the target person, such as the action related to the mouth.

[0055] Generally speaking, the output is a value between 0 and 1. In addition, a threshold is set between 0 and 1. If the output value is greater than or equal to the threshold, it means that the similarity between the speech feature and the visual feature is high, and the speech segment and the image segment are synchronized. If the output value is less than the threshold, it means that the similarity between the speech feature and the visual feature is low, and the speech segment and the image segment are not synchronized. The specific range and threshold of the value are not specifically limited here.

[0056] From the above content, it can be seen that the method for measuring the synchronization of speech and image provided by the embodiment of the present application, after obtaining the speech segment and image segment in the video, first converts the speech segment into a specific signal that is unrelated to the individual characteristics of the speaker therein, and then obtains the speech features in the specific signal through the speech neural network, and obtains the visual features of the image segment through the visual neural network, and finally, determines whether the speech segment and the image segment are synchronized based on the speech features and the visual features. That is to say, before the speech segment is input into the speech neural network, the speech segment is first processed to remove the features related to the individual characters in the speech segment, and then the speech data obtained after the speech segment is processed is input into the speech neural network. In this way, the speech features obtained by the speech neural network no longer carry the characteristics of the speaker himself, thereby improving the accuracy of the measurement of the synchronization of speech and image.

[0057] Further, as Figure 1 As a refinement and extension of the method shown, the embodiment of the present application also provides a method for measuring the synchronization of speech and image. Figure 2 The process diagram of the method for measuring the synchronization of speech and image in the embodiment of the present application is as follows: Figure 2 , see Figure 2 As shown, the method may include:

[0058] S201: Acquire voice segments and image segments in a video.

[0059] The implementation method of step S201 is the same as that of step S101 and will not be repeated here.

[0060] The following will process the speech and image segments before inputting into the neural network from the two aspects of speech and image respectively, and explain in detail the corresponding process of processing into speech data and image data.

[0061] 1. Voice fragment processing

[0062] Since the speech clip contains the speaker's own characteristics, such as timbre, intonation, etc. Therefore, before inputting the speech clip into the speech neural network to obtain the speech features, the speaker's own characteristics in the speech clip are first erased, and then the speech data with the speaker's own characteristics erased is input into the speech neural network, which can improve the accuracy of the synchronization comparison between speech and image.

[0063] S202: Convert the sampling frequency of the speech segment to a specific frequency.

[0064] After separating the voice segments from the video and converting them to a single channel, the sampling frequencies of the voices are different due to the different configurations of the terminals that collect the video. In order to accurately process the voice segments later, the sampling frequencies of the voice segments need to be unified first.

[0065] In practical applications, the sampling frequency of the voice segments can be unified to 16kHz. Of course, the sampling frequency of the voice segments can also be unified to other values, such as 8kHz, 20kHz, etc. The specific value can be set according to the actual situation and is not limited here.

[0066] S203: De-noising the speech segment.

[0067] Here, step S203 may include two aspects.

[0068] S2031: Remove background sound from the voice clip.

[0069] Specifically, the spectral subtraction method in the short-time spectrum estimation can be used to denoise the speech segment to suppress the background sound in the speech segment and highlight the speech in the speech segment. Of course, other methods can also be used to remove the background sound in the speech segment, such as: adaptive filtering technology. As for the specific method used to remove the background sound in the speech segment, it is not limited here.

[0070] S2032: Separate the speech of different speakers in the speech segment to obtain at least one speech sub-segment.

[0071] Sometimes there is not only one person speaking in a voice segment, but multiple people may be speaking at the same time. In this case, it is necessary to separate the voices of different speakers in the voice segment and obtain the voice sub-segments of each speaker.

[0072] After obtaining the speech sub-segments of multiple speakers, sometimes it is only necessary to determine whether the speech of a certain speaker is synchronized with the image, and sometimes it is necessary to determine whether the speech of multiple speakers is synchronized with the image. In this case, according to the actual judgment situation, the speech sub-segment of a certain speaker or the speech sub-segments of several speakers can be selected as the speech segment after denoising.

[0073] S204: Divide the speech segment into a plurality of speech frames by using a sliding weighting method.

[0074] There is overlap between adjacent speech frames.

[0075] Specifically, a window function can be used to slide and weight the speech segment into multiple speech frames. The window function can be a Hamming window function or other types of window functions. The multiple speech frames can be 25ms segments or segments of other lengths. Each segment is called a speech frame. Adjacent speech frames generally maintain an overlap of 10ms. This is because: the speech frame is too short and may not complete a sound. Therefore, maintaining a certain degree of overlap between adjacent speech frames can more fully understand the semantics, thereby improving the accuracy of the measurement of speech and image synchronization.

[0076] It should be noted that the execution order of steps S202, S203, and S204 may not be in the order of the magnitude of the sequence numbers, but may be in any order. The execution order of steps S202, S203, and S204 is not specifically limited here.

[0077] S205: Convert each speech frame into a specific signal.

[0078] Among them, the specific signal is independent of the individual characteristics of the speaker in the speech segment.

[0079] In the prior art, before inputting a speech segment into a speech neural network, the speech segment needs to be converted into a Mel-scale Frequency Cepstral Coefficient (MFCC) signal, and then the MFCC signal is input into the speech neural network to obtain the corresponding speech features. However, the MFCC signal cannot effectively erase the speaker's own features, that is, identity information, in the speech segment, and the obtained speech features will also contain the speaker's identity information, thereby reducing the accuracy of the measurement of speech and image synchronization.

[0080] In view of this, before inputting the speech segment into the speech neural network, the speech segment can be converted into a specific signal. The specific signal here has nothing to do with the speaker's own characteristics in the speech segment, that is, it can better erase the speaker's own characteristics in the speech segment. In this way, when the specific signal is input into the speech neural network, the speech features obtained no longer contain the speaker's own characteristics, thereby improving the accuracy of the measurement of speech and image synchronization.

[0081] In practical applications, the specific signal may be a phonetic posterior gram (PPG) signal. The PPG signal can better erase information related to the speaker's identity in the voice segment. In addition, the PPG signal can further erase the background sound in the voice segment, reduce the variance of the voice neural network input, and thus improve the accuracy of the voice and image synchronization measurement.

[0082] Of course, the speech fragments can also be converted into other types of signals, such as the features extracted by the DeepSpeech model, as long as the speaker's identity information can be erased. There is no limitation on the specific type of specific signal.

[0083] In practical applications, in order to convert a speech segment into a PPG signal, the speech segment can be input into a speaker-independent automatic speech recognition (SI-ASR) system, and the speech segment is processed by the SI-ASR system to generate a PPG signal. In the SI-ASR system, the specific dimension P of the PPG signal and the number of phonemes supported by SI-ASR are related to the supported language. Here, an SI-ASR system that supports Chinese and English is used, which supports a total of P=400 phonemes. The PPG signal obtained from a speech frame is a 1×400-dimensional feature vector. The PPG signal obtained from T consecutive speech frames is a T×400-dimensional feature matrix. When using other SI-ASR systems, corresponding adjustments can be made according to the number of supported phonemes.

[0084] Of course, other methods can be used to convert speech fragments into signals that erase the speaker's identity information, such as the deep learning model DeepSpeech. This deep learning model can convert speech signals into corresponding text. Therefore, among the features extracted by DeepSpeech, there is only the content of the speech itself, and no personal features such as the speaker's timbre. In this way, after extraction, the speaker's identity information and background and other content that is irrelevant to semantics can also be erased.

[0085] Figure 3 This is a flow chart of processing voice clips in the embodiment of the present application, see Figure 3As shown, first, the speech is input into the preprocessing module. In the preprocessing module, the speech can be processed by uniform sampling frequency, denoising, segmentation, etc. Then, the speech data obtained after processing the speech segment is input into the SI-ASR system. In the SI-ASR system, the speech segment can be converted into a PPG signal.

[0086] 2. Image fragment processing

[0087] Since the image fragment contains the speaker's own features, such as the thickness of lips, the size of mouth, etc. Therefore, before inputting the image fragment into the image neural network to obtain the image features, the speaker's own features in the image fragment are first erased, and then the image data with the speaker's own features erased is input into the image neural network, which can improve the accuracy of the synchronization comparison between speech and image.

[0088] The following takes the extraction of lower half face features from an image segment as an example to illustrate the generation of a target person's contour map based on an image segment. The extracted contour map here is irrelevant to the individual features of the target person.

[0089] S206: Perform face detection on the image segment to obtain a face detection frame.

[0090] Generally speaking, face detection is performed on each frame of image to obtain a face detection frame.

[0091] S207: Horizontally align the faces in the face detection frame.

[0092] Specifically, the dense face alignment algorithm can be used to find the positions of the key points of the face in the face detection frame in the original image, including but not limited to the center of the left eye, the center of the right eye, the left corner of the mouth, and the right corner of the mouth. The left and right mentioned above are the left and right of the face in the image in the physiological sense, not the left and right in the image, and it is assumed that the face in the image is frontal. Using the position information of the key points of the face, the face image is processed into a form that conforms to the rules based on rule calculation. The rules here can be as follows:

[0093] Calculate the middle position between the left eye center key point and the right eye center key point, recorded as P_eyecentre;

[0094] Calculate the middle position between the left mouth corner key point and the right mouth corner key point, recorded as P_mouthcentre;

[0095] Calculate the vector from the left eye center key point to the right eye center key point, recorded as V_eyetoeye;

[0096] Calculate the vector from P_eyecentre to P_mouthcentre and rotate it 90 degrees counterclockwise to make it form an acute angle with V_eyetoeye, recorded as V_eyetomouth;

[0097] Calculate the vector difference between V_eyetoeye and V_eyetomouth, and normalize the vector difference to obtain the unit vector X_unit;

[0098] Enlarge X_unit to the larger value of 2 times the modulus length of V_eyetoeye and 1.8 times the modulus length of V_eyetomouth to obtain vector X, and rotate X 90 degrees counterclockwise to obtain vector Y;

[0099] With P_eyecentre moved 0.1 times V_eyetomouth as the center C, a rectangle can be obtained in the image, the coordinates of the upper left corner of the rectangle are C+X+Y, and the coordinates of the lower right corner are CXY;

[0100] The image in the above rectangle is taken out by using an interpolation algorithm and scaled to a predetermined size, such as 256*256 pixels, to obtain the aligned face.

[0101] The dense face alignment algorithm used to find the key points of the face here can be a three-dimensional dense face alignment (3D Dense Face Alignment, 3DDFA) algorithm. Of course, other alignment algorithms can also be used to obtain the key points of the face, and then the above rules are used to achieve face alignment. The specific algorithm used here is not limited.

[0102] Compared with the more commonly used method of aligning faces by calculating the affine transformation between facial key points and a preset frontal facial key point template, this method is compatible with the alignment of large-angle side faces and front faces.

[0103] S208: Extracting the expression coefficient of the target person from the face.

[0104] Specifically, the expression coefficient of the target person in the face detection frame can be extracted by a three-dimensional deformable parametric face model parameter estimation algorithm (3DMM), and the expression coefficient meets the standard of the three-dimensional deformable parametric face model.

[0105] After taking the content in the face detection frame as input and processing it using the 3DMM parameter estimation algorithm, the identity coefficient and expression coefficient of the target person that meet the 3DMM model standard can be obtained. The expression coefficient can be denoted as α exp .

[0106] Among them, the 3DMM parameter estimation algorithm is an algorithm that can estimate 3DMM parameters, which is used to estimate the identity coefficient and expression coefficient of the face, and the identity coefficient and expression coefficient meet the standards defined by 3DMM.

[0107] Specifically, the 3DMM parameter estimation algorithm used in this application is implemented using a deep neural network model. A pre-trained deep neural network model can be used to input the face image in the aligned face detection frame and the identity coefficient corresponding to the existing target person into the model, extract the expression coefficient and identity coefficient of the target person in the aligned face image, and update the identity coefficient corresponding to the existing target person according to the output identity coefficient for subsequent image frame estimation. Here, the identity coefficient corresponding to the target person is the sliding weighted average of the identity coefficients estimated in adjacent image frames in time sequence.

[0108] Compared with directly calculating the expression coefficient of the target person from the aligned face image alone, here by inputting the calculation results of the identity coefficient of the target person in adjacent image frames in time sequence into the deep neural network model, the model can better use the expression coefficient instead of changing the identity coefficient to fit the morphological changes of the face, thereby obtaining a more accurate expression coefficient.

[0109] Similarly, other 3DMM parameter estimation algorithms that can stabilize the identity coefficient can also be used here, such as the Face2Face algorithm (Thies, Justus, et al. "Face2face: Real-time face capture and reenactment of rgb videos." Proceedings of the IEEE conference on computervision and pattern recognition. 2016.) to obtain the expression coefficient of each frame.

[0110] Expression coefficient α exp It includes features that are not related to the individual speaker, such as the position of the mouth and the degree of mouth opening. The features related to the individual speaker are represented in the identity coefficient. Therefore, only based on the expression coefficient α exp and the standard identity coefficient (the standard identity coefficient is used here to replace the identity coefficient of the target person and remove the individual characteristics of the target person), and the universal parameterized face model is input to generate the face contour map of the target person, which can eliminate the individual characteristics of the speaker and thus improve the accuracy of the measurement of mouth movement and speech synchronization.

[0111] S209: extracting the lower face expression coefficient corresponding to the lower face in the expression coefficient.

[0112] Under the definition of 3DMM, all expression coefficients have an impact on the whole face, but some have a greater impact on the mouth and a negligible impact on the eyes. Therefore, the expression coefficients with a high correlation with the movement of the lower half of the face are extracted as the expression coefficients of the lower half of the face.

[0113] If we need to measure the synchronization between a certain part of the target person in the image and the voice, we extract the coefficient of the target person's part that is not related to individual characteristics from the system. Here, we need to measure the synchronization between the lower face movement and the voice, so we extract the expression coefficient of the lower face from the expression coefficient, which is recorded as α halfface , and then generate a lower face contour map based on the expression coefficient of the lower face to measure the synchronization with the speech.

[0114] S210: Inputting the expression coefficient of the lower face into the general three-dimensional face model to obtain a three-dimensional face model corresponding to the lower face of the target person.

[0115] The three-dimensional face model corresponding to the lower half of the target person's face is a three-dimensional face model of the target person's lower half face expression coefficient combined with the standard identity coefficient.

[0116] The general 3D face model is an abstract face model. In the general 3D face model, the data of eyebrows, eyes, nose, face, mouth and other parts are obtained by averaging many faces and are universal.

[0117] After inputting the lower face expression coefficient into the general three-dimensional face model, the three-dimensional face model corresponding to the lower face of the target person is obtained.

[0118] Specifically, in the general 3D face model, the predefined complete expression orthogonal basis B exp The corresponding B is changed to the B related to the movement of the lower face. halfface The specific formula is as follows:

[0119]

[0120] Among them, S is the geometric model of the target person’s mouth shape under neutral expression. is the average face geometry model corresponding to the predefined neutral expression, B halfface is the orthogonal basis associated with mouth movement, α halfface is the expression coefficient of the lower face.

[0121] In this way, the obtained three-dimensional face model corresponding to the expression of the lower half of the face of the target person can eliminate the influence of irrelevant expressions.

[0122] S211: Obtain a vertex set of the lower half of the face in the three-dimensional face model.

[0123] The so-called lower face refers to the facial area below the line connecting the bottom of the left and right ears and the tip of the nose. Figure 4 This is a schematic diagram of the range of the lower half of the face in the embodiment of the present application, see Figure 4As shown, the position 401 of the bottom of the left ear, the position 402 of the nose tip, and the position 404 of the bottom of the right ear are connected to obtain a line 404. The line 404 divides the face into an upper half and a lower half. The face below the line 404 is the lower half.

[0124] When the lower half of the face is selected, the line 604 can be adjusted to a certain extent, such as moving upward to the eye position, or moving downward to the nose position, etc. That is, the selection of the lower half of the face can be adjusted according to actual needs.

[0125] S212: Projecting the vertex set onto a two-dimensional plane to obtain a lower face contour map of the target person, and using the lower face contour map as a face contour map of the target person.

[0126] Specifically, the vertices corresponding to the mouth contour and chin area on the obtained geometric model S are collected to obtain a vertex set V. Then, the vertex set V is projected onto a two-dimensional plane using scale orthographic projections to obtain a contour map I of the lower half of the face, as shown in the following formula (2):

[0127]

[0128] Where I is the two-dimensional contour map of the lower half of the target person's face, f is the scale factor, P is the orthogonal projection matrix, and S(v) is the vertex set of the lower half of the face in the three-dimensional face model. Here, the size of the contour map I can be a rectangle of 128×256, with the mouth and the contours of the lower half of the face centered. In particular, to enhance the visibility of the contour map, when projecting, each vertex is projected into a two-dimensional Gaussian circular spot with a radius of r pixels and a vertex projection position as the center. The value of the radius r is positively correlated with the size of I. Corresponding to I of 128×256, r=2 pixels is taken here.

[0129] In the process of processing the image fragment, the model is not rotated or translated. Instead, the expression coefficient of the target person in the image fragment is directly obtained through 3DMM, and then the universal three-dimensional face model is obtained by combining the standard identity coefficient to generate a lower half face contour map that eliminates the individual features of the target person. The obtained contour map is a contour map under the frontal face features, which eliminates the influence of facial posture and occlusions in the original image.

[0130] Figure 5 This is a schematic diagram of the process of processing image fragments in the embodiment of the present application, see Figure 5 As shown, first, the image is densely aligned to obtain the aligned image; then, the expression coefficient of the face 3D model is extracted from the aligned image; then, a frontal perspective and a standard face shape are used to generate a 3D model according to the extracted expression coefficients; finally, the 3D model is projected at the corresponding vertices to obtain the two-dimensional contour of the lower half of the face.

[0131] After processing the speech segment into a PPG signal and the image segment into a two-dimensional contour map of the lower half of the front face, the PPG signal can be input into the speech neural network, and the two-dimensional contour map can be input into the visual neural network to obtain speech features and visual features respectively, and then the speech features are compared with the visual features to determine whether the speech segment and the image segment are synchronized.

[0132] S213: Acquire speech features of a specific signal through a speech neural network.

[0133] S214: Obtain visual features of the face contour map through a visual neural network.

[0134] S215: Determine whether the voice segment and the image segment are synchronized according to the voice features and the visual features.

[0135] Steps S213, S214, and S215 are implemented in the same manner as the aforementioned steps S103, S104, and S105, and will not be repeated here.

[0136] Figure 6 This is a schematic diagram of the architecture for measuring the synchronization of voice and image in the embodiment of the present application, see Figure 6 As shown, after respectively extracting the voice segment and the image segment from the video, on the one hand, the voice segment is input into the voice neural network to obtain the voice features. On the other hand, the image segment is input into the visual neural network to obtain the visual features. Finally, the voice features and the visual features are input into the synchronization measurement module, and the synchronization measurement module determines whether the corresponding voice segment and the image segment are synchronized through the voice features and the visual features. The synchronization measurement module here is a module that determines whether the corresponding voice segment and the image segment are synchronized by comparing the voice features with the visual features. The specific form of the synchronization measurement module is not limited here.

[0137] In practical applications, in order to obtain the speech features of a speech segment, the speech segment can be input into a speech neural network for processing to obtain the speech features. And in order to obtain the visual features of an image segment, the image segment can be input into a visual neural network for processing to obtain the visual features. The following is an explanation of the three aspects of neural network construction, training data sampling, and training.

[0138] 1. Neural Network Construction

[0139] 1. Construction of speech neural network

[0140] Before the speech fragment is input into the speech neural network, it has been converted into a specific signal, specifically a PPG signal with a dimension of T×P. And each dimension has a clear physical meaning, P is the number of phonemes, T is the number of sampling times in time, and each column is the posterior probability distribution of the phonemes corresponding to a speech frame. Based on these clear physical meanings, the speech neural network can be specifically constructed as follows.

[0141] Figure 7 This is a schematic diagram of the architecture of the speech neural network in the embodiment of this application, see Figure 7 As shown, the speech neural network at least includes: convolution layer (Conv1D(3×1,stride=(2,1))LeakyReLU(0.02)), ...convolution layer (Conv1D(3×1,stride=(2,1))LeakyReLU(0.02)), reorganization layer (Reshape), fully connected layer (FullyConnection Layer LeakyReLU(0.02)), fully connected layer (Fully Connection Layer LeakyReLU(0.02)), fully connected layer (Fully Connection Layer LeakyReLU(0.02)), linear projection layer (LinearProjection Layer).

[0142] Taking into account the overlap between adjacent speech segments, multiple 1D convolutional layers (with a convolution kernel size of 3×1, a convolution step of (2, 1), and valid padding) are first used to process the time dimension. The obtained matrix is ​​then reorganized into a feature vector. Next, three fully connected layers are used to process the feature vector. Finally, a 512-dimensional speech feature vector is obtained after one linear projection layer. Among them, the number of convolutional layers is related to the duration of the input specific signal (the feature matrix corresponding to the PPG signal). The dimension of the final output speech feature vector is consistent with the dimension of the subsequent output visual feature vector. The speech feature vector in the embodiment of the present application is also the speech feature, and the visual feature vector is also the visual feature.

[0143] Specifically, when P = 400, input duration = 200ms, T = 13, the PPG feature matrix is ​​13 × 400 dimensions. Correspondingly, two layers of 1D convolutional layers can be used to obtain a 3 × 400 feature matrix. After being reorganized into a 1 × 1200 feature vector, it passes through three fully connected layers and one linear layer to obtain the final 512-dimensional speech feature vector.

[0144] Figure 8 This is a flow chart of generating speech features in the embodiment of the present application, see Figure 8As shown, the process may include:

[0145] S801: Use multiple 1-dimensional convolutional layers to process the specific signal in the time dimension to obtain a feature matrix.

[0146] Among them, the number of 1D convolutional layers is related to the duration corresponding to a specific signal.

[0147] S802: Reorganize the feature matrix into a feature vector.

[0148] S803: Use three fully connected layers and one linear projection layer to process the feature vector to obtain a 512-dimensional speech feature vector.

[0149] Of course, the dimension of the final speech feature is not limited to 512 dimensions. The dimension of the speech feature is related to the amount of speech data input into the model and the type of loss function used by the speech neural network.

[0150] 2. Visual neural network construction

[0151] Because the factors that interfere with the movement information of the lower half of the face in the image fragment (such as lighting, expression, posture, etc.) have been largely removed before the image fragment is input into the visual neural network, the visual neural network can adopt a network structure with lighter computational complexity.

[0152] Specifically, the visual neural network can use the backbone network of ResNet18 and make the following changes:

[0153] (1) If the input image segment is multiple images, the multiple images can be arranged in time increasing order along the channel dimension and used as the input of the visual neural network. Therefore, the parameter dimension of the convolution in the first layer of the visual neural network needs to be adjusted accordingly.

[0154] (2) Since the image segment is processed as a profile of the lower half of the face, the resolution is 128×256 and the aspect ratio is 1:2, which is different from the default input aspect ratio of ResNet18 of 1:1. To this end, it is necessary to use a larger convolution kernel size in the first convolution layer of ResNet18, for example: 7×7, and set the convolution step size to (1, 2).

[0155] The above convolution size and step size are only specific values, which are not intended to limit the convolution kernel size and step size used in the embodiments of the present application to only 7×7 and (1, 2). In practical applications, the convolution kernel size and step size of the convolution layer are related to the size of the contour map. The corresponding step size can be set according to the aspect ratio of the contour map, and the size of the convolution kernel can be set slightly larger. In this way, using a convolution layer with a larger convolution kernel can convolute the contour map. Figure 1Of course, it is also possible to use multiple convolution layers with smaller convolution kernels to perform multiple processing.

[0156] (3) A fully connected layer is added to the end of the ResNet18 backbone network, so that a 512-dimensional visual feature vector can be obtained.

[0157] Of course, the dimension of the final visual feature vector is not limited to 512 dimensions. The dimension of the visual feature vector is related to the amount of visual data input into the model and the type of loss function used by the visual neural network.

[0158] Of course, in addition to using the ResNet18 backbone network, the visual neural network can also use other deep neural networks after modification, such as MobilenetV2.

[0159] Fig. 9 This is a schematic diagram of the process of generating visual features in the embodiment of the present application, see Fig. 9 As shown, the process may include:

[0160] S901: Use a convolutional layer to process the contour image to obtain a feature matrix.

[0161] Among them, the convolution kernel size and step size of the convolution layer are related to the size of the contour map.

[0162] S902: Use the backbone network of the visual neural network to process the feature matrix to obtain a feature vector.

[0163] The backbone network here refers to the main architecture in the neural network. In order to construct the visual neural network in the embodiment of the present application, after obtaining an existing visual neural network, the architecture in the existing visual neural network, i.e., the backbone network, is adopted, and the parameters in certain layers are adaptively modified to obtain the visual neural network in the embodiment of the present application.

[0164] S903: Use a fully connected layer to process the feature vector to obtain a 512-dimensional visual feature vector.

[0165] Of course, the dimension of the final visual feature is not limited to 512 dimensions. The dimension of the visual feature is related to the amount of contour data input into the model and the type of loss function used by the visual neural network.

[0166] 2. Training Data Sampling

[0167] For training videos, a portrait video of a single person speaking is used. In the portrait video, the interference degree of the background sound is less than a certain degree. In other words, a video of a single person speaking with relatively clean background sound is needed. In addition, there can be a large number of training videos so that subsequent training can be more sufficient. In practical applications, a 25Hz high-definition video can be used. In this way, the accuracy of visual feature extraction training can be improved.

[0168] After the training video is collected, the audio signal in each video is first processed into 16kHz, and the video signal is cut into frames, and the timeline is recorded. In this way, voice segments and image segments are obtained. Then, the voice segments are processed using the processing method in the above steps S202-S205 to obtain a specific signal, which is referred to as voice in subsequent sampling, and the image segments are processed using the processing method in the above steps S206-S212 to obtain a face contour map, which is referred to as vision in subsequent sampling.

[0169] Next, we can formally sample the training data. Here, it mainly includes positive sample sampling and negative sample sampling. The so-called positive sample means that the input speech and vision are synchronized. The so-called negative sample means that the input speech and vision are not synchronized. By inputting positive samples and negative samples for training, the accuracy of measuring the synchronization of speech and image can be improved.

[0170] 1. Positive Sample Sampling

[0171] The so-called positive samples mean that the speech and vision used in training need to come from the same training video and be synchronized in time.

[0172] Moreover, if the speech length is too short, a complete pronunciation may not be included in the speech, and it may even affect the understanding of the semantics in the speech. In view of this, in order to improve the accuracy of speech feature recognition and thus improve the accuracy of synchronization measurement, the speech frame length can be made larger than the visual frame length. The specific selection of the speech frame length can be determined based on the frame rate of the training video.

[0173] For example, for a training video with a frame rate of 25Hz, you can select a frame of image at time T and a speech segment of (T-20ms, T+20ms), and after processing, form a positive sample pair. At this time, the visual length is 1 frame, and the speech length is 40ms. Obviously, this is to make the speech frame length longer than the visual frame length. The speech length is set to 40ms to match the 25Hz frame rate in the training video. If a training video with other frame rates is used, the speech length can be adjusted accordingly.

[0174] A training video is selected from the training video set, referred to as the first training video; another training video is selected from the training video set, referred to as the second training video. The first training video and the second training video are different training videos.

[0175] In the embodiment of the present application, the first image segment and the first voice segment obtained from the first training video are processed into the first image data and the first voice data to form the positive sample.

[0176] 2. Negative Sample Sampling

[0177] Negative samples are when the speech and vision used in training are not synchronized. There are many kinds of asynchrony here. In order to conduct more comprehensive training, samples can be collected for all asynchrony situations.

[0178] In the existing method, when collecting negative sample pairs, image segments and voice segments are collected from different videos, or image segments and voice segments are collected from different time points in the same video to form negative samples. However, there may still be positive samples in the negative sample pairs collected in this way. For example: if the voice segment in video A is the same as the voice segment in video B, then the image segments corresponding to the voice segment in video A and the voice segment in video B are also synchronized. If the image segments corresponding to the voice segment in video A and the voice segment in video B are combined to form negative samples, in fact, the above two constitute positive samples. For another example: if the voice corresponding to a certain image segment in video A is silent, and the voice corresponding to another image segment in video B is also silent, if the image segment in video A and the voice segment corresponding to the image segment in video B are combined, in fact, they constitute positive samples. In this way, unreasonable negative samples appear in the negative sample pairs, thereby reducing the accuracy of neural network training and the accuracy of subsequent synchronization measurement.

[0179] In view of this, in the embodiment of the present application, when collecting negative samples, it is necessary to remove unreasonable negative samples, that is, to clean the training database and remove negative samples that are not suitable for training. In this way, the accuracy of negative samples can be improved, thereby improving the accuracy of neural network training, and thereby improving the accuracy of speech and image synchronization measurement.

[0180] Specifically, negative sample sampling can be performed in the following three ways.

[0181] (1) Misplaced negative samples

[0182] The so-called misaligned negative samples mean that although the speech and vision come from the same training video, the speech and vision are not synchronized in time, that is, there is a small amount of misalignment.

[0183] For example, a frame of image at time T and a speech segment of (Tt-20ms, T-t+20ms) are collected and processed to form a negative sample pair. That is, the image segment is processed into image data, and the speech segment is processed into speech data, and then the sample pair <speech data, image data> is constructed, which is abbreviated as <speech, vision>.

[0184] For example, for misaligned negative samples: <speech, vision> negative samples are taken from the same video, with a small timeline misalignment. A frame of image at time T and a speech segment (Tt-20ms, T-t+20ms) form a negative sample pair, where |t|>80ms. That is, the speech and vision need to be misaligned by at least 80ms, corresponding to the time length of two frames of images, to be considered as a negative sample pair. And ensure that the speech segment (T-20ms, T+20ms) and the speech segment (Tt-20ms, T-t+20ms) have different semantics.

[0185] Specifically, when constructing misaligned negative samples, the misalignment duration between speech and vision needs to be greater than or equal to twice the visual duration. This ensures that the speech in the misaligned negative samples is completely misaligned with the speech in the misaligned negative samples that is synchronized with the visual counterpart, thereby ensuring the accuracy of subsequent training.

[0186] If training videos with other frame rates are used, the frame length of the speech can be adjusted accordingly, and the frame length of the vision can also be adjusted accordingly.

[0187] In addition, in order to further improve the accuracy of subsequent training, it is also necessary to ensure that the speech in the misaligned negative samples is semantically different from the speech in the misaligned negative samples that is synchronized with the visual counterpart.

[0188] In the embodiment of the present application, the first image segment and the second voice segment obtained from the first training video are processed into the first image data and the second voice data to form the first negative sample.

[0189] (2) Negative samples with fixed speech

[0190] The so-called speech-fixed negative sample means that the speech is extracted from the same training video, while the vision is randomly extracted from another training video other than this training video. The speech in the other training video is semantically different from the speech extracted from the same training video.

[0191] For example, for a fixed speech segment negative sample: <speech, vision> negative samples are collected from different videos, where the speech segment is fixed, and a frame of image is randomly sampled from other training videos to form a negative sample pair. Among them, ensure that the speech in the negative sample pair and the speech in the positive sample pair to which the vision belongs have different semantics. If the semantics of the speech in the negative sample pair is "silent", the semantics of the speech in the positive sample pair to which the vision belongs in the negative sample pair cannot be "silent".

[0192] In the embodiment of the present application, the first image segment obtained from the first training video and the random voice segment obtained from the second training video are processed into the first image data and the third voice data to form the second negative sample.

[0193] (3) Visually fixed negative samples

[0194] The so-called visual fixed negative sample means that the visual is extracted from the same training video, while the speech is randomly extracted from another training video. The visual in the other training video and the visual extracted from the training video have different movements of the lower half of the face of the person in the image.

[0195] For example, for fixed visual frame negative samples: <speech, vision> negative samples are taken from different videos, where the video frame is fixed and a speech clip is randomly sampled from other videos to form a negative sample pair. It is ensured that the video frame in the negative sample pair and the visual image in the positive sample pair to which the speech clip belongs have sufficient difference in the movement of the lower half of the face.

[0196] In the embodiment of the present application, the second voice segment obtained from the first training video and the random image segment obtained from the second training video are processed into the second voice data and the second image data to form the third negative sample.

[0197] The first image segment and the random image segment are both images at one or more consecutive time points.

[0198] In addition, in practical applications, considering that a single frame image has no contextual information and thus cannot fully express the movement information of the lower half of the face of the person in the image, when sampling, images at consecutive T time points can be collected to obtain vision, and speech segments corresponding to images at T time points can be collected to obtain speech, and then the obtained vision and speech are processed to form sample pairs and input into the neural network for training. Generally speaking, T can be set to 5, and the corresponding speech segment is 200ms.

[0199] After obtaining the above three candidate negative samples, visual rule judgment and speech rule judgment need to be performed on these three candidate negative samples, and only candidate negative samples that pass both judgments are retained as qualified negative samples. The specific judgment process is as follows:

[0200] 1) Phonetic rule determination:

[0201] In the judgment negative sample <speech a, vision v>, the speech a and the speech a_positive in the positive sample pair to which the vision v belongs need to be different in semantics.

[0202] Specifically, the core idea is to measure the differences between PPG feature sequences.

[0203] Since the speech samples have been processed into PPG feature sequences, each PPG feature is the posterior probability distribution of the phonemes contained in the corresponding speech frame. Therefore, by taking the maximum probability of the posterior probability distribution, the phonemes corresponding to the speech frame can be obtained, so that the PPG feature sequence can be converted into a phoneme sequence P = [p0…p i …p t ].

[0204] After obtaining the phoneme sequence of speech a in the negative sample and the corresponding speech a_positive in the positive sample, the edit distance between the two phoneme sequences is calculated. Specifically, the edit distance D = L (P1, P2) between speech P1 in the negative sample and speech P2 in the corresponding positive sample can be calculated using Levenshtein Distance. That is, how many steps are required to transform P1 into P2 through deletion, insertion, and substitution operations, and the more similar the sequences, the fewer the steps required. When the value of D is lower than a preset threshold, the two speech samples are judged to be too similar; when the value of D is higher than the preset threshold, the two speech samples are judged to be sufficiently different. The preset threshold can be obtained by statistics from a database.

[0205] 2) Visual rule determination:

[0206] It is determined that in the negative sample <speech a, visual v>, the visual v and the visual v_positive in the positive sample pair to which speech a belongs are sufficiently different in the movement of the lower half of the face.

[0207] Specifically, the core idea is to determine the similarity between the vision in the negative sample and the corresponding positive sample.

[0208] Since the above two visual samples have been preprocessed and processed into the contour map of the lower half of the face, and because they use the same standard identity information and projection coordinate system, they are already aligned. Therefore, the threshold can be used to convert the two contour maps from 0 to 255 grayscale images to 0 / 1 binary contour maps, denoted as M v1 and Mv2 。

[0209] For the lower half of the face, then calculate the absolute difference D1 = ∑|M v1 - M v2 | between two binary contour maps, and calculate the structural similarity (SSIM) D2 = SSIM(M v1 , M v2 ). Then, obtain the weighted sum D = λ1D1 + λ2D2 of the two. When the value of D is lower than a preset threshold, it is determined that the two visual samples are too similar; when the value of D is higher than the preset threshold, it is determined that the two visual samples have sufficient differences. The weights λ1, λ2 and the preset threshold can be obtained by statistical analysis of a database.

[0210] The above double determination of visual rule determination and speech rule determination is very important because for many words with different pronunciations, the mouth movements are very similar. For example, for the character "育" in "体育" (physical education) and the character "出" in "出门" (go out), both will pout the lips once. Therefore, only the negative samples that pass the double determination belong to reasonable negative samples and can be used to train the neural network subsequently. In this way, the accuracy of neural network training can be improved, and further the accuracy of measuring the synchronization of speech and images can be improved.

[0211] After screening these three candidate negative samples, the first negative sample, the second negative sample, and the third negative sample are obtained, and then the first negative sample, the second negative sample, and the third negative sample are used for neural network training.

[0212] III. Neural Network Training

[0213] Based on Figure 6 the architecture diagram shown, input the positive samples, the first negative sample, the second negative sample, and the third negative sample collected above into the speech and image synchronization measurement model for training, and the various parameters in the speech and image synchronization measurement model can be adjusted, so as to more accurately measure the synchronization of speech and images.

[0214] Here, the speech and image synchronization measurement model is composed of a speech neural network, a visual neural network, and a synchronization measurement model.

[0215] Fig.10 is a schematic flowchart of training the neural network in the embodiment of the present application. As shown in Fig.10 , this process can include two stages: the pre-training stage and the post-training stage, which are specifically as follows:

[0216] 1. Pre-training stage

[0217] S1001: The positive samples, the first negative samples, the second negative samples and the third negative samples are divided into different batches and input into the speech and image synchronization measurement model for training, and the parameters in the speech and image synchronization measurement model are adjusted. The number of positive samples and the number of negative samples in each batch are similar by balancing the sampling, which is helpful for model training.

[0218] Specifically, the parameters in the speech and image synchronization measurement model can be adjusted through the loss function, and the loss function is specifically shown in the following formula (3):

[0219]

[0220] Among them, L represents the loss value, N represents the batch size, n represents the sample number, and y n Represents the label of the sample, y n =1 indicates a positive sample, y n =0 indicates negative samples, d p represents the positive sample distance, d n represents the negative sample distance, v represents the visual features extracted by the visual neural network, a represents the speech features extracted by the speech neural network, and margin1 is a specific value. The margin1 here can be different from the margin2 in the later stage of training.

[0221] The specific method of adjusting the parameters in the model based on the loss function value, for example, can be: the model can be trained by using the Adam optimization algorithm, and the corresponding parameters are beta_1=0.99, beta_2=0.999. In the early stage of training, the batch size is set to 256, 1000 epochs are trained, and the learning rate is initially set to 0.005, and the learning rate is gradually decayed to 0 after 100 epochs using the cosine decay strategy. Similarly, in the late stage of training, 500 epochs are trained, the learning rate is initially set to 0.001, and the learning rate is gradually decayed to 0 after 100 epochs using the cosine decay strategy. The above-mentioned specific training parameters and model parameters need to be adjusted accordingly as the database changes. Of course, other specific methods can also be used, which are not limited here.

[0222] 2. Late training

[0223] In order to further optimize the speech and image synchronization measurement model, after the model is trained in the early stage, the online difficult sample mining strategy can continue to be used in each training batch, and the model can be trained again using the difficult samples mined online until the trained model is in a certain accuracy range and no longer produces large fluctuations.

[0224] Specifically, unlike the early stage of training, in the late stage of training, all positive samples are only divided into different batches (such as M batches), and negative samples are obtained online by combining different positive samples in the batch, which are called negative samples in the batch. The positive samples and negative samples in each batch are sorted according to the rules and the loss value output by the loss function; the difficult positive samples in the positive samples in the current batch are obtained according to the loss value; and multiple difficult negative samples in the negative samples in the current batch are obtained according to the loss value.

[0225] S1002: Obtain difficult positive samples from a batch of positive samples.

[0226] Specifically, after randomly sampling N positive samples (speech and vision) in the training set to form a training batch, the speech feature a is extracted through the current speech neural network and visual neural network respectively. i and visual features v i , i∈N. Then, find the hard positive samples in each batch. The hard positive samples are specifically shown in the following formula (4):

[0227]

[0228] in, represents a difficult positive sample, v represents the visual features extracted by the visual neural network, a represents the speech features extracted by the speech neural network, and N represents the label of the sample.

[0229] S1003: Obtain negative samples in the batch and multiple difficult negative samples in the negative samples.

[0230] Specifically, negative samples in the batch are generated according to the positive samples in the batch, and a plurality of difficult negative samples in the negative samples in the batch are obtained.

[0231] Among them, negative samples in this batch are generated according to the positive samples in each batch. Specifically, the N speech features and N visual features obtained from the N positive samples in the training batch in step S1202 are combined in pairs to form an N×N matrix. The positive sample combinations on the diagonal are excluded, and N×(N-1) combinations are obtained as candidate negative samples. After visual rule judgment and speech rule judgment, the qualified negative samples obtained are the negative samples in the batch.

[0232] Among them, multiple negative samples correspond to each sample in the positive sample. That is to say, each positive sample in step S1002 corresponds to multiple negative samples. Step S1003 is to find a difficult negative sample in the multiple negative samples corresponding to each positive sample.

[0233] Among them, multiple difficult negative samples are obtained from the negative samples in each batch, specifically for the speech feature a iThe corresponding negative samples are sorted according to the loss value output by the loss function, and the speech feature a is obtained according to the loss value. i corresponding hard negative samples; and / or visual features v i The corresponding negative samples are sorted according to the loss value output by the loss function, and the visual features v are obtained according to the loss value. i The corresponding hard negative samples.

[0234] For example, assuming there are 3 positive samples, we can form a 3×3 matrix. Excluding the positive sample combinations on the diagonal, there are 6 candidate negative samples, namely After removing the unqualified negative samples in the matrix, all the remaining negative samples in the matrix are qualified negative samples. Each i-th row in the matrix is ​​the negative sample corresponding to the speech of the i-th positive sample, and the one with the largest loss function in each row is recorded as the difficult negative sample corresponding to the speech of the i-th positive sample; similarly, the i-th column in the matrix is ​​the negative sample corresponding to the vision of the i-th positive sample, and the one with the largest loss function in each column is recorded as the difficult negative sample corresponding to the vision of the i-th positive sample.

[0235] Among them, in this embodiment, the loss function is the largest, corresponding to the distance Minimum.

[0236] In particular, when a row or a column does not contain qualified negative samples, the hard negative samples are not counted.

[0237] The hard negative samples are specifically shown in the following equations (5) and (6):

[0238]

[0239]

[0240] in, Indicates the distance between the jth positive sample’s speech and the hard negative sample, represents the distance between the visual counterpart of the jth positive sample and the hard negative sample, v represents the visual features extracted by the visual neural network, and a represents the speech features extracted by the speech neural network.

[0241] Among them, when the j-th row does not contain qualified negative samples, margin2 is a specific value. Similarly, when the jth column does not contain qualified negative samples,

[0242] In other words, the essence of hard negative sample mining is sorting. j , traverse all visual samples in the batch and construct a negative sample pair combination (v0, a j ),...,(v N , a j), if there is a qualified negative sample, select a difficult negative sample pair from the qualified negative samples. And for a visual sample v j , traverse all speech samples in the batch and construct a negative sample pair combination (v j , a0),...,(v j , a N ), select a difficult negative sample pair from the qualified negative samples.

[0243] S1004: Input the difficult positive sample and the multiple difficult negative samples into the speech and image synchronization measurement model with adjusted parameters for training, and adjust the parameters in the speech and image synchronization measurement model again.

[0244] After mining the hard positive samples and negative samples online from the positive samples and negative samples, there is no need to calculate the loss of all the positive samples and negative samples in the batch. Therefore, the loss function corresponding to the speech and image synchronization measurement model will also change accordingly. The changed loss function is shown in the following formula (7):

[0245]

[0246] Among them, L represents the loss value, represents the hard positive sample distance, Indicates the distance between the jth positive sample’s speech and the hard negative sample, It represents the distance between the visual counterpart of the jth positive sample and the hard negative sample, N represents the batch size of the sample, and margin2 is a specific value.

[0247] Through difficult positive samples and negative samples, as well as the loss function after corresponding changes, it is possible to further adjust the parameters in the speech and image synchronization measurement model, further optimize the model, and improve the accuracy of model prediction.

[0248] In the actual model optimization process, generally speaking, optimization is not performed only once, but multiple times. That is, after optimizing the model once using the current batch of training data, the next batch of training data is used again to obtain the corresponding difficult positive and negative samples, and then input into the current model for training again, and repeat this process multiple times until the output value of the corresponding loss function remains in a stable area, that is, the output value is within a certain accuracy range and no longer fluctuates greatly.

[0249] S1005: Obtain the difficult positive samples in the next batch of positive samples again.

[0250] S1006: Obtain negative samples in the batch and multiple difficult negative samples in the negative samples again.

[0251] Among them, multiple hard negative samples correspond to each sample in the positive sample.

[0252] S1007: Input the re-acquired difficult positive samples and multiple difficult negative samples into the speech and image synchronization measurement model after adjusting the parameters again for training, and adjust the parameters in the speech and image synchronization measurement model until the loss value output by the loss function corresponding to the speech and image synchronization measurement model converges, that is, the loss value is within a certain accuracy range and no longer fluctuates greatly.

[0253] The specific implementation methods of steps S1005, S1006, and S1007 are similar to those of the above-mentioned steps S1002, S1003, and S1004, and will not be repeated here.

[0254] At this point, after processing m batches of samples in the above manner, the speech and image synchronization measurement model has been trained. Wherein, m is less than or equal to M (M is the batch of positive samples). When it is necessary to measure whether the speech segment and image segment in a video are synchronized, the speech segment and image segment in the video are processed through the above steps S202-S205 and S206-S212 respectively, and then respectively input into the speech and image synchronization measurement model, and the output result of the model can characterize whether the speech segment and image segment in the video are synchronized.

[0255] Here, the process of the method for measuring the synchronization of speech and image provided in the embodiment of the present application is fully described.

[0256] Fig.11 This is a complete flow chart of the method for measuring the synchronization of speech and image in the embodiment of the present application, see Fig.11As shown in FIG. 1 , after the video stream is obtained, it is divided into two paths. In one path, the video stream is input into the preprocessing module to preprocess the video stream and obtain a voice segment. Then the voice segment is input into the SI-AR system to process the video stream into a PPG signal. Then the PPG signals of multiple single frames are accumulated into a voice data. Then the voice data is input into the voice neural network to obtain voice features. In the other path, the video stream is densely aligned frame by frame. In one frame of image, there may be multiple faces, and the following steps need to be performed for each face: Extract the expression coefficient from the face. Use the frontal posture and standard ID to generate a 3D model with the expression coefficient extracted from the face image. Project the corresponding vertices in the 3D model to obtain 2D associated point lines. Accumulate the obtained multi-frame 2D associated point lines into an image data. Then the image data is input into the visual neural network to obtain visual features. Finally, the voice features and visual features are input into the synchronization measurement module to measure whether the voice and image in the video stream are synchronized. If the threshold is met, synchronization is determined; if the threshold is not met, it is determined to be out of synchronization. The synchronization measurement module can be used to determine the synchronization of speech features and visual features. The specific synchronization measurement can be achieved by calculating the distance between speech features and visual features on a vector and then comparing it with a preset threshold. Finally, the synchronization measurement module can be used to determine the face with the best synchronization. If the synchronization of all faces in the video does not reach the preset threshold, it is determined that there is no suitable face in the video image in the current time segment.

[0257] In practical applications, the method provided by the embodiment of the present application can be applied in various scenarios where it is necessary to determine whether the voice and the image are synchronized. The following takes three specific scenarios as examples to further illustrate the method provided by the embodiment of the present application.

[0258] Scenario 1: Determine the speaker.

[0259] When there are multiple people talking in a video, in order to determine the current speaker, first, the corresponding voice clips and image clips are extracted from the video; then, the voice clips are processed into PPG signals to erase the speaker's timbre, intonation and other personal features, and the image clips are extracted with the 3DMM parameter estimation algorithm to obtain expression coefficients, which are processed into a two-dimensional contour map of the lower half of the front face to eliminate interference from the side and occlusion. There are as many two-dimensional contour maps as there are faces in the image; then, the voice data obtained after processing the voice clips is input into the voice neural network, and the image data obtained after processing the image clips is input into the visual neural network to obtain voice features and multiple visual features respectively; finally, the multiple visual features are synchronously matched with the voice features respectively, and then the visual features with the highest synchronization with the voice features are determined, and then the person corresponding to the visual features is determined as the current speaker in the video.

[0260] Scenario 2: Forged video identification.

[0261] The sounds or images in some videos may not be original, but may be artificially added later. For example, some celebrity videos are re-dubbed with words that the celebrities never said. Another example: in some interactive liveness authentication, users are required to read the words displayed on the screen, and then record them as a video and upload them. In order to pass the verification, criminals obtain the user's image in advance, dub it, and upload it as a video.

[0262] In order to determine whether a video is fake, first, extract the corresponding voice clips and image clips from the video; then, process the voice clips into PPG signals to erase the speaker's timbre, intonation and other personal features, and extract the expression coefficients from the image clips through the 3DMM parameter estimation algorithm, and process them into a two-dimensional contour map of the lower half of the front face to eliminate interference from the side and occlusion; then, input the voice data obtained after processing the voice clips into the voice neural network, and input the image data obtained after processing the image clips into the visual neural network to obtain voice features and visual features respectively; finally, perform synchronization matching of the voice features and visual features. The higher the matching degree, the more it means that the image and voice in the video are synchronized, rather than artificially added later. When the matching degree is higher than a certain value, it can be determined that the image and voice in the video are generated by the same person at the same time, that is, the voice clip in the video belongs to the person in the image clip.

[0263] Scenario 3: Video modulation.

[0264] When recording videos, some non-professional multimedia devices often use separate devices for collecting voice and images. A microphone can be used to collect voice, and a camera can be used to collect images. The collected voice and images are then merged into a video. This can easily cause the voice and image in the video to be out of sync in time, that is, the audio and image are out of sync.

[0265] In order to solve the problem of audio and video being out of sync in the video, first, the corresponding voice clips and image clips are extracted from the video; then, the voice clips are processed into PPG signals to erase the speaker's personal characteristics such as timbre and intonation, and the image clips are extracted with the 3DMM parameter estimation algorithm to form a two-dimensional contour map of the lower half of the front face to eliminate interference from the side and occlusion; then, the voice data obtained after processing the voice clips is input into the voice neural network, and the image data obtained after processing the image clips is input into the visual neural network to obtain voice features and visual features respectively; finally, the voice features are synchronously matched with the visual features to determine the degree of misalignment between the voice and the image, and then auxiliary calibration is performed, so that the voice and image are aligned in time according to the calibration to eliminate the misalignment.

[0266] Based on the same inventive concept, as an implementation of the above method, the embodiment of the present application also provides a device for measuring the synchronization of speech and image. Fig.12 The structure of the device for measuring the synchronization of speech and image in the embodiment of the present application is shown in FIG. Figure 1 , see Fig.12 As shown, the device may include:

[0267] The receiving module 1201 is used to obtain the voice segments and image segments in the video.

[0268] The data processing module 1202 is used to convert the speech segment into a specific signal, where the specific signal is irrelevant to the individual characteristics of the speaker in the speech segment.

[0269] The feature extraction module 1203 is used to obtain the speech features of the specific signal through a speech neural network.

[0270] The feature extraction module 1203 is used to obtain the visual features of the image segment through a visual neural network.

[0271] The synchronization measurement module 1204 is used to determine whether the speech segment and the image segment are synchronized according to the speech feature and the visual feature.

[0272] Further, as Fig.12 As a refinement and extension of the device shown, the embodiment of the present application also provides a device for measuring the synchronization of speech and image. Fig.13 The structure of the device for measuring the synchronization of speech and image in the embodiment of the present application is shown in FIG. Figure 2 , see Fig.13 As shown, the device may include:

[0273] The receiving module 1301 is used to obtain the voice segments and image segments in the video.

[0274] The pre-processing module 1302 is used to convert the sampling frequency of the speech segment into a specific frequency.

[0275] And, the pre-processing module 1302 is used to remove the background sound in the speech segment and separate the speech of different speakers in the speech segment after the background sound is removed to obtain at least one speech sub-segment.

[0276] And, the pre-processing module 1302 is used to divide the speech segment into a plurality of speech frames by using a sliding weighting method, and there is overlap between adjacent speech frames.

[0277] Correspondingly, the data processing module 1303 is used to convert the speech segment converted into a specific frequency into a specific signal.

[0278] And, the data processing module 1303 is used to convert the speech sub-segment into a specific signal.

[0279] And, the data processing module 1303 is used to convert the multiple voice frames into multiple specific signals respectively.

[0280] Wherein, the specific signal is a language category posterior probability PPG signal.

[0281] The data processing module 1303 is specifically configured to convert the speech segment into a language category posterior probability PPG signal through a speaker automatic speech recognition SI-ASR system.

[0282] The data processing module 1303 is further configured to generate a contour image of the target person according to the image segment, wherein the contour image is independent of individual features of the target person.

[0283] The feature extraction module 1304 includes:

[0284] The first extraction unit 1304a is used to process the specific signal in the time dimension using multiple 1-dimensional convolutional layers to obtain a feature matrix, and the number of the 1-dimensional convolutional layers is related to the time length corresponding to the specific signal.

[0285] The second extraction unit 1304b is used to reorganize the feature matrix into a feature vector.

[0286] The third extraction unit 1304c is used to process the feature vector using three fully connected layers and one linear projection layer to obtain 513-dimensional speech features.

[0287] The feature extraction module 1304 is also used to obtain the visual features of the contour map through a visual neural network.

[0288] When the video is a video of a conversation between multiple people, the synchronization measurement module 1305 is used to determine the speaker corresponding to the voice segment in the video according to the voice features and the visual features.

[0289] When the video is a video to be subjected to forgery identification, the synchronization measurement module 1305 is used to determine whether the voice segment in the video belongs to the person in the image segment according to the voice feature and the visual feature.

[0290] When the video is to be modulated, the synchronization measurement module 1305 is used to align the start positions of the voice segment and the image segment in the video according to the voice features and the visual features, so that the voice segment is synchronized with the image segment.

[0291] It should be noted here that the description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0292] Based on the same inventive concept, an embodiment of the present application also provides an electronic device. Fig.14 This is a schematic diagram of the structure of the electronic device in the embodiment of the present application, see Fig.14 As shown, the electronic device may include: a processor 1401, a memory 1402, and a bus 1403; wherein the processor 1401 and the memory 1402 communicate with each other via the bus 1403; the processor 1401 is used to call program instructions in the memory 1402 to execute the methods in one or more of the above embodiments.

[0293] It should be noted that the description of the above electronic device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the electronic device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0294] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, which may include: a stored program; wherein, when the program is running, the device where the storage medium is located is controlled to execute the method in one or more of the above embodiments.

[0295] It should be noted here that the description of the above storage medium embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the storage medium embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0296] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for measuring the synchronization of speech and image, characterized in that: The method comprises: Acquire a voice segment and an image segment in a video, wherein the voice segment and the image segment have a corresponding relationship in the video; Converting the speech segment into a specific signal, wherein the specific signal is independent of individual characteristics of the speaker in the speech segment; Obtaining the speech feature of the specific signal through a speech neural network; Extracting expression coefficients of the target person that are independent of individual features from the image segments, wherein the expression coefficients meet the standards of a three-dimensional deformable parameterized face model; Generating a facial contour map of the target person based on the expression coefficient and the universal parameterized face model; the contour map is independent of individual features of the target person; Obtaining visual features of the contour map through a visual neural network; The speech feature and the visual feature are input into a synchronization measurement model for comparison to determine whether the speech segment and the image segment are synchronized, wherein the synchronization is used to characterize that the sound in the speech segment matches the action of the target person in the image segment.

2. The method according to claim 1, characterized in that The specific signal is a language category posterior probability PPG signal.

3. The method according to claim 2, characterized in that The converting the speech segment into a specific signal comprises: The speech segment is converted into a language category posterior probability PPG signal through a speaker automatic speech recognition SI-ASR system.

4. The method according to claim 1, characterized in that: Before converting the speech segment into a specific signal, the method further includes: Converting the sampling frequency of the speech segment to a specific frequency; The converting the speech segment into a specific signal comprises: Convert the speech fragment after conversion to a specific frequency into a specific signal.

5. The method according to claim 1, characterized in that Before converting the speech segment into a specific signal, the method further includes: Removing background sound from the speech segment; Separating the voices of different speakers in the voice segment after removing the background sound to obtain at least one voice sub-segment; The converting the speech segment into a specific signal comprises: The speech sub-segment is converted into a specific signal.

6. The method according to claim 1, characterized in that Before converting the speech segment into a specific signal, the method further includes: The speech segment is divided into a plurality of speech frames by using a sliding weighting method, and there is overlap between adjacent speech frames; The converting the speech segment into a specific signal comprises: The multiple speech frames are converted into multiple specific signals respectively.

7. The method according to claim 1, characterized in that The obtaining of the speech feature of the specific signal by means of a speech neural network comprises: Using multiple one-dimensional convolutional layers to process the specific signal in the time dimension to obtain a feature matrix, wherein the number of the one-dimensional convolutional layers is related to the duration corresponding to the specific signal; Reorganizing the feature matrix into feature vectors; The feature vector is processed using three fully connected layers and one linear projection layer to obtain speech features, and the dimension of the speech features is related to the data volume of the speech segment and the type of loss function adopted by the speech neural network.

8. The method according to any one of claims 1 to 7, characterized in that The universal parameterized face model is a universal three-dimensional face model; the step of generating a face contour image of the target person based on the expression coefficient and the universal parameterized face model comprises: Extracting the lower half face expression coefficient corresponding to the lower half face from the expression coefficient; The lower face expression coefficient is input into the universal three-dimensional face model to obtain a three-dimensional face model corresponding to the lower face of the target person, and the three-dimensional face model is processed into a face contour map of the target person.

9. The method according to claim 8, characterized in that The step of inputting the lower face expression coefficient into the universal three-dimensional face model to obtain a three-dimensional face model corresponding to the lower face of the target person, and processing the three-dimensional face model into a face contour map of the target person includes: Inputting the lower face expression coefficient into the general three-dimensional face model to obtain a three-dimensional face model corresponding to the lower face of the target person; Obtaining a vertex set of the lower half of the face in the three-dimensional face model; The vertex set is projected onto a two-dimensional plane to obtain a lower half face contour map of the target person, and the lower half face contour map is used as a face contour map of the target person.

10. The method according to any one of claims 1 to 7, characterized in that The video is a video of a conversation between multiple people; the inputting the voice features and the visual features into a synchronization measurement model for comparison to determine whether the voice segment and the image segment are synchronized includes: inputting the voice features and the visual features into a synchronization measurement model for comparison to determine the speaker corresponding to the voice segment in the video; Alternatively, the video is a video to be subjected to forgery identification; the step of inputting the voice feature and the visual feature into a synchronization measurement model for comparison to determine whether the voice segment and the image segment are synchronized includes: inputting the voice feature and the visual feature into a synchronization measurement model for comparison to determine whether the voice segment in the video belongs to a person in the image segment; Alternatively, the video to be modulated; the inputting of the speech features and the visual features into a synchronization measurement model for comparison to determine whether the speech segment and the image segment are synchronized, including: inputting the speech features and the visual features into a synchronization measurement model for comparison to align the starting positions of the speech segment and the image segment in the video so that the speech segment is synchronized with the image segment.

11. A device for measuring the synchronization of speech and image, characterized in that: The device comprises: A receiving module, used for acquiring a voice segment and an image segment in a video, wherein the voice segment and the image segment have a corresponding relationship in the video; A data processing module, used for converting the speech segment into a specific signal, wherein the specific signal is independent of individual characteristics of a speaker in the speech segment; A feature extraction module, used for obtaining the speech features of the specific signal through a speech neural network; The data processing module is further used to extract the expression coefficient of the target person that is unrelated to individual characteristics from the image fragment, and the expression coefficient meets the standard of the three-dimensional deformable parameterized face model; Generating a facial contour map of the target person based on the expression coefficient and the universal parameterized face model; the contour map is independent of individual features of the target person; The feature extraction module is used to obtain the visual features of the contour map through a visual neural network; The synchronization measurement module is used to input the voice features and the visual features into a synchronization measurement model for comparison to determine whether the voice segment and the image segment are synchronized, wherein the synchronization is used to characterize that the sound in the voice segment matches the action of the target person in the image segment.

12. An electronic device, characterized in that: include: Processor, memory, bus; Wherein, the processor and the memory communicate with each other through the bus; The processor is used to call the program instructions in the memory to execute the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: include: A stored program; wherein, when the program is running, the device where the storage medium is located is controlled to execute the method as described in any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images