A multi-system data management method based on multimodal model

Abstract

The present invention provides a multi-system data management method based on a multimodal model, belonging to the technical field of multimodal data processing, the method comprising: step S1, obtaining target picture data containing a target person from the Internet according to text description data corresponding to the target person; step S2, obtaining target video segment data containing the target person from a multimedia database according to the target picture data, the multimedia database at least including a plurality of video files; step S3, extracting target voice segment data of the target person from the target video segment data; step S4, establishing an association relationship between text description data and target voice segment data. The technical solution of the present invention associates the text description data of the target person with the target picture data, the target video segment data, and the target voice segment data in sequence, thereby finally achieving the purpose of associating the text description data with the target voice segment data, which is conducive to realizing multimodal data management of the person.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data processing, and in particular to a multi-system data management method based on a multimodal model. Background Art

[0002] Multimodality includes a series of different forms of data expressions around a certain topic, such as text, pictures, voice, and video. Traditionally, different modal data on a certain topic often exist independently, lacking effective associations between them, which is not conducive to the effective management and efficient indexing of related subject data. Among them, various types of data such as character text descriptions, images, voice, and videos are the most typical multimodal data with the largest data volume and the most extensive application. In the prior art, when managing multimodal data of characters, most of them only realize the association between text data and image data, and how to realize the association between text data and video data and voice data of characters has always been a technical problem in this field; in particular, no relevant content has been disclosed for the scheme of associating text data and voice data of characters. Summary of the invention

[0003] The present invention can solve at least one of the technical problems existing in the prior art, and proposes a multi-system data management method based on a multimodal model.

[0004] The multi-system data management method based on the multimodal model includes:

[0005] Step S1, acquiring target image data containing the target person from the Internet according to the text description data corresponding to the target person;

[0006] Step S2, acquiring target video segment data containing the target person from a multimedia database according to the target image data, wherein the multimedia database includes at least a plurality of video files;

[0007] Step S3, extracting the target voice segment data of the target person from the target video segment data;

[0008] Step S4: establishing an association relationship between the text description data and the target speech segment data.

[0009] Optionally, step S1 includes:

[0010] Step S101, obtaining the name of the target person from the text description data, and using the name as a keyword to perform an image search on the Internet to obtain image search results;

[0011] Step S102: Filter out pictures containing frontal face images from the picture search results, and all the filtered pictures constitute a picture set to be processed;

[0012] Step S103: clustering the pictures in the to-be-processed picture set based on the facial features of the frontal images of the faces contained in the pictures, and calculating the proportion of the number of pictures corresponding to each face category according to the clustering result, wherein the proportion of the number of pictures is equal to the ratio of the number of pictures contained in the corresponding face category to the number of pictures contained in the to-be-processed picture set;

[0013] Step S104, determining whether the largest proportion of the number of pictures is greater than or equal to a first proportion threshold;

[0014] If the judgment result of step S104 is yes, then execute step S105;

[0015] Step S105: taking all the pictures included in the face category corresponding to the largest proportion of the number of pictures as the target picture data.

[0016] Optionally, in step S1, before step S101, the following steps are further included: step S100;

[0017] Step S100, let i be 0;

[0018] If the judgment result of step S104 is no, then execute step S106;

[0019] Step S106, adding 1 to the value of i;

[0020] Step S107, obtaining i characteristic words of the target person from the text description data, and using the name and the i characteristic words as keywords to perform an image search on the Internet to obtain image search results;

[0021] After step S107 is completed, step S102 is executed again;

[0022] Between step S102 and step S103, the method further includes:

[0023] Step S103a, determining whether the number of pictures in the to-be-processed picture set is greater than or equal to a first number threshold;

[0024] If yes, execute step S103; if no, execute step S103b;

[0025] Step S103b: obtaining manually specified target image data containing the target person.

[0026] Optionally, step S2 includes:

[0027] Step S201, using the picture in the target picture data as a reference, for each video frame picture in each of the video files, detecting whether the video frame picture contains the target person;

[0028] Step S202: among all the video frame pictures containing the target person, a plurality of the video frame pictures that are continuous in the corresponding video file are divided into a continuous picture set, and the video segments corresponding to the continuous picture set are extracted from the corresponding video file, and all the extracted video segments are used as the target video segment data.

[0029] Optionally, step S3 includes:

[0030] Step S301: Selecting a video segment containing voice from all video segments included in the target video data as a video segment to be processed;

[0031] Step S302: extracting the timbre features of the speech in each of the video segments to be processed;

[0032] Step S303: clustering the video segments to be processed according to the timbre features to obtain a timbre clustering result, wherein the timbre clustering result includes n timbre categories and the video segments to be processed included in each timbre category;

[0033] Step S304, determine whether n is equal to 1;

[0034] If the judgment result of step S304 is yes, then execute step S305;

[0035] Step S305 , extracting corresponding speech segments from each of the to-be-processed video segments included in only one timbre category in the timbre clustering result, and using all the extracted speech segments as the target speech segment data.

[0036] Optionally, in step S3, if the judgment result of step S304 is no, then step S306 is executed;

[0037] Step S306: for each of the to-be-processed video segments in the timbre clustering results, recognize each Chinese character voice in the to-be-processed video segment one by one according to the pronunciation of the Chinese characters, and express the recognition result of each Chinese character voice in the form of pinyin, and the recognition results of all Chinese character voices in the to-be-processed video segment are arranged in sequence to form a first pinyin recognition result; and, perform voice recognition on each sentence in the to-be-processed video segment in a complete sentence recognition manner through voice recognition software, and express the recognition result of each sentence voice in the form of pinyin, and the recognition results of all sentence voices in the to-be-processed video segment are arranged in sequence to form a second pinyin recognition result;

[0038] Step S307: for each of the to-be-processed video segments in the timbre clustering results, detecting the recognition result similarity between the corresponding first pinyin recognition result and the second pinyin recognition result;

[0039] Step S308, detecting whether the similarities of the recognition results corresponding to all the to-be-processed video segments in the timbre clustering results are simultaneously greater than or equal to a first similarity threshold or simultaneously less than the first similarity threshold;

[0040] If the detection result of step S308 is that there exists at least one recognition result corresponding to the to-be-processed video segment whose similarity is greater than or equal to the first similarity threshold and there exists at least one recognition result corresponding to the to-be-processed video segment whose similarity is less than the first similarity threshold, then step S309 is executed;

[0041] Step S309, deleting the to-be-processed video segments whose recognition result similarity is greater than or equal to a first similarity threshold from the timbre clustering result, so as to update the timbre clustering result;

[0042] Step S310, determining whether the number of timbre categories in the updated timbre clustering result is equal to 1;

[0043] If the determination result of step S310 is yes, step S305 is executed.

[0044] Optionally, in step S3, if the detection result of step S310 is no, step S311 is executed;

[0045] Step S311: for each of the to-be-processed video segments in the timbre clustering results, detecting a pronunciation feature stability index of the speech segment contained in the to-be-processed video segment;

[0046] Step S312, detecting whether the pronunciation feature stability indexes corresponding to all the to-be-processed video segments in the timbre clustering result are simultaneously greater than or equal to the corresponding preset stability threshold or simultaneously less than the corresponding preset stability threshold;

[0047] If the detection result of step S312 is that there is at least one pronunciation feature stability index corresponding to the to-be-processed video segment that is greater than or equal to the preset stability threshold and there is at least one pronunciation feature stability index corresponding to the to-be-processed video segment that is less than the preset stability threshold, then step S313 is executed;

[0048] Step S313, deleting the to-be-processed video segments whose pronunciation feature stability index is greater than or equal to a preset stability threshold from the timbre clustering result, so as to update the timbre clustering result;

[0049] Step S314, determining whether the number of timbre categories in the updated timbre clustering result is equal to 1;

[0050] If the determination result of step S310 is yes, step S305 is executed.

[0051] Optionally, in step S3, if the detection result of step S314 is no, then execute step S315;

[0052] Step S315, in the timbre clustering result, respectively calculate the total duration of the video segments corresponding to each timbre category, the total duration of the video segments corresponding to the timbre category being equal to the sum of the durations of all the to-be-processed video segments contained in the timbre category;

[0053] Step S316: extract corresponding voice segments from each of the to-be-processed video segments included in the timbre category corresponding to the largest total duration of the video segment, and use all the extracted voice segments as the target voice segment data.

[0054] Optionally, the step of establishing an association relationship between the text description data and the target speech segment data includes:

[0055] An index data structure of the text description data, the target image data, the target video segment data, and the target voice segment data is constructed, wherein the index data structure is a directed graph data structure, and the nodes representing the text description data in the directed graph data structure respectively point to the nodes representing the target image data, the nodes representing the target video segment data, and the nodes representing the target voice segment data.

[0056] Optionally, the multi-system data management method based on the multimodal model further includes:

[0057] Step S5, storing the data of each video segment in the target video segment data;

[0058] In step S5, the step of storing data of a target video segment in the target video segment data includes:

[0059] Step S501: Calculate the image similarity between any two adjacent video frame images in the target video segment, and perform segmentation between two adjacent video frame images whose image similarity is less than or equal to a second similarity threshold, so as to divide the target video segment into a plurality of video sub-segments;

[0060] Step S502: for each of the video sub-segments, using a semantic segmentation model to divide each video frame image in the video sub-segment to determine a foreground image and a background image in each of the video images, wherein the foreground image is an image of an area surrounded by an outer contour edge of the target person in the video frame image, and the background image is other images in the video frame image except the foreground image, and the background image includes: a moving object background image and a fixed object background image;

[0061] Step S503: for each of the video sub-segments, extracting a face image and body posture information from a foreground image in each video frame picture in the video sub-segment, and determining a shared body appearance image corresponding to the video sub-segment based on the foreground image in at least part of the video frame pictures in the video sub-segment;

[0062] Step S504: for each of the video sub-segments, spatially perform fixed background merging processing on the fixed background partial images in all the video frame pictures included in the video sub-segment to obtain a complete fixed background image corresponding to the video sub-segment, and record the regional position information of the fixed background partial images in each video frame picture in the video sub-segment in the complete fixed background image;

[0063] Step S505: for each of the video sub-segments, determining the sub-segment storage data corresponding to the video sub-segment, the sub-segment storage data including: a shared body appearance image corresponding to the video sub-segment, a complete fixed background image corresponding to the video sub-segment, the face image corresponding to each of the video frame images included in the video sub-segment, the body posture information corresponding to each of the video frame images included in the video sub-segment, and the region position information corresponding to each of the video frame images included in the video sub-segment;

[0064] Step S506: Store the sub-segment storage data of all the video sub-segments included in the target video segment.

[0065] In the technical solution of the present invention, target image data of the target person is obtained from the Internet through the text description data of the target person, and then target video segment data containing the target person is obtained from a multimedia database based on the target image data, and then target voice segment data of the target person is extracted from the target video segment data, that is, the target image data and the target video segment data are used as association bridges, and the text description data of the target person is sequentially associated with the target image data, the target video segment data, and the target voice segment data, so as to finally achieve the purpose of associating the text description data with the target voice segment data, which is conducive to realizing multimodal data management of the person. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flowchart of a multi-system data management method based on a multimodal model provided by the present invention.

[0067] Figure 2 This is a flow chart of an optional implementation method of step S1 in the present invention.

[0068] Figure 3 This is a flow chart of an optional implementation method of step S2 in the present invention.

[0069] Figure 4 This is a flow chart of an optional implementation method of step S3 in the present invention.

[0070] Figure 5 It is a relationship diagram corresponding to the multimodal data in the present invention.

[0071] Figure 6 It is a directed graph data structure corresponding to the multimodal data in the present invention.

[0072] Figure 7 A flowchart of another multi-system data management method based on a multimodal model provided by the present invention.

[0073] Figure 8 This is a flow chart of an optional implementation method of step S5 in the present invention. DETAILED DESCRIPTION

[0074] Example embodiments will be described more fully below with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. On the contrary, the purpose of providing these embodiments is to make the present invention thorough and complete and to enable those skilled in the art to fully understand the scope of the present invention.

[0075] The terms used herein are only used to describe specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an" and "the" are also intended to include plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms "comprise" and / or "comprising" are used in this specification, the existence of steps, operations, elements and / or components is specified, but the existence or addition of one or more other steps, operations, elements, elements and / or components thereof is not excluded.

[0076] It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited to these terms.

[0077] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present invention, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined as such herein.

[0078] See also Figure 1 As shown, the multi-system data management method based on the multimodal model includes:

[0079] Step S1: acquiring target image data containing the target person from the Internet according to the text description data corresponding to the target person.

[0080] Among them, text description data refers to data that describes the target person in written language, which is a sentence or a combination of multiple sentences with complete and systematic meaning (Message); a text can be a sentence (Sentence), a paragraph (Paragraph) or a chapter (Discourse). In the present invention, the text description data at least records the name of the target object. In addition, the text description data generally also contains some feature words (generally located in the sentence) used to describe the target object.

[0081] The target image data includes at least one image containing a target person.

[0082] Step S2: acquiring target video segment data containing the target person from a multimedia database according to the target image data.

[0083] The multimedia database includes at least a plurality of video files, and the video files generally include video data and audio data; the target video segment data includes at least one video segment including a target person.

[0084] Step S3: extracting the target voice segment data of the target person from the target video segment data.

[0085] The target voice segment data includes at least one voice segment of the voice uttered by the target person. It should be noted that the voice segment acquired in step S3 is extracted from the video segment in the target video segment data.

[0086] Step S4: establishing an association relationship between the text description data and the target speech segment data.

[0087] The text description data and the target speech segment data may be associated with each other by defining a data structure.

[0088] In the present invention, target image data of the target person is obtained from the Internet through text description data of the target person, and then target video segment data containing the target person is obtained from a multimedia database based on the target image data, and then target voice segment data of the target person is extracted from the target video segment data, that is, the target image data and the target video segment data are used as association bridges, and the text description data of the target person is sequentially associated with the target image data, the target video segment data, and the target voice segment data, so as to finally achieve the purpose of associating the text description data with the target voice segment data, which is conducive to realizing multimodal data management of the person.

[0089] To help those skilled in the art better understand the technical solution of the present invention, the technical solution of the present invention will be exemplarily described below in conjunction with some embodiments.

[0090] See also Figure 2 As shown, optionally, step S1 includes the following steps S101 to S105.

[0091] Step S101: Obtain the name of the target person from the text description data, and use the name as a keyword to perform an image search on the Internet to obtain image search results.

[0092] Generally, when searching for images on the Internet using only the name as a keyword, the image search results obtained will include a large number of images. Therefore, when there are a large number of images in the image search results, the first m (for example, m is 100) images can be intercepted as the final image search results for subsequent processing.

[0093] Step S102: Filter out pictures containing frontal face images from the picture search results, and all the filtered pictures constitute a set of pictures to be processed.

[0094] Step S103: cluster the images in the image set to be processed based on the facial features of the frontal images of the human faces contained in the images, and calculate the proportion of the number of images corresponding to each face category according to the clustering results.

[0095] The images in the image set to be processed can be clustered according to facial features using a clustering algorithm based on feature similarity calculation, a K-Means clustering algorithm, a hierarchical clustering algorithm, a DBSCAN clustering algorithm, etc. The present invention does not limit the clustering algorithm used in step S103.

[0096] The image quantity ratio is equal to the ratio of the number of images included in the corresponding face category to the number of images included in the image set to be processed. That is, through step S103, it can be obtained that the images in the image set to be processed are finally divided into several categories and the image quantity ratio corresponding to each category.

[0097] If the clustering result includes several face categories, it can indicate to some extent that the images in the to-be-processed image set come from several people (one face category corresponds to one person). When the number of face categories is greater than or equal to 2, it can indicate that the people in the to-be-processed image set have the same name.

[0098] Step S104: determine whether the maximum proportion of the number of pictures is greater than or equal to a first proportion threshold.

[0099] The first percentage threshold is a constant pre-designed based on actual experience. Generally, the first percentage threshold is greater than or equal to 85%, for example, the first percentage threshold may be 90%, 95%, and the like.

[0100] If the judgment result of step S104 is yes, that is, the largest number of pictures accounts for a proportion greater than or equal to the first proportion threshold, then there is great credibility that the person in the face category corresponding to the largest number of pictures is the target person, and then step S105 is executed.

[0101] Step S105: taking all pictures included in the face category corresponding to the largest picture quantity ratio as target picture data.

[0102] Continue to see Figure 2 As shown, optionally, in step S1 and between step S102 and step S103 , step S103 a and step S103 b are also included.

[0103] Step S103a: determine whether the number of pictures in the picture set to be processed is greater than or equal to a first number threshold.

[0104] When the number of pictures in the to-be-processed picture set is greater than or equal to the first number threshold, it means that the number of pictures in the to-be-processed picture set is sufficient for subsequent processing, and then step S103 is executed. When the number of pictures in the to-be-processed picture set is less than the first number threshold, it means that the number of pictures in the to-be-processed picture set is too small, and the credibility reflected by the subsequent processing result is low, and then step S103b is executed.

[0105] Step S103b: obtaining manually specified target image data containing a target person.

[0106] See again Figure 2 As shown, optionally, in step S1 and before step S101 , the following steps are also included: step S100 , and after step S105 , step S106 and step S107 are also included.

[0107] Step S100, let i be 0.

[0108] If the judgment result of step S104 is no, it means that it is difficult to determine the picture of the target object from the current picture set to be processed (there are many pictures whose names contain the same name as the target object), and then step S106 is executed.

[0109] Step S106: add 1 to the value of i.

[0110] Step S107, obtaining i characteristic words of the target person from the text description data, and using the name and the i characteristic words as keywords to search for images on the Internet to obtain image search results. After step S107 is completed, step S102 is executed again.

[0111] The process of step S107 is to increase the number of keywords to conduct a more accurate search on the Internet, so as to reduce the number of pictures containing characters with the same name as the target object as much as possible.

[0112] See also Figure 3 As shown, optionally, step S2 includes the following steps S201 and S202.

[0113] Step S201: using the pictures in the target picture data as a reference, for each video frame picture in each video file, detecting whether the video frame picture contains the target person.

[0114] As an example, when the number of pictures in the target picture data is small, the similarity between each video frame picture in the video file and each picture in the target picture data can be calculated, and the similarity between the video frame picture and each picture in the target picture data can be used to obtain the credibility that the video frame picture contains the target person (for example, the maximum value, minimum value, average value or median value of the similarity between the video frame picture and each picture in the target picture data is used as the credibility), and then the credibility is compared with a preset credibility threshold; if the credibility is greater than the preset credibility threshold, it is detected that the test video frame picture contains the target person; otherwise, it is detected that the test video frame picture does not contain the target person.

[0115] As another example, when the number of pictures in the target picture data is large, a target person detection model that can be used to detect the target person can be trained using the pictures in the target picture data through machine learning, and then the target person detection model can be used to detect each video frame picture in each video file.

[0116] Step S202: among all the video frame images containing the target person, a plurality of video frame images that are continuous in the corresponding video file are divided into a continuous image set, and video segments corresponding to the continuous image set are extracted from the corresponding video file, and all the extracted video segments are used as target video segment data.

[0117] In the present invention, based on the image detection method combined with the continuity of the video frame images containing the target person in the video file, the corresponding video segment can be extracted from the video file, thereby realizing the association between the target object and the video segment, that is, the association between the text description data of the target object and the target video segment data is also realized.

[0118] See also Figure 4 As shown, optionally, step S3 includes the following steps S301 to S305.

[0119] Step S301: Select a video segment containing voice from all video segments included in the target video data as a video segment to be processed.

[0120] Step S302: extracting the timbre features of the speech in each video segment to be processed.

[0121] Generally, timbre feature extraction is performed by performing spectrum analysis on audio signals to extract frequency components and features therein, thereby obtaining timbre features.

[0122] Step S303: clustering the video segments to be processed according to the timbre features to obtain timbre clustering results.

[0123] The timbre clustering result includes n timbre categories and the to-be-processed video segments contained in each timbre category. Generally, the timbre of different characters is different, so n timbre categories represent the existence of n different characters' voices.

[0124] Step S304: determine whether n is equal to 1.

[0125] If the judgment result of step S304 is yes, which means that the voice comes from one person (ie, the target object), step S305 is executed.

[0126] Step S305 , extract corresponding speech segments from each to-be-processed video segment included in only one timbre category in the timbre clustering result, and use all extracted speech segments as target speech segment data.

[0127] Continue to see Figure 4 As shown, optionally, in step S3 and after step S304, steps S306 to S310 are further included. If the judgment result of step S304 is no, that is, it means that the voice comes from more than one person, step S306 is executed.

[0128] Step S306: for each video segment to be processed in the timbre clustering result, recognize each Chinese character voice in the video segment to be processed one by one according to the pronunciation of the Chinese characters, and express the recognition result of each Chinese character voice in the form of pinyin, and the recognition results of all Chinese character voices in the video segment to be processed are arranged in sequence to form a first pinyin recognition result. And, perform voice recognition on each sentence in the video segment to be processed in a complete sentence recognition manner through voice recognition software, and express the recognition result of each sentence voice in the form of pinyin, and the recognition results of all sentence voices in the video segment to be processed are arranged in sequence to form a second pinyin recognition result.

[0129] In step S306, the speech in each video segment to be processed is recognized using two completely different recognition technologies: 1) Chinese character-by-character speech recognition; 2) sentence-by-sentence recognition using speech recognition software. At the same time, the recognition results are all expressed in pinyin.

[0130] Among them, single Chinese character speech recognition recognizes the original pronunciation of the character, while speech recognition software is based on the overall recognition of the sentence, usually with a single-word recognition and error correction function combined with contextual semantics.

[0131] Step S307: for each to-be-processed video segment in the timbre clustering result, detect the recognition result similarity between the corresponding first pinyin recognition result and the second pinyin recognition result.

[0132] The similarity between two pinyin segments may be calculated using a similarity calculation algorithm (for example, it may be considered as a similarity calculation between two pinyin character strings).

[0133] When the similarity between the first pinyin recognition result and the second pinyin recognition result is relatively high, it means that the Chinese pronunciation of the corresponding character is standard (the corresponding voice may be a narration, and ordinary people do not use very standard pronunciation for every word when speaking normally). Otherwise, it means that the Chinese pronunciation of the corresponding character is not standard.

[0134] Step S308: Detect whether the similarities of the recognition results corresponding to all the to-be-processed video segments in the timbre clustering results are simultaneously greater than or equal to a first similarity threshold or simultaneously less than the first similarity threshold.

[0135] If the detection result of step S308 is that there is at least one video segment to be processed whose corresponding recognition result similarity is greater than or equal to the first similarity threshold and there is at least one video segment to be processed whose corresponding recognition result similarity is less than the first similarity threshold, that is, there are characters whose pronunciation is too standard and there are characters whose pronunciation is not too standard, then execute step S309.

[0136] Step S309: deleting the to-be-processed video segments whose recognition result similarity is greater than or equal to the first similarity threshold from the timbre clustering result, so as to update the timbre clustering result.

[0137] The first similarity threshold may be limited as needed according to actual conditions. Generally, the first similarity threshold is greater than or equal to 90% and less than 1. For example, the first similarity threshold is 90%, 95%, and the like.

[0138] That is, through step S309, some to-be-processed videos that are voice or narration (pronunciation is too standard) can be deleted.

[0139] Step S310: Determine whether the number of timbre categories in the updated timbre clustering result is equal to 1.

[0140] If the determination result of step S310 is yes, step S305 is executed.

[0141] Continue to see Figure 4 As shown, optionally, in step S3 and after step S310, steps S311 to S314 are also included.

[0142] Optionally, if the detection result of step S308 is that the similarities of the recognition results corresponding to all the video segments to be processed are simultaneously greater than or equal to the first similarity threshold or simultaneously less than the first similarity threshold, step S311 is executed.

[0143] If the detection result of step S310 is no, step S311 is executed.

[0144] Step S311: for each video segment to be processed in the timbre clustering result, detect the pronunciation feature stability index of the speech segment contained in the video segment to be processed.

[0145] In the present disclosure, the pronunciation feature stability index is used to describe the stability of the pronunciation features in the speech segment. The larger the pronunciation feature stability index is, the better the stability of the pronunciation features is, and the greater the probability of being a narration; the smaller the pronunciation feature stability index is, the worse the stability of the pronunciation features is, and the smaller the probability of being a narration is (usually there are differences in the pronunciation features of announcers, interviewers and ordinary people).

[0146] Optionally, the pronunciation feature stability index may be represented by the inverse of the variance or the inverse of the standard deviation of a plurality of pronunciation feature values ​​collected as the speech segment changes over time.

[0147] In some embodiments, the pronunciation feature smoothness index may include: a volume smoothness index (which can be represented by the inverse of the variance of the volume or the inverse of the standard deviation) or a speech rate smoothness index (which can be represented by the inverse of the variance of the speech rate or the inverse of the standard deviation).

[0148] Step S312: Detect whether the pronunciation feature stability indexes corresponding to all the to-be-processed video segments in the timbre clustering results are simultaneously greater than or equal to the corresponding preset stability threshold or simultaneously less than the corresponding preset stability threshold.

[0149] If the detection result of step S312 is that there is at least one video segment to be processed whose pronunciation feature smoothness index is greater than or equal to the preset smoothness threshold and there is at least one video segment to be processed whose pronunciation feature smoothness index is less than the preset smoothness threshold, then execute step S313.

[0150] Step S313: Delete the to-be-processed video segments whose pronunciation feature stability index is greater than or equal to a preset stability threshold from the timbre clustering result, so as to update the timbre clustering result.

[0151] Step S314: Determine whether the number of timbre categories in the updated timbre clustering result is equal to 1.

[0152] If the determination result of step S314 is yes, step S305 is executed.

[0153] Continue to see Figure 4 As shown, optionally, in step S3 and after step S314, step S315 and step S316 are also included.

[0154] If the detection result of step S312 is that the pronunciation feature stability indicators corresponding to all the video segments to be processed are simultaneously greater than or equal to the preset stability threshold or simultaneously less than the preset stability threshold, step S315 is executed.

[0155] If the detection result of step S314 is no, that is, based on the previous "pronunciation standard" and "pronunciation feature", the speech segment of the target object cannot be effectively determined, then step S315 is executed.

[0156] Step S315: In the timbre clustering results, the total duration of the video segments corresponding to each timbre category is calculated respectively. The total duration of the video segments corresponding to the timbre category is equal to the sum of the durations of all the to-be-processed video segments included in the timbre category.

[0157] Step S316: extract corresponding speech segments from each to-be-processed video segment included in the timbre category corresponding to the maximum total duration of the video segment, and use all extracted speech segments as target speech segment data.

[0158] In some embodiments, step S4 specifically includes: constructing an index data structure of text description data, target image data, target video segment data, and target voice segment data.

[0159] See also Figure 5 and Figure 6As shown, the text description data uses the target image data and the target video segment data as the association bridge, so that the text description data of the target person is associated with the target image data, the target video segment data, and the target voice segment data in sequence, and a relationship diagram of the corresponding four modal data can be established ( Figure 5 Then, based on the indexing requirements, an index data structure can be established according to the relationship graph, wherein the index data structure can be a directed graph data structure, in which the nodes representing the text description data in the directed graph data structure respectively point to the nodes representing the target image data, the nodes representing the target video segment data, and the nodes representing the target voice segment data. In the present disclosure, by establishing the above-mentioned index data structure, the association between different modal data can be effectively established, and the indexing efficiency can be effectively improved. Of course, in practical applications, it is also possible to Figure 6 The index data structure shown establishes associations between different data; for example, if the user is accustomed to continue searching for image data after searching for video data, a new association can be added from video data to image data.

[0160] See also Figure 7 and Figure 8 As shown, in some embodiments, the multi-system data management method based on the multimodal model also includes step S5.

[0161] Step S5: storing the data of each video segment in the target video segment data.

[0162] In multimodal data mainly including text, pictures, voice and video, video data is usually large in volume, occupies a large storage space and is more time-consuming to retrieve. To this end, the present disclosure provides a new method for storing data of video segments.

[0163] In step S5, the step of storing data of a target video segment in the target video segment data includes:

[0164] Step S501: Calculate the image similarity between any two adjacent video frame images in the target video segment, and perform segmentation between any two adjacent video frame images whose image similarity is less than or equal to a second similarity threshold, so as to divide the target video segment into a plurality of video sub-segments.

[0165] Step S502: for each video sub-segment, a semantic segmentation model (such as a Sam model) is used to divide each video frame image in the video sub-segment to determine a foreground image and a background image in each video image.

[0166] The foreground image is the image of the area surrounded by the outer contour edge of the target person in the video frame image, and the background image is the other images in the video frame image except the foreground image. The background image includes: moving object background image and fixed object background image. The moving object background image refers to the image of the object that can move in the background, such as the image of cars and pedestrians. The fixed object background image refers to the image of the fixed (unable to move) object in the background, such as the image of roads, buildings, and trees.

[0167] Step S503: for each video sub-segment, extract the face image and body posture information from the foreground image in each video frame picture in the video sub-segment, and determine the shared body appearance image corresponding to the video sub-segment based on the foreground image in at least part of the video frame pictures in the video sub-segment.

[0168] The shared body appearance image may be an image generated by a certain algorithm based on the other partial images of the foreground partial image in at least part of the video frame image, except the face image. It may also be the image remaining after the face image is directly deleted from the foreground partial image in a certain video frame image. The present disclosure does not limit the algorithm for determining the shared body appearance image.

[0169] The shared body appearance image and body posture information are configured to be used for subsequent restoration of the actual body appearance image of the target person in the foreground image of the corresponding video frame.

[0170] Optionally, the foreground is generated frame by frame using a generative adversarial network training model from the face image and body posture information, and is "supervised learning" using the shared body appearance image to achieve character restoration. It can be seen as generating the actual body appearance image of the target person in the corresponding video frame image using the body posture information and the shared body appearance image, and then splicing the actual body appearance image with the face image to obtain the foreground part image in the corresponding video frame image.

[0171] Step S504: for each video sub-segment, spatially perform fixed background merging processing on the fixed background part images in all the video frame pictures included in the video sub-segment to obtain a complete fixed background image corresponding to the video sub-segment, and record the regional position information of the fixed background part images in each video frame picture in the video sub-segment in the complete fixed background image.

[0172] It should be noted that the merged complete fixed background image may be larger than the video shooting range (such as the shooting position is moving). The complete fixed background image is actually stitched together from the fixed background part images of multiple video frames.

[0173] The region position information is configured to be used for subsequently extracting the corresponding fixed background partial image from the complete fixed background image. In some embodiments, the region position information includes: the upper left corner coordinates and the lower right corner coordinates of the fixed background partial image in the complete fixed background image; a rectangular region can be determined in the complete fixed background image through the upper left corner coordinates and the lower right corner coordinates to serve as the fixed object background image in the corresponding video frame picture.

[0174] Step S505: for each video sub-segment, determine the sub-segment storage data corresponding to the video sub-segment.

[0175] Among them, the sub-segment storage data includes: a shared body appearance image corresponding to the video sub-segment, a complete fixed background image corresponding to the video sub-segment, a face image corresponding to each video frame image included in the video sub-segment, body posture information corresponding to each video frame image included in the video sub-segment, and area position information corresponding to each video frame image included in the video sub-segment.

[0176] Compared with a video file of a video sub-segment, the storage space occupied by the sub-segment storage data corresponding to a video sub-segment in the present invention is much smaller, and the sub-segment storage data is easier to retrieve.

[0177] Step S506: store the sub-segment storage data of all video sub-segments included in the target video segment.

[0178] Of course, the management method provided by the present invention may also include the step of storing text description data, target image data, and target voice segment data.

[0179] In addition, in actual applications, after the target speech segment data of the target person is obtained through step S3, feature extraction can be performed on at least part of the speech segments in the obtained target speech segment data to obtain the speech features of the target person, and then a person detection model based on speech features can be obtained through machine training. Subsequently, the person detection model based on speech features can be directly used to detect some speech segments (for example, there is a pure speech file in the multimedia database) to determine whether the corresponding speech segment belongs to the target person.

[0180] Based on the same inventive concept, the present invention also provides a multi-system data management system based on a multimodal model, which can be used to implement the multi-system data management method based on a multimodal model provided in the previous embodiment.

[0181] The multi-system data management system based on the multimodal model includes: a picture acquisition module, a video segment acquisition module, a voice segment acquisition module, and an association module.

[0182] The image acquisition module is configured to acquire target image data containing the target person from the Internet according to the text description data corresponding to the target person.

[0183] The video segment acquisition module is configured to acquire target video segment data containing a target person from a multimedia database according to the target image data, and the multimedia database includes at least a plurality of video files.

[0184] The voice segment acquisition module is configured to extract target voice segment data of a target person from target video segment data.

[0185] The association module is configured to establish an association relationship between the text description data and the target speech segment data.

[0186] For the description of the above-mentioned functional modules, please refer to the corresponding contents in the previous embodiments, which will not be repeated here.

[0187] According to an embodiment of the present invention, a computer readable medium is further provided, wherein a computer program is stored on the computer readable medium, wherein when the program is executed by a processor, the steps of the multi-system data management method based on a multimodal model as described in any of the above embodiments are implemented.

[0188] It will be appreciated by those skilled in the art that all or some of the steps in the method disclosed above, and the functional modules / units in the device may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0189] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A multi-system data management method based on a multimodal model, characterized in that: include: Step S1, acquiring target image data containing the target person from the Internet according to the text description data corresponding to the target person; Step S2, acquiring target video segment data containing the target person from a multimedia database according to the target image data, wherein the multimedia database includes at least a plurality of video files; Step S3, extracting the target voice segment data of the target person from the target video segment data; Step S4, establishing an association relationship between the text description data and the target speech segment data; Wherein, step S3 comprises: Step S301: Selecting a video segment containing voice from all video segments included in the target video segment data as a video segment to be processed; Step S302: extracting the timbre features of the speech in each of the video segments to be processed; Step S303: clustering the video segments to be processed according to the timbre features to obtain a timbre clustering result, wherein the timbre clustering result includes n timbre categories and the video segments to be processed included in each timbre category; Step S304, determine whether n is equal to 1; If the judgment result of step S304 is yes, then execute step S305; Step S305, extracting corresponding speech segments from each of the to-be-processed video segments included in only one timbre category in the timbre clustering result, and using all the extracted speech segments as the target speech segment data; In step S3, if the judgment result of step S304 is no, then execute step S306; Step S306: for each of the to-be-processed video segments in the timbre clustering results, recognize each Chinese character voice in the to-be-processed video segment one by one according to the pronunciation of the Chinese characters, and express the recognition result of each Chinese character voice in the form of pinyin, and the recognition results of all Chinese character voices in the to-be-processed video segment are arranged in sequence to form a first pinyin recognition result; and, perform voice recognition on each sentence in the to-be-processed video segment in a complete sentence recognition manner through voice recognition software, and express the recognition result of each sentence voice in the form of pinyin, and the recognition results of all sentence voices in the to-be-processed video segment are arranged in sequence to form a second pinyin recognition result; Step S307: for each of the to-be-processed video segments in the timbre clustering results, detecting the recognition result similarity between the corresponding first pinyin recognition result and the second pinyin recognition result; Step S308, detecting whether the similarities of the recognition results corresponding to all the to-be-processed video segments in the timbre clustering results are simultaneously greater than or equal to a first similarity threshold or simultaneously less than the first similarity threshold; If the detection result of step S308 is that there exists at least one recognition result corresponding to the to-be-processed video segment whose similarity is greater than or equal to the first similarity threshold and there exists at least one recognition result corresponding to the to-be-processed video segment whose similarity is less than the first similarity threshold, then step S309 is executed; Step S309, deleting the to-be-processed video segments whose recognition result similarity is greater than or equal to a first similarity threshold from the timbre clustering result, so as to update the timbre clustering result; Step S310, determining whether the number of timbre categories in the updated timbre clustering result is equal to 1; If the determination result of step S310 is yes, step S305 is executed.

2. The method according to claim 1, characterized in that Step S1 includes: Step S101, obtaining the name of the target person from the text description data, and using the name as a keyword to perform an image search on the Internet to obtain image search results; Step S102: Filter out pictures containing frontal face images from the picture search results, and all the filtered pictures constitute a picture set to be processed; Step S103: clustering the pictures in the to-be-processed picture set based on the facial features of the frontal images of the faces contained in the pictures, and calculating the proportion of the number of pictures corresponding to each face category according to the clustering result, wherein the proportion of the number of pictures is equal to the ratio of the number of pictures contained in the corresponding face category to the number of pictures contained in the to-be-processed picture set; Step S104, determining whether the largest proportion of the number of pictures is greater than or equal to a first proportion threshold; If the judgment result of step S104 is yes, then execute step S105; Step S105: taking all the pictures included in the face category corresponding to the largest proportion of the number of pictures as the target picture data.

3. The method according to claim 2, characterized in that In step S1, before step S101, the method further includes: step S100; Step S100, let i be 0; If the judgment result of step S104 is no, then execute step S106; Step S106, adding 1 to the value of i; Step S107, obtaining i characteristic words of the target person from the text description data, and using the name and the i characteristic words as keywords to perform an image search on the Internet to obtain image search results; After step S107 is completed, step S102 is executed again; Between step S102 and step S103, the method further includes: Step S103a, determining whether the number of pictures in the to-be-processed picture set is greater than or equal to a first number threshold; If yes, execute step S103; if no, execute step S103b; Step S103b: obtaining manually specified target image data containing the target person.

4. The method according to claim 1, characterized in that: Step S2 includes: Step S201, using the picture in the target picture data as a reference, for each video frame picture in each of the video files, detecting whether the video frame picture contains the target person; Step S202: among all the video frame pictures containing the target person, a plurality of the video frame pictures that are continuous in the corresponding video file are divided into a continuous picture set, and the video segments corresponding to the continuous picture set are extracted from the corresponding video file, and all the extracted video segments are used as the target video segment data.

5. The method according to claim 1, characterized in that In step S3, if the detection result of step S310 is no, then step S311 is executed; Step S311: for each of the to-be-processed video segments in the timbre clustering results, detecting a pronunciation feature stability index of the speech segment contained in the to-be-processed video segment; Step S312, detecting whether the pronunciation feature stability indexes corresponding to all the to-be-processed video segments in the timbre clustering result are simultaneously greater than or equal to the corresponding preset stability threshold or simultaneously less than the corresponding preset stability threshold; If the detection result of step S312 is that there is at least one pronunciation feature stability index corresponding to the to-be-processed video segment that is greater than or equal to the preset stability threshold and there is at least one pronunciation feature stability index corresponding to the to-be-processed video segment that is less than the preset stability threshold, then step S313 is executed; Step S313, deleting the to-be-processed video segments whose pronunciation feature stability index is greater than or equal to a preset stability threshold from the timbre clustering result, so as to update the timbre clustering result; Step S314, determining whether the number of timbre categories in the updated timbre clustering result is equal to 1; If the determination result of step S314 is yes, step S305 is executed.

6. The method according to claim 5, characterized in that In step S3, if the detection result of step S314 is no, then execute step S315; Step S315, in the timbre clustering result, respectively calculate the total duration of the video segments corresponding to each timbre category, the total duration of the video segments corresponding to the timbre category being equal to the sum of the durations of all the to-be-processed video segments contained in the timbre category; Step S316: extract corresponding voice segments from each of the to-be-processed video segments included in the timbre category corresponding to the largest total duration of the video segment, and use all the extracted voice segments as the target voice segment data.

7. The method according to any one of claims 1 to 6, characterized in that: Step S4 specifically includes: An index data structure of the text description data, the target image data, the target video segment data, and the target voice segment data is constructed, wherein the index data structure is a directed graph data structure, and the nodes representing the text description data in the directed graph data structure respectively point to the nodes representing the target image data, the nodes representing the target video segment data, and the nodes representing the target voice segment data.

8. The method according to claim 1, characterized in that: Also includes: Step S5, storing the data of each video segment in the target video segment data; In step S5, the step of storing data of a target video segment in the target video segment data includes: Step S501: Calculate the image similarity between any two adjacent video frame images in the target video segment, and perform segmentation between two adjacent video frame images whose image similarity is less than or equal to a second similarity threshold, so as to divide the target video segment into a plurality of video sub-segments; Step S502: for each of the video sub-segments, using a semantic segmentation model to divide each video frame image in the video sub-segment to determine a foreground image and a background image in each of the video images, wherein the foreground image is an image of an area surrounded by an outer contour edge of the target person in the video frame image, and the background image is other images in the video frame image except the foreground image, and the background image includes: a moving object background image and a fixed object background image; Step S503: for each of the video sub-segments, extracting a face image and body posture information from a foreground image in each video frame picture in the video sub-segment, and determining a shared body appearance image corresponding to the video sub-segment based on the foreground image in at least part of the video frame pictures in the video sub-segment; Step S504: for each of the video sub-segments, spatially perform fixed background merging processing on the fixed object background images in all the video frame pictures included in the video sub-segment to obtain a complete fixed background image corresponding to the video sub-segment, and record the regional position information of the fixed object background image in each video frame picture in the video sub-segment in the complete fixed background image; Step S505: for each of the video sub-segments, determining the sub-segment storage data corresponding to the video sub-segment, the sub-segment storage data including: a shared body appearance image corresponding to the video sub-segment, a complete fixed background image corresponding to the video sub-segment, the face image corresponding to each of the video frame images included in the video sub-segment, the body posture information corresponding to each of the video frame images included in the video sub-segment, and the region position information corresponding to each of the video frame images included in the video sub-segment; Step S506: Store the sub-segment storage data of all the video sub-segments included in the target video segment.

Images