A cross-modal artificial intelligence information processing system and retrieval method

Abstract

A cross-modal artificial intelligence information processing system and a cross-modal information retrieval method. The system includes: a separation module, which is configured to separate the first modal information to obtain a plurality of continuous first modal information segments; a feature extraction module, which is configured to extract features of the content expressed by each first modal information segment to form an event map; an identification module, which is configured to identify the elements in the event map using the second modal information to form the second modal identification information; a second encoding module, which is configured to encode the second modal identification information to form the second modal information data; an association module, which is configured to associate the second modal information data with each frame of data in the first modal information segment of the corresponding segment to generate an association identifier; a first insertion module, which is configured to insert the association identifier into the first modal data frame; and a second insertion module, which is configured to insert the association identifier into the second modal data frame.

Description

Technical Field

[0001] The present invention relates to a cross-modal artificial intelligence information processing system and a retrieval method, and belongs to the technical field of artificial intelligence. Background Art

[0002] In the prior art, full-text search can be performed on text information by using keywords, but for audio / video information, it is impossible to search for information of interest in an audio time segment and a video time segment of a certain length. Summary of the invention

[0003] The purpose of the present invention is to provide a cross-modal artificial intelligence information processing system and retrieval method, which can quickly perform cross-modal information retrieval and reproduction.

[0004] To achieve the above-mentioned purpose, the present invention provides a cross-modal artificial intelligence information processing system, which includes: a separation module, which is configured to separate the first modal information into multiple continuous segments of the first modal information; a feature extraction module, which is configured to extract features of the content expressed by the first modal information to form an event graph representing the events and their relationships in the content expressed by the first modal information; an identification module, which is configured to identify the elements in the event graph using the second modal information to form second modal identification information; a second encoding module, which is configured to encode the second modal identification information to form second modal information data; an association module, which is configured to associate the second modal information data with each frame of data in the corresponding segment of the first modal information data to generate an association identifier; a first insertion module, which is configured to insert the association identifier into the first modal data frame, and then store it in the first modal information database; a second insertion module, which is configured to insert the association identifier into the second modal data frame and then store it in the second modal information database.

[0005] Preferably, the first modal information includes voice and / or video; and the second modal information includes text.

[0006] Preferably, the feature extraction module includes an event graph establishment module and an accumulation module, wherein the event graph establishment module is configured to establish an event graph based on the content expressed by the first modal information, and the accumulation module is configured to accumulate the duration of consecutive identical event graphs; the separation module is also configured to separate the first modal information according to the duration to obtain multiple continuous first modal information segments.

[0007] Preferably, the cross-modal artificial intelligence information processing system also includes a first encoding module, which is used to encode the separated first modal information segments to generate first modal information data.

[0008] Preferably, the first modality information includes a video signal; and the second modality information includes text.

[0009] Preferably, the feature extraction module includes a conversion module, an artificial intelligence module, an event graph establishment module and an accumulation module, wherein the conversion module converts the first modal information data into a two-dimensional image; the artificial intelligence module is configured to identify the feature values ​​of each frame of the two-dimensional image, and the feature values ​​include foreground image feature values ​​and background image feature values; the event graph establishment module is configured to establish an event graph based on the relationship between the image elements represented by the foreground image feature values ​​of each frame of the image and the relationship between the image elements represented by the foreground image feature values ​​and the image elements represented by the background image feature values; the accumulation module is configured to accumulate the duration of consecutive identical event graphs; the separation module is also configured to separate the first modal information according to the duration to obtain multiple continuous first modal information segments.

[0010] To achieve the above-mentioned purpose of the invention, the present invention also provides a method for cross-modal information retrieval using the above-mentioned system, which includes the following steps: searching for corresponding second modal data in a second modal information database based on the input second modal information; extracting an association identifier of the second modal data; retrieving a first modal information data frame from a first modal information database based on the association identifier, and reproducing the first modal information using the first modal information data frame.

[0011] Compared with the prior art, the purpose of the present invention is to provide a cross-modal artificial intelligence information processing system and retrieval method, which can quickly perform cross-modal information retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of a cross-modal artificial intelligence information processing system provided by the first embodiment of the present invention;

[0013] Figure 2 is a schematic diagram showing dividing the first modal information into a plurality of information segments;

[0014] Figure 3 It is a block diagram of the composition of the first encoding module in the cross-modal artificial intelligence information processing system provided by one embodiment of the present invention;

[0015] Figure 4 is a block diagram of an inter-frame prediction processing module provided by an embodiment of the present invention;

[0016] Figure 5 is a block diagram of a cross-modal artificial intelligence information processing system provided by a second embodiment of the present invention;

[0017] Figure 6 It is a flow chart of the cross-modal information retrieval method provided by the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] In the description of the present invention, it should be noted that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0020] First embodiment

[0021] Figure 1 is a block diagram of a cross-modal artificial intelligence information processing system provided by the first embodiment of the present invention, such as Figure 1 As shown, the first embodiment provides a cross-modal artificial intelligence information processing system including: first modal information 510, which is, for example, audio information obtained by an acoustic-electric converter or image information obtained by a photoelectric converter; a separation module 520, which is configured to separate the first modal information into a plurality of continuous first modal information segments; a feature extraction module, which is configured to extract features of the content expressed by the first modal information segments to form an event graph representing events and their relationships in the content expressed by the first modal information, wherein the event graph is organized in the form of a tree structure, in which each node is called an element; an identification module 580, which is configured to identify the elements in the event graph using the second modal information to form second modal identification information; a second encoding module 590, which is configured to extract features of the content expressed by the first modal information segments to form an event graph representing events and their relationships in the content expressed by the first modal information. The two-modal identification information is encoded to form the second modal information data, that is, the second modal information is encoded using a character string, and the character string includes a binary character string; an association module 570, which is configured to associate the second modal information data with the first modal information data of the corresponding segment to generate an association identifier (or an association pointer); a first insertion module 540, which is configured to insert the association identifier into each frame of the first modal data information data segment, and then store it in the first modal information database or send it to the channel encoder, and then send it to the communication unit after channel encoding; a second insertion module 600, which is configured to insert the association identifier into the second modal data frame and then store it in the second modal information database or send it to the channel encoder, and then send it to the communication unit after channel encoding.

[0022] In the first embodiment, the first modal information includes voice and / or video, and the voice includes voices in multiple languages, dialects, etc.; the second modal information includes text, and the text includes texts in multiple languages.

[0023] In the first embodiment, each frame of data in the first modal information data has the following format:

[0024] Each frame of data in the second modal information data has the following format:

[0025]

[0026] The first modal information data for inserting the association identifier has the following format:

[0027]

[0028] The second modal information data for inserting the association identifier has the following format:

[0029]

[0030] In the first embodiment, the feature extraction module includes an event graph establishment module 550 and an accumulation module 560. The event graph establishment module 550 is configured to establish an event graph according to the content expressed by the first modal information, and the accumulation module 560 is configured to accumulate the duration of continuous identical event graphs, that is, the first modal information represents the same event time period; the separation module 520 is also configured to separate the continuous first modal information with a certain duration according to the duration to obtain multiple continuous first modal information segments. Figure 2 As shown, a video signal with a set duration T expresses four events: event 1, event 2, event 3 and event 4. The separation module divides the video into four segments with durations of T1, T2, T3 and T4, respectively. Preferably, each event can be further subdivided according to different contents expressed.

[0031] In the first embodiment, the cross-modal artificial intelligence information processing system further includes a first encoding module 530, which is used to encode the separated first modal information segments to generate first modal information data. In the present invention, when the first modal information is a video signal, the first encoding module uses Figure 3 The structural form shown.

[0032] Figure 3 FIG. 1 is a block diagram of the composition of the first encoding module in the present invention. Figure 3As shown, in the first encoding module, the prediction residual signal generation module 103 obtains the difference between the input video signal and the prediction signal outputted by the inter-frame prediction processing module 102, and outputs it as a prediction residual signal. The conversion module 104 performs an orthogonal transform such as discrete cosine transform on the prediction residual signal, quantizes the transform coefficients, and outputs the quantized transform coefficients. The entropy coding module 105 performs entropy coding on the quantized transform coefficients and outputs them as a coded stream. On the other hand, the quantized transform coefficients are also inputted to the inverse conversion module 106, where they are inversely quantized and inversely orthogonally transformed, and a prediction residual signal is outputted. In the decoded image signal generation module 107, the prediction residual signal and the prediction signal outputted by the inter-frame prediction processing module 102 are added to generate a decoded image signal of the encoded encoding target block. The decoded image signal is outputted to the loop filter processing module 108 in order to be used as a reference image in the inter-frame prediction processing module 102. The loop filter processing module 108 performs filtering processing to reduce coding distortion, and outputs the filtered image as a decoded video signal to the inter-frame prediction processing module 102 .

[0033] Figure 4 is a block diagram of the inter-frame prediction processing module 102 of the present invention, such as Figure 4As shown, the inter-frame prediction processing module 102 includes a reduced image generation unit 291, a pre-search processing unit 292, a first mode determination unit 293, an integer pixel search processing unit 294, a fractional image generation unit 295, a fractional pixel search processing unit 296, and a second mode determination unit 297. The reduced image generation unit 291 inputs the current frame image signal and the previous frame image signal, performs a reduction process, and outputs the reduced image signal. The reduction process is performed by, for example, a convolutional neural network CNN. The pre-search processing unit 292 inputs the reduced current frame image signal and the previous frame image signal, performs a motion search process on the reduced current frame image signal, and transmits the searched motion vector to the integer pixel search processing unit 294. In addition, the first mode determination unit 293 inputs the coding mode information from the pre-search processing unit 292. The integer pixel search processing unit 294 performs an integer pixel search process according to the motion vector and the coding mode. The decimal image generation unit 295 generates a decimal pixel interpolation image at the corresponding previous frame image position and outputs it to the decimal pixel search processing unit 296; the second mode determination unit 297 receives the coding mode information from the integer pixel search processing unit 294 and inputs it to the decimal pixel search processing unit 296; the decimal pixel search processing unit 296 performs decimal pixel search processing by the motion vector and coding mode respectively specified by the integer pixel search processing unit 294 and the second mode determination unit 297. The predicted residual image and motion vector information are retrieved in the decimal pixel search processing unit 296, and the feature value is extracted according to the predicted residual image and motion vector information. The first embodiment of the present invention can improve the coding efficiency through the above scheme.

[0034] Second embodiment

[0035] Figure 5 is a block diagram of a cross-modal artificial intelligence information processing system provided by a second embodiment of the present invention, such as Figure 5As shown, the second embodiment provides a cross-modal artificial intelligence information processing system including: first modal information data 310, which is configured to obtain first modal information data from multiple information sources, such as audio data and / or video data obtained through a channel decoder, audio data and / or video data obtained through a network, the first modal information data has multiple time series data frames, and the first modal information data can be displayed through a display component to express the development process of one or more events; a separation module 320, which is configured to separate the first modal information data into multiple continuous first modal information data segments, each first modal information data segment has multiple time series data frames; a feature extraction module, which is configured to extract features of the content expressed by the reproduced first modal information data to form events that characterize the events and their relationships that reproduce the content of each first modal information data segment. A graph; an identification module 370, which is configured to identify the elements in the event graph using the second modal information to form the second modal identification information; a second encoding module 390, which is configured to encode the second modal identification information to form the second modal information data; an association module 380, which is configured to associate the second modal information data with each frame of data in the first modal information data segment of the corresponding segment to generate an association identifier; a first insertion module 410, which is configured to insert the association identifier into the first modal information data frame, and then store it in the first modal information database or send it to the channel encoder, and then send it to the communication unit after channel encoding; a second insertion module 400, which is configured to insert the association identifier into the second modal data frame and then store it in the second modal information database or send it to the channel encoder, and then send it to the communication unit after channel encoding.

[0036] In the first embodiment, the first modality information includes voice data and / or video data; the second modality information includes text.

[0037] In the first embodiment, the feature extraction module includes a conversion module 330, an artificial intelligence module 340, an event graph establishment module 350 and an accumulation module 360, wherein the conversion module 330 converts the first modal information data into a two-dimensional image in time series; the artificial intelligence module is configured to identify the deep image feature value of each frame of the two-dimensional image, and the deep image feature value includes a background image feature value and a plurality of foreground image feature values; the event graph establishment module 350 is configured to establish an event graph according to the relationship between the image elements represented by the plurality of foreground image feature values ​​of each frame of the image and the relationship between the image elements represented by the background image feature values; the accumulation module 360 ​​is configured to accumulate the duration of continuous identical event graphs; the separation module 320 is also configured to separate the first modal information according to the duration to obtain a plurality of continuous first modal information segments,

[0038] In the second embodiment, each frame of data in the second modal information data has the following format:

[0039]

[0040] Each frame of data in the second modal information data has the following format:

[0041]

[0042] The first modal information data for inserting the association identifier has the following format:

[0043] The second modal information data for inserting the association identifier has the following format:

[0044]

[0045] In the second embodiment, the artificial intelligence module includes a convolutional neural network (CNN), which is configured to classify the input image into background image feature values ​​and foreground image feature values, and to divide the foreground image feature values ​​into multiple foreground image element feature values. The convolutional neural network is applied to image recognition technology for recognizing a specified shape or pattern from image data as input data, and has an intermediate layer and a fully combined layer. The intermediate layer is composed of a plurality of feature extraction processing layers connected hierarchically. The intermediate layer has a convolution layer and a pooling layer.

[0046] Figure 6 is a flow chart of the artificial intelligence cross-modal information retrieval method provided by the present invention, such as Figure 6 As shown, the method for cross-modal information retrieval using the above-mentioned system provided by the present invention includes the following steps: searching for corresponding second modal information data in a second modal information database according to second modal information (such as text keywords) input by a user; extracting an association identifier of the second modal information data; retrieving first modal information data (such as a video data stream, an audio data stream) from a first modal information database according to the association identifier, and reproducing the first modal information using the retrieved first modal data, such as reproducing an image through a display device and reproducing a sound using a speaker.

[0047] When performing text keyword retrieval through the technical solution provided by the present invention, the associated audio / video data segments can be quickly found according to the event graph, and reproduced using the audio / video data segments without converting the entire audio / video data into audio and / or video, thereby realizing cross-modal information retrieval and improving retrieval efficiency; at the same time, it allows users to watch the videos they want to watch and / or listen to the audio segments they want to listen to without having to care about the parts they do not want to pay attention to, thereby improving user time utilization.

[0048] The present invention can be implemented by a computer in each of the above-mentioned embodiments, or the program for implementing each embodiment can be recorded in a computer-readable recording medium, and the computer system reads and executes the program recorded in the recording medium to implement the present invention. In addition, the "computer system" mentioned here includes hardware such as OS and peripheral devices. In addition, the "computer-readable recording medium" refers to a removable medium such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into the computer system.

[0049] Furthermore, the "computer-readable recording medium" may include a medium that dynamically retains the program for a short period of time, such as a communication line that transmits the program through a network such as the Internet or a communication line such as a telephone line, or may include a medium that retains the program for a predetermined time, such as a volatile memory in a computer system used as a server or client in this case. In addition, the above-mentioned program may be a program for realizing a part of the aforementioned functions, a program that realizes the aforementioned functions by combining with a program already recorded in a computer system, or a program realized by using hardware such as a PLD or FPGA.

[0050] The above embodiments are only used to illustrate the present invention, wherein the structure, location and connection mode of each component may be changed. Any equivalent transformations and improvements based on the technical solution of the present invention should not be excluded from the protection scope of the present invention.

Claims

1. A cross-modal artificial intelligence information processing system, characterized in that: include: A separation module, a feature extraction module, an identification module, a first encoding module, a second encoding module, a first insertion module, an association module, and a second insertion module, wherein the separation module is configured to separate the first modal information to obtain a plurality of continuous first modal information segments, and the first modal information is a video signal; the first encoding module is used to encode the separated first modal information segments to generate first modal information data; the feature extraction module is configured to extract features from the content expressed by the first modal information to form an event map representing events and their relationships in the content expressed by the first modal information; the identification module is configured to identify elements in the event map using the second modal information to form second modal identification information; the second encoding module is configured to extract features from the content expressed by the first modal information to form second modal identification information; The identification information is encoded to form second modal information data; the association module is configured to associate the second modal information data with each frame of data in the first modal information data of the corresponding segment to generate an association identifier; the first insertion module is configured to insert the association identifier into the first modal data frame, and then store it in the first modal information database; the second insertion module is configured to insert the association identifier into the second modal data frame and then store it in the second modal information database; the first encoding module includes: a prediction residual signal generation module, an inter-frame prediction processing module, a first conversion module, an entropy coding module, an inverse conversion module, a decoded image signal generation module and a loop filter processing module, wherein the prediction residual signal generation module calculates the input video signal and the inter-frame prediction as The difference between the prediction signal output by the detection processing module and the prediction signal is output as a prediction residual signal; the first conversion module performs an orthogonal transformation on the prediction residual signal, quantizes the transformation coefficients, and outputs the quantized transformation coefficients; the entropy coding module performs entropy coding on the quantized transformation coefficients and outputs them as a coded stream; the quantized transformation coefficients are also input to the inverse transformation module, inversely quantized and inversely orthogonally transformed, and a prediction residual signal is output; the decoded image signal generation module adds the prediction residual signal and the prediction signal output by the inter-frame prediction processing module to generate a decoded image signal of the encoded encoding object block, and the decoded image signal is output to the loop filter processing module for reducing the encoding in order to be used as a reference image in the inter-frame prediction processing module. The inter-frame prediction processing module comprises a reduced image generation unit, a pre-search processing unit, a first mode determination unit, an integer pixel search processing unit, a fractional image generation unit, a fractional pixel search processing unit and a second mode determination unit. The reduced image generation unit inputs the current frame image signal and the previous frame image signal, performs reduction processing and outputs the reduced image signal; the pre-search processing unit inputs the reduced current frame image signal and the previous frame image signal, performs motion search processing on the reduced current frame image signal and transmits the searched motion vector to the integer pixel search processing unit; the first mode determination unit inputs the coding mode information from the pre-search processing unit;The integer pixel search processing unit performs integer pixel search processing according to the motion vector and the coding mode; the fractional image generation unit generates a fractional pixel interpolation image of the corresponding previous frame image position and outputs it to the fractional pixel search processing unit; the second mode determination unit receives the coding mode information from the integer pixel search processing unit and inputs it to the fractional pixel search processing unit; the fractional pixel search processing unit performs fractional pixel search processing according to the motion vector and the coding mode respectively specified by the integer pixel search processing unit and the second mode determination unit; the fractional pixel search processing unit retrieves the prediction residual image and the motion vector information, and extracts the feature value according to the prediction residual image and the motion vector information.; 2. The cross-modal artificial intelligence information processing system according to claim 1, characterized in that: The feature extraction module includes an event graph establishment module and an accumulation module, wherein the event graph establishment module is configured to establish an event graph according to the content expressed by the first modal information, and the accumulation module is configured to accumulate the duration of consecutive identical event graphs; The separation module is also configured to separate the first modal information according to duration to obtain a plurality of continuous first modal information segments.

3. The cross-modal artificial intelligence information processing system according to claim 1, characterized in that: The second modal information includes text.

4. The cross-modal artificial intelligence information processing system according to claim 1, characterized in that: The feature extraction module includes a second conversion module, an artificial intelligence module, an event graph establishment module and an accumulation module, wherein the second conversion module converts the first modal information data into a two-dimensional image; the artificial intelligence module is configured to identify the feature values ​​of each frame of the two-dimensional image, and the feature values ​​include foreground image feature values ​​and background image feature values; the event graph establishment module is configured to establish an event graph based on the relationship between the image elements represented by the foreground image feature values ​​of each frame of the image and their relationship with the image elements represented by the background image feature values; the accumulation module is configured to accumulate the duration of consecutive identical event graphs; the separation module is configured to separate the first modal information according to the duration to obtain multiple continuous first modal information segments.

5. A cross-modal information retrieval method, characterized in that: Using the cross-modal artificial intelligence information processing system described in any one of claims 1 to 4 to perform cross-modal information retrieval, comprising the following steps: searching for corresponding second modal data in a second modal information database based on input second modal information; extracting an association identifier of the second modal data; retrieving a first modal information data frame from a first modal information database based on the association identifier, and reproducing the first modal information using the first modal information data frame.

Images