Information storage method of multi-modal data and related equipment
By dividing the blurred text areas into image blocks in the multimodal data information storage and performing feature repair processing, the problems of text information identification errors and low storage efficiency are solved, and more efficient information storage is achieved.
Patent Information
- Application Number
- CN202510411869.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-02
- Publication Date
- 2025-08-01
AI Technical Summary
In the existing multimodal data information entry method, text information identification errors and information storage efficiency are low, making it difficult to apply to real-time information entry scenarios.
By acquiring image data, clear and blurred text areas are determined, blurred text areas are divided into multiple image blocks, and feature repair processing is performed using clear image blocks, text information of repair text areas is extracted, and text information is stored in the target database.
This solves the problem of text information recognition errors caused by image blurring, reduces noise and interference in the target text, and improves information storage efficiency.
Smart Images
Figure CN120407704A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of information processing technologies, and in particular, to a method for storing information of multimodal data and related devices. Background Art
[0002] The information entry based on multimodal data such as text, images, and voices has been widely applied in fields such as government affairs, medical care, education, law, and finance. By combining technologies such as text recognition, speech recognition, and image recognition, functions such as automated document processing, intelligent customer service, and government information entry can be realized, thus significantly improving the information collection efficiency.
[0003] In terms of image text extraction, common methods include super-resolution reconstruction and data augmentation. Super-resolution reconstruction recovers more details and textures from low-resolution images, but it cannot accurately recover all the information in high-resolution images, resulting in possible errors in the reconstructed text. At the same time, efficient super-resolution algorithms usually require a large amount of computing resources and time, especially when processing high-resolution images, which may affect the processing speed and real-time performance of the algorithms. Data augmentation technology improves the generalization ability of the model by transforming images (such as rotation, scaling, cropping, etc.). However, data augmentation may have limited effects on certain types of image blurs and requires specific strategies or preprocessing methods to effectively repair. In addition, data augmentation operations need to be applied in real time during each training iteration, which will increase the hardware resources and time costs required for training.
[0004] Therefore, the related methods for storing information of multimodal data still have problems such as incorrect text information recognition and low information storage efficiency, and it is difficult to be applicable to the scenario of real-time information entry. Summary of the Invention
[0005] Embodiments of this application provide a method for storing information of multimodal data and related devices, so as to at least solve the problems of incorrect text information recognition and low information storage efficiency in the related methods for storing information of multimodal data.
[0006] To solve the above technical problems, this application is implemented as follows: In a first aspect, an information storage method for multimodal data provided by an embodiment of the present application includes: obtaining at least one type of modal data, where the at least one type of modal data includes image data; determining a clear text region and a blurred text region according to text features in the image data; extracting first text information of the clear text region, and dividing the blurred text region into multiple image blocks, performing feature repair processing on the blurred image blocks in the multiple image blocks based on clear image blocks that match the blurred image blocks in the multiple image blocks to obtain a repaired text region, and extracting second text information of the repaired text region; storing target text in the text information in a target database, where the text information includes the first text information and the second text information.
[0007] In a second aspect, an information storage device for multimodal data provided by an embodiment of the present application includes: a data acquisition module for obtaining at least one type of modal data, where the at least one type of modal data includes image data; a region determination module for determining a clear text region and a blurred text region according to text features of the image data; an information extraction module for extracting first text information of the clear text region, and dividing the blurred text region into multiple image blocks, performing feature repair processing on the blurred image blocks in the multiple image blocks based on the clear image blocks in the multiple image blocks to obtain a repaired text region, and extracting second text information of the repaired text region; an information storage module for storing target text in the text information in a target database, where the text information includes the first text information and the second text information.
[0008] In a third aspect, an electronic device provided by an embodiment of the present application includes a processor and a memory, where the memory stores a program or instruction that can run on the processor, and when the program or instruction is executed by the processor, the steps of the method described in the first aspect above are implemented.
[0009] In a fourth aspect, a readable storage medium provided by an embodiment of the present application stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect above are implemented.
[0010] In a fifth aspect, a computer program product provided by an embodiment of the present application includes a computer program stored on a non-transitory computer-readable storage medium, where the computer program includes program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the steps of the method described in the first aspect above.
[0011] In an embodiment of the present application, at least one type of modality data is obtained, and the at least one type of modality data includes image data; according to the text features in the image data, a clear text region and a blurred text region are determined; first text information of the clear text region is extracted, and the blurred text region is divided into multiple image blocks, and feature repair processing is performed based on clear image blocks that match the blurred image blocks in the multiple image blocks to obtain a repaired text region, and second text information of the repaired text region is extracted; target text in the text information is stored in a target database, and the text information includes the first text information and the second text information. In this way, during the information storage process of the image data, the blurred text region is divided into multiple image blocks, and the clear image blocks that match the blurred image blocks are used to perform feature repair on it, which can solve the problem of incorrect recognition of text information caused by image blurring, reduce noise and interference in the target text, and improve information storage efficiency.
[0012] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The accompanying drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.
[0014] Figure 1 The flowchart shows the information storage method for multi-modal data provided by some embodiments of the present application; Figure 2 The flowchart shows the method for extracting blurred image information provided by some embodiments of the present application; Figure 3 The flowchart shows the method for extracting information from voice data provided by some embodiments of the present application; Figure 4 The structural diagram shows the target information extraction model provided by some embodiments of the present application; Figure 5 The flowchart shows the information storage method for multi-modal data provided by some other embodiments of the present application; Figure 6 The structural diagram shows the information storage device for multi-modal data provided by some embodiments of the present application; Figure 7 The structural diagram shows the electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS
[0015] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0016] Figure 1 A flowchart of an information storage method for multi-modal data provided by an embodiment of the present application is shown. The execution subject of this method can be a terminal device or a server. Among them, the terminal device can be a device such as a personal computer, or a mobile terminal device such as a mobile phone or a tablet computer. The terminal device can be a terminal device used by a user. The server can be an independent server or a server cluster composed of multiple servers. Moreover, the server can be a background server for a certain service or a background server for a certain platform or application program (for example, a government information entry system, a medical image analysis system, a security monitoring system, etc.). In the embodiment of the present application, the case where the execution subject is a server is taken as an example for description. For the case of the terminal device, it can be processed according to the following relevant content and will not be elaborated herein. As shown in the figure, the method 100 may include the following steps: Step 101: Obtain at least one type of modal data, where the at least one type of modal data includes image data.
[0017] Among them, the above at least one type of modal data may include text data, image data, and voice data.
[0018] In the embodiment of the present application, image data can be collected and obtained in real time through a camera, a webcam, a scanner, etc.; image data pre-stored in a hard disk or a database can also be obtained, where the image data can be a certain frame in a video stream or a single image.
[0019] Step 102: Determine a clear text area and a blurred text area according to the text features in the image data.
[0020] In a specific implementation, image data can be input into the text detection and localization module. Through this text detection and localization module, text information in the image is localized, and text features are extracted. By methods such as edge detection, image gradient, frequency domain analysis, and contrast calculation, clear text regions and blurred text regions are determined. For example, by calculating the gradient of the image, that is, the rate of change of pixel values, the clarity of the text is judged. Regions with a larger gradient indicate obvious textures or edges and are clear text regions; regions with a smaller gradient are blurred text regions. Another example is to perform a Fourier transform on the image to convert the image from the spatial domain to the frequency domain. In the frequency domain, since edges and details are manifestations of high-frequency signals, the clarity of the text can be determined based on high-frequency information.
[0021] Step 103: Extract the first text information of the clear text region, and divide the blurred text region into multiple image blocks. Based on the clear image blocks that match the blurred image blocks in the multiple image blocks, perform feature repair processing to obtain a repaired text region, and extract the second text information of the repaired text region.
[0022] In a specific implementation, if the text information is clear, the first text information of the clear text region is extracted; if the text information is blurred, the blurred text region is input into the blurred information matching module. Through this blurred information matching module, the blurred text region is divided into multiple image blocks. For example, the blurred text region is divided into image blocks of 8×8 pixels, the blurred image blocks in the multiple image blocks are obtained, and feature repair processing is performed according to the clear image blocks that match the blurred image blocks to obtain a repaired text region, and then the second text information of the repaired text region is extracted.
[0023] Since there is usually no color requirement for the storage of text information, before dividing the blurred text region into multiple image blocks, image preprocessing can also be performed on the blurred text region. Among them, image preprocessing includes regularizing the size of the blurred text region and converting the blurred text region into a grayscale image. In this way, while maintaining the original text, the three-channel color dimension is reduced to a one-channel dimension, and then the low-frequency part is extracted through discrete Fourier transform, which is beneficial to reducing the amount of calculation and improving the feature repair efficiency of the blurred text region.
[0024] Step 104: Store the target text in the text information into the target database, where the text information includes the first text information and the second text information.
[0025] In a specific implementation, the target text that needs to be stored in the first text information and the second text information can be extracted, and the target text is stored in the target database. This target database can be a database corresponding to a government affairs platform, a medical image analysis system, a security monitoring system, etc.
[0026] Through the above steps, during the information storage process of the image data, the blurred text area is divided into multiple image blocks, and the clear image blocks are used to perform feature repair on the blurred image blocks, which can solve the problem of incorrect text information recognition caused by image blurring, reduce the noise and interference in the target text, and improve the information storage efficiency.
[0027] In some embodiments, in step 103 above, feature repair processing is performed based on the clear image blocks that match the blurred image blocks among the multiple image blocks to obtain a repaired text area, including: For each image block, according to the gray value of each pixel in the image block and the average gray value of the pixels in the image block, determine the relative gray average value of the image block; based on the average gray value of the pixels in the blurred text area and the relative gray average values of the multiple image blocks, determine the perceptual hash value of each image block; based on the perceptual hash values of each image block, determine the clear image blocks that match the blurred image blocks among the multiple image blocks; perform feature repair processing on the blurred image blocks according to the clear image blocks to obtain a repaired text area.
[0028] In a specific implementation, for each image block, the relative gray average value of the image block can be determined according to the difference between the gray value of each pixel in the image block and the average gray value of the pixels; the difference between the average gray value of the pixels in the blurred text area and the relative gray average value of the image block is determined as the perceptual hash value of the image block, and this value is the normalized feature of the original image. By comparing the Manhattan distance between image blocks of the same size within a fixed threshold, the clear image blocks that match the blurred image blocks are found, and then feature repair processing is performed on the blurred image blocks according to the clear image blocks to obtain a repaired text area.
[0029] In some embodiments, after performing the feature repair processing on the blurred image blocks according to the clear image blocks to obtain a repaired text area, it further includes: Determine whether the repaired text area has a correct semantics through a semantic recognizer based on regular expressions; in response to the repaired text area having a correct semantics, extract the second text information of the repaired text area.
[0030] In some embodiments, based on the perceptual hash values of each image block, determining the clear image blocks that match the blurred image blocks among the multiple image blocks includes: Determine the clear image blocks that match the blurred image blocks according to the distance between the perceptual hash value of the blurred image block and the perceptual hash values of multiple clear image blocks.
[0031] In a specific implementation, when there is image translation or one image is a part of another image, the initial scanning algorithm may not be able to detect a clear image that matches the blurred image. Therefore, when such a failure occurs, by dividing the blurred text area into multiple image blocks and using the distance between perceptual hash values to estimate the clear image blocks that match the blurred image blocks, in this way, regarding the bounding box as meaningful semantics, by extracting the similarity between the text bounding box and all clear image blocks of the same corresponding size in the image to be matched, and finding the clear image block that matches it, this means completing the text information of the blurred part by partial matching of the existing clear text image part to achieve the repair of the blurred information.
[0032] In an exemplary embodiment, as Figure 2 shown, taking the image data as an ID card picture as an example, the above-mentioned information storage method for multi-modal data includes the following steps: Step 1: Input the ID card picture; Step 2: Input the ID card picture into the text detection and localization module to obtain multiple text areas; Step 3: Obtain a complete clear image, determine the text area of the complete clear image, and divide the text area to obtain multiple clear image blocks; Step 4: According to the text features in the ID card picture, judge whether to perform fuzzy matching. Perform fuzzy matching on the blurred text area in the ID card picture and proceed to Step 5; there is no need to perform fuzzy matching on the clear text area in the ID card picture and proceed to Step 6; Step 5: Input the blurred text area into the fuzzy information matching module. The fuzzy information matching module divides the blurred text area into multiple image blocks, and based on the clear image blocks that match the blurred image blocks in the multiple image blocks, for example, "1", "Name: XX", etc., perform feature repair processing to obtain a repaired text area; Step 6: Extract the text information of the clear text area and the repaired text area to obtain the first text information and the second text information, and end the process.
[0033] In some embodiments, in the above-mentioned Step 101, at least one type of modal data includes voice data, and the text information includes third text information; after obtaining at least one type of modal data in the above-mentioned Step 101 and before storing the target text in the text information in the target database in the above-mentioned Step 104, it further includes: Determine a position encoding sequence according to the voiceprint features in the voice data; input the voiceprint features and the position encoding sequence into an encoder after adding them to obtain an acoustic feature sequence; input the position encoding sequence and the acoustic feature sequence into a decoder after adding them to obtain a decoding sequence; input the decoding sequence into a connectionist temporal classification model after normalization processing, and predict the characters corresponding to the decoding sequence through the connectionist temporal classification model to obtain a character prediction result; determine the third text information according to the character prediction result.
[0034] In an exemplary embodiment, first extract the voiceprint features in the voice data, then use positional encoding to ensure that the position information of the words in the input voice remains unchanged, add the extracted features and the positional encoding content to obtain a position encoding sequence, and use this position encoding sequence as the input sequence of the transformer encoder. After passing through the encoder, enhance the acoustic features of each voice vector to obtain an acoustic feature sequence, which is used as a part of the input in the decoder. Furthermore, add the position encoding sequence and the acoustic feature sequence, and input the generated three-dimensional acoustic features into the decoder to obtain a decoding sequence. The decoding sequence is input into a connectionist temporal classification model (CTC) after being normalized by a linear layer and a Softmax layer. The input of the CTC model is the feature sequence of the audio signal, and the output sequence is the label sequence of words or characters, to obtain a character prediction result; determine the third text information according to the character prediction result.
[0035] Among them, the encoder has six identical neural network stacks. In this stack, the first sub-layer of each layer is composed of a residual connection layer and a multi-head attention network layer, and the second sub-layer is followed by a fully connected feed-forward neural network after layer normalization. The third layer adds the outputs of the first layer and the second layer and then performs layer normalization processing. The calculation formulas of the residual connection layer and layer normalization are as follows: ; ; ; ; Among them, b and g respectively represent bias parameters and gate parameters with the same dimension, represents the weight of the recurrent hidden layer, represents the weight of the line from the input layer to the hidden layer.
[0036] The decoder consists of 6 identical neural network layers, and each layer of this decoder has three sub-layers. The first sub-layer consists of a residual connection layer and masked multi-head attention. The second sub-layer consists of a residual connection layer, a normalization layer, and a multi-head attention layer. The input of this layer is composed of the output of the first layer and the output of the encoder. The third sub-layer consists of two layer normalizations, a residual connection layer, and a feed-forward layer.
[0037] Among them, in the three-dimensional acoustic features, Q is the query vector, K is the key vector, and V is the value vector. The multi-head attention can not only process the value vector V, but also process the values of the query vector Q and the key vector K at the same time. The multi-head attention can capture different positions of the subspace representation. The calculation formula of the multi-head attention is as follows: ; ; ; Among them, , SDPA is the scaled dot-product attention mechanism. SDPA Due to its low time complexity and space complexity, it is used in Transformer. Since d k the larger the dimension, the greater the computational cost of the dot product, so the Softmax function is used to reduce the gradient. The parameter matrices of the projection , , .
[0038] The CTC module is used to solve the problem that the training data in speech recognition needs to be pre-segmented and post-processed for the label sequence. The CTC calculation formula for the character sequence is as follows: ; ; Among them, is the activation function; is the function for the linear layer operation used to transform the hidden layer vector; is the function for capturing all the input and output vectors of the hidden layer; , and are the state transition probability, the language model probability of Chinese characters, and the prior probability of the state respectively; X is the input feature sequence, usually the speech signal representation after preprocessing (such as acoustic feature extraction, position encoding, etc.); z tLet \(X\) be the model output probability distribution at time step \(t\), generated by the Softmax layer; \(p(Z)\) is the prior probability of the state, usually fixed or pre - calculated by statistical methods; \(p(C)\) is the language model probability, representing the prior probability of the character sequence \(C\), usually provided by an external language model; \(C\) is the target character sequence, which is the output that the model needs to predict, and \(p(C|Z)\) is the posterior probability of the character sequence \(C\) given the input feature \(Z\).
[0039] In some embodiments, at least one of the above - mentioned modal data includes text data, and the text information includes fourth text information; after obtaining at least one modal data in step 101 above and before storing the target text in the text information into the target database in step 104, it further includes: Perform text segmentation on the text data to obtain target phrases; in response to the target phrases not satisfying the preset regular expression rules, determine the fourth text information according to the target phrases.
[0040] In a specific implementation, obtain the preset regular expression rules. For example, according to the characteristics of government affairs information, configure the regular expressions for the required information, such as identity number information, name information, home address information, etc. Obtain the text data, perform text segmentation on the text data to get target phrases, match the target phrases with the regular expression rules. If the target phrases do not satisfy the preset regular expression rules, then determine the fourth text information according to the target phrases.
[0041] In some embodiments, after performing text segmentation on the text data to obtain target phrases as described above, it further includes: In response to the target phrases satisfying the preset regular expression rules, store the target phrases into the target database.
[0042] In some embodiments, in step 104 above, storing the target text in the text information into the target database includes: Input the text information into a target information extraction model to obtain the target text in the text information; the text information extraction model includes an input layer, a convolutional layer, a pooling layer, a fully - connected layer, and an output layer. The input layer is used to obtain the text information, the convolutional layer is used to extract the text features of the text information, the pooling layer is used to reduce the dimension of the text features and select key information, the fully - connected layer is used to perform feature fusion on the key information to obtain a text feature vector, and the output layer is used to determine the target text based on the text feature vector; store the target text into the target database.
[0043] In an exemplary embodiment, the above-mentioned first text information, second text, third text, and fourth text are input into a target information extraction model, and key information is extracted through this model to obtain the target information in the text information. The text information extraction model includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer.
[0044] For text information with an input length of 𝑛, the convolutional layer learns text features by convolving the text input vector using h different-sized sliding windows, and the convolutional feature values are obtained through the convolutional kernel at position 𝑖. The calculation method of the convolutional layer is as follows: ; where k is the dimension of the word vector corresponding to each word in the text sequence, w is the convolutional kernel with a dimension of , represents the sliding window composed of the first row of the input matrix. b is the bias parameter, f is the non-linear mapping function. The maximum pooling strategy is used to filter out a maximum eigenvalue from each sliding window, and the formula for the maximum pooling layer is: ; All the combined eigenvalues are concatenated together through the fully connected layer to obtain the high-level feature vector of the text, and the calculation method is: ; where n is the number of words in the text sequence, C is the obtained text feature vector with a dimension of . After the operations of the convolutional layer and the pooling layer, the fully connected neural network layer is connected to classify the extracted message text and extract the required key information. The output layer of the classification uses the feature vector output after the text feature extraction layer, and after obtaining the text feature vector , the text of the government affairs input information is classified through the Softmax classifier, and the calculation formula of the Softmax classifier is: ; where represents that the predicted text information belongs to the category of the required target text and does not belong to the target text required by the target database, W is the weight matrix, b is the bias.
[0045] In an exemplary embodiment, taking the target database as a government affairs platform as an example, multi-modal data is input through an information input system such as a camera and a scanner. The multi-modal data includes image data, voice data, and text data; The image data passes through a fuzzy image information intelligent extraction module. The fuzzy image information intelligent extraction module is used to determine a clear text area and a fuzzy text area according to the text features in the image data; extract the first text information of the clear text area, and divide the fuzzy text area into multiple image blocks. Based on the clear image blocks that match the fuzzy image blocks in the multiple image blocks, perform feature repair processing on the fuzzy image blocks to obtain a repaired text area, extract the second text information of the repaired text area, and then perform text conversion of the image data; The voice data passes through a complex voice information intelligent extraction module. The complex voice information intelligent extraction module is used to determine a position encoding sequence according to the voiceprint features in the voice data; input the sum of the voiceprint features and the position encoding sequence into an encoder to obtain an acoustic feature sequence; input the sum of the position encoding sequence and the acoustic feature sequence into a decoder to obtain a decoded sequence; the decoded sequence is input into a connectionist temporal classification model after normalization processing. The connectionist temporal classification model predicts the characters corresponding to the decoded sequence to obtain a character prediction result; according to the character prediction result, determine the third text information, and then perform text conversion of the voice data; The text data passes through a text information specification discrimination module. The text information specification discrimination module is used to divide the text data to obtain target phrases, and determine whether the target phrases meet the preset regular expression rules. If the regular expression match review passes, store the target phrases in the government affairs platform; if the regular expression match review fails, determine the fourth text information according to the target phrases; Input the above first text information, second text information, third text information, and fourth text information into a key information extraction module. Through the key information extraction module, extract key information and perform information division to obtain a target text, and input the target text into the government affairs platform.
[0046] Figure 6 Shows a schematic structural diagram of an information storage device for multi-modal data provided by some embodiments of the present application. The information storage device can implement all or part of the content in the embodiments shown as follows. The information storage device 600 includes: Figure 1 As shown in the embodiment, the data acquisition module 610 is used to acquire at least one type of modal data, and the at least one type of modal data includes image data; The area determination module 620 is used to determine a clear text area and a fuzzy text area according to the text features of the image data; An information extraction module 630 is configured to extract first text information of the clear text region, divide the blurred text region into multiple image blocks, perform feature restoration processing on the blurred image blocks in the multiple image blocks based on the clear image blocks in the multiple image blocks to obtain a restored text region, and extract second text information of the restored text region; An information storage module 640 is configured to store the target text in the text information into a target database, where the text information includes the first text information and the second text information.
[0047] In some embodiments, when the information extraction module 630 is used to perform feature restoration processing on the blurred image blocks based on the clear image blocks that match the blurred image blocks in the multiple image blocks to obtain a restored text region, it is specifically configured to: For each image block, determine the relative gray mean value of the image block according to the gray value of each pixel in the image block and the pixel gray mean value of the image block; Based on the pixel gray mean value of the blurred text region and the relative gray mean values of the multiple image blocks, determine the perceptual hash values of the respective image blocks; Based on the perceptual hash values of the respective image blocks, determine the clear image blocks that match the blurred image blocks in the multiple image blocks; Perform feature restoration processing on the blurred image blocks according to the clear image blocks to obtain a restored text region.
[0048] In some embodiments, the at least one type of modal data includes voice data, and the text information includes third text information; the information storage device 600 further includes: A voice information extraction module is configured to determine a position encoding sequence according to the voiceprint feature in the voice data; obtain an acoustic feature sequence by inputting the sum of the voiceprint feature and the position encoding sequence into an encoder; obtain a decoded sequence by inputting the sum of the position encoding sequence and the acoustic feature sequence into a decoder; the decoded sequence is input into a connectionist temporal classification model after being normalized, and the connectionist temporal classification model predicts the characters corresponding to the decoded sequence to obtain a character prediction result; determine the third text information according to the character prediction result.
[0049] In some embodiments, the at least one type of modal data includes text data, and the text information includes fourth text information; the information storage device 600 further includes: A text information extraction module is configured to perform text division on the text data to obtain target phrases; in response to the target phrases not satisfying a preset regular expression rule, determine the fourth text information according to the target phrases.
[0050] In some embodiments, the text information extraction module is further configured to store the target phrase in the target database in response to the target phrase satisfying a preset regular expression rule.
[0051] In some embodiments, when the information storage module 640 is configured to store the target text in the text information in the target database, it is specifically configured to: Input the text information into a target information extraction model to obtain the target text in the text information; the text information extraction model includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input layer is used to obtain the text information, the convolutional layer is used to extract the text features of the text information, the pooling layer is used to reduce the dimension of the text features and select key information, the fully connected layer is used to perform feature fusion on the key information to obtain a text feature vector, and the output layer is used to determine the target text based on the text feature vector; store the target text in the target database.
[0052] The embodiment of the present application provides an information storage device for multi-modal data, including a data acquisition module, a region determination module, an information extraction module, and an information storage module; the data acquisition module acquires at least one type of modal data, and the at least one type of modal data includes image data; the region determination module determines a clear text region and a blurred text region according to the text features of the image data; the information extraction module extracts the first text information of the clear text region, and divides the blurred text region into multiple image blocks, and performs feature repair processing on the blurred image blocks in the multiple image blocks based on the clear image blocks in the multiple image blocks to obtain a repaired text region, and extracts the second text information of the repaired text region; the information storage module stores the target text in the text information in the target database, and the text information includes the first text information and the second text information. In this way, during the information storage process of image data, the blurred text region is divided into multiple image blocks, and the clear image blocks matching the blurred image blocks are used to perform feature repair on it, which can solve the problem of incorrect text information recognition caused by image blurring, reduce the noise and interference in the target text, and improve the information storage efficiency.
[0053] Figure 7The figure shows a schematic diagram of the hardware structure of the electronic device provided by the embodiments of the present application. Referring to this figure, at the hardware level, the electronic device 700 includes a processor 710. Optionally, it includes an internal bus 720, a network interface 730, and a memory 740. Among them, the memory 740 may include a memory 741, such as a high-speed random access memory (Random-Access Memory, RAM), and may also include a non-volatile memory 742 (non-volatile memory), such as at least one disk memory, etc. Of course, this electronic device may also include other hardware required for other services.
[0054] The processor 710, the network interface 730, and the memory can be interconnected through the internal bus 720. The internal bus 720 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a bidirectional arrow is used in this figure, but it does not mean that there is only one bus or one type of bus.
[0055] The memory 740 stores programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory 740 may include a memory 741 and a non-volatile memory 742, and provide instructions and data to the processor 710.
[0056] The processor 710 reads the corresponding computer program from the non-volatile memory 742 into the memory and then runs it, forming a device for locating the target user at the logical level. The processor 710 executes the program stored in the memory and specifically executes: Figure 1 The methods disclosed in the illustrated embodiments implement the functions and beneficial effects of the various methods described in the foregoing method embodiments, and will not be elaborated herein.
[0057] The above is as described in the present application Figure 1The method disclosed in the illustrated embodiment can be applied to or implemented by the processor 710. The processor 710 may be an integrated circuit chip with signal processing capabilities. During implementation, the steps of the above method can be completed through the integrated logic circuit in hardware or instructions in software form in the processor 710. The above-mentioned processor 710 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps of the above method.
[0058] This computer device can also execute the various methods described in the foregoing method embodiments and achieve the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be elaborated here.
[0059] Of course, in addition to the software implementation manner, the electronic device of the present application does not exclude other implementation manners, such as a logic device or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and can also be hardware or a logic device.
[0060] The embodiments of the present application also propose a computer-readable storage medium. The computer-readable medium stores one or more programs. When the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device is caused to execute Figure 1 the method disclosed in the illustrated embodiment and achieve the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be elaborated here.
[0061] Among them, the computer-readable storage medium includes a read-only memory (ROM for short), a random access memory (RAM for short), a magnetic disk, an optical disc, etc.
[0062] Furthermore, the embodiment of the present application also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the following process is implemented: Figure 1 The method disclosed in the illustrated embodiment realizes the functions and beneficial effects of the various methods described in the foregoing method embodiments, which will not be elaborated herein.
[0063] The embodiment of the present application can be applied to various scenarios of electronic device collaboration or interconnection, including: collaboration and interconnection between a mobile phone and a laptop / tablet computer; collaboration and interconnection between a mobile terminal and a smart TV / display; collaboration and interconnection between a mobile phone or a tablet computer and an in-vehicle entertainment system; collaboration and interconnection between a mobile terminal and a smart conference system, etc. Thus, it can meet the diverse scenario requirements of users in smart home, smart office, smart travel, etc.
[0064] In summary, the above are only the preferred embodiments of the present application, and do not limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
[0065] The systems, devices, modules or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.
[0066] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0067] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.
[0068] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.
Claims
1. An information storage method for multimodal data, characterized in that, Including: Obtain at least one modality data, where the at least one modality data includes image data; Determine a clear text area and a blurred text area according to text features in the image data; Extract first text information of the clear text area, and divide the blurred text area into multiple image blocks, perform feature repair processing on the blurred image blocks based on clear image blocks that match the blurred image blocks in the multiple image blocks to obtain a repaired text area, and extract second text information of the repaired text area; Store target text in the text information into a target database, where the text information includes the first text information and the second text information.
2. The method according to claim 1, wherein The performing feature repair processing on the blurred image blocks based on clear image blocks that match the blurred image blocks in the multiple image blocks to obtain a repaired text area includes: For each image block, determine a relative gray mean value of the image block according to gray values of each pixel in the image block and a pixel gray mean value of the image block; Determine a perceptual hash value of each image block based on the pixel gray mean value of the blurred text area and the relative gray mean values of the multiple image blocks; Determine clear image blocks that match the blurred image blocks in the multiple image blocks based on the perceptual hash values of each image block; Perform feature repair processing on the blurred image blocks according to the clear image blocks to obtain a repaired text area.
3. The method according to claim 1, characterized in that The at least one modality data includes voice data, and the text information includes third text information; after obtaining the at least one modality data and before storing target text in the text information into the target database, it further includes: Determine a position encoding sequence according to voiceprint features in the voice data; Obtain an acoustic feature sequence by inputting the sum of the voiceprint features and the position encoding sequence into an encoder; Obtain a decoded sequence by inputting the sum of the position encoding sequence and the acoustic feature sequence into a decoder; The decoded sequence is input into a connectionist temporal classification model after being normalized, and the connectionist temporal classification model predicts characters corresponding to the decoded sequence to obtain a character prediction result; Determine the third text information according to the character prediction result.
4. The method according to claim 1, wherein The at least one modality data includes text data, and the text information includes fourth text information; after obtaining the at least one modality data and before storing target text in the text information into the target database, it further includes: Perform text division on the text data to obtain a target phrase; In response to the target phrase not satisfying a preset regular expression rule, determine the fourth text information according to the target phrase.
5. The method according to claim 4, characterized in that After performing text division on the text data to obtain a target phrase, it further includes: In response to the target phrase satisfying the preset regular expression rule, store the target phrase into the target database.
6. The method according to any one of claims 1 to 5, characterized in that The storing target text in the text information into the target database includes: By inputting the text information into the target information extraction model, the target text in the text information is obtained; the text information extraction model includes an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The input layer is used to obtain the text information, the convolutional layer is used to extract the text features of the text information, the pooling layer is used to reduce the dimension of the text features and select key information, the fully connected layer is used to perform feature fusion on the key information to obtain a text feature vector, and the output layer is used to determine the target text based on the text feature vector; Store the target text in the target database.
7. An information storage device for multimodal data, characterized in that, Including: A data acquisition module for acquiring at least one type of modal data, where the at least one type of modal data includes image data; A region determination module for determining a clear text region and a blurred text region according to the text features of the image data; An information extraction module for extracting the first text information of the clear text region, and dividing the blurred text region into a plurality of image blocks, performing feature repair processing on the blurred image blocks in the plurality of image blocks based on the clear image blocks in the plurality of image blocks to obtain a repaired text region, and extracting the second text information of the repaired text region; An information storage module for storing the target text in the text information in the target database, where the text information includes the first text information and the second text information.
8. An electronic device, characterized in that, The electronic device includes a processor and a memory. The memory stores a program or instruction that can run on the processor. When the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium, characterized in that, A program or instruction is stored on the computer-readable storage medium. When the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to execute the steps of the method according to any one of claims 1 to 6.