Image voice multi-modal data analysis method, device and equipment and storage medium

Abstract

The application discloses an image voice multimodal data analysis method, device and equipment and a storage medium. According to the technical scheme of the application, two sets of text data are obtained by respectively identifying images and audio, the picture content is automatically bound with the corresponding pronunciation, the preset audio shape comparison mapping table is used to simultaneously carry out character level equivalent comparison and semantic level equivalent comparison of the two sets of text data, different conflict type scenes are accurately distinguished, different processing strategies are adopted for different conflict types, and the local analysis result is generated according to the confidence, so that the audio picture data automatic association is realized, and the accurate text analysis result can be obtained by capturing the picture.

Description

Technical Field

[0001] This invention relates to the field of image and speech data processing technology, and in particular to a method, apparatus, device and storage medium for parsing multimodal image and speech data. Background Technology

[0002] With the popularization of portable smart recording devices and translation terminals, multimodal translation and parsing technologies that combine OCR (Optical Character Recognition) image recognition and ASR (Automatic Speech Recognition) speech recognition have been widely applied.

[0003] Currently, OCR translation devices, multimodal translation terminals, and smart voice recorders can already achieve basic functions such as photo recognition and audio transcription. However, most existing technologies rely on single-modal recognition of images or speech, resulting in poor accuracy in outdoor blurry environments and noisy environments; images and audio need to be manually captured separately by the user, and it is impossible to automatically bind the corresponding audio segment at the moment of taking a photo; there is a lack of local cross-modal cross-validation, and cloud-based joint audio-image reasoning and correction is also impossible; at the same time, there is a lack of dedicated output capabilities such as pronunciation annotation and difference highlighting for foreign language learners, and usability is insufficient when network conditions are poor. In practical scenarios such as address recognition, existing methods cannot fully utilize the complementary advantages of image and speech information, resulting in cumbersome operation and insufficient reliability of recognition results.

[0004] Therefore, there is an urgent need for a method for parsing multimodal image and speech data that fully utilizes the complementary advantages of image and speech information. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for parsing multimodal data of images and audio. By jointly parsing multimodal data that automatically binds images and audio, accurate text parsing results can be obtained simply by capturing an image.

[0006] In a first aspect, embodiments of the present invention provide a method for parsing multimodal image and speech data, the method comprising: In response to a photo capture command, acquire the target image and the photo capture timestamp of the target image, as well as target audio data of a preset time length centered on the photo capture timestamp; The target image is identified to obtain first text data, and the first confidence level of each character in the first text data and the first average confidence level of the first text data as a whole are determined; the target audio data is identified to obtain second text data, and the second confidence level of each character in the second text data and the second average confidence level of the second text data as a whole are determined. Based on a preset phonetic-graphic mapping table, semantic-level equivalence comparison and character-level equivalence comparison are performed on the first text data and the second text data to determine the semantic-level judgment result and the character-level judgment result of the first text data and the second text data. Based on the character-level and semantic-level determination results, it is determined whether there is a conflict between the first text data and the second text data, and a conflict determination result is obtained. Based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data.

[0007] Secondly, embodiments of the present invention also provide an image and speech multimodal data parsing apparatus, the apparatus comprising: The data acquisition module is used to respond to the photo capture command to acquire the target image and the photo capture timestamp of the target image, as well as the target audio data of a preset time length centered on the photo capture timestamp; The data recognition and confidence determination module is used to recognize the target image to obtain first text data, determine the first confidence of each character in the first text data and the first average confidence of the first text data as a whole; and to recognize the target audio data to obtain second text data, determine the second confidence of each character in the second text data and the second average confidence of the second text data as a whole. The equivalence comparison module is used to perform semantic-level equivalence comparison and character-level equivalence comparison on the first text data and the second text data based on a preset phonetic-graphic mapping table, and to determine the semantic-level judgment result and the character-level judgment result of the first text data and the second text data. The conflict determination module is used to determine whether there is a conflict between the first text data and the second text data based on the character-level determination result and the semantic-level determination result, and to obtain the conflict determination result. The parsing result generation module is used to generate a local parsing result based on the conflict determination result, combined with the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, and based on the first text data and the second text data.

[0008] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image and speech multimodal data parsing method as described in any of the embodiments of the present invention.

[0009] Fourthly, embodiments of the present invention also provide a storage medium for storing computer-executable instructions, which, when executed by a computer processor, are used to perform the image and speech multimodal data parsing method as described in any of the embodiments of the present invention.

[0010] The technical solution of this invention obtains two sets of text data based on image and audio recognition respectively, automatically binds the image content with the corresponding pronunciation, and simultaneously performs character-level and semantic-level equivalence comparisons of the two sets of text data through a preset sound-image mapping table. This accurately distinguishes scenarios with different conflict types, adopts differentiated processing strategies for different conflict types, and generates local parsing results based on confidence levels, thereby achieving automatic association of sound and image data. Accurate text parsing results can be obtained simply by capturing an image.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of an image and speech multimodal data parsing method provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the structure of an image and speech multimodal data parsing device provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device that implements the image and speech multimodal data parsing method of this invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1 This is a flowchart illustrating an image-speech multimodal data parsing method according to Embodiment 1 of the present invention. This embodiment is applicable to image-speech multimodal data parsing. The method can be executed by an image-speech multimodal data parsing device, which can be implemented in hardware and / or software. This device can be configured in any electronic device with network communication and computing capabilities. Figure 1 As shown, the method includes: S110. In response to the photo capture command, acquire the target image and the photo capture timestamp of the target image, as well as the target audio data of a preset time length centered on the photo capture timestamp; In this embodiment, the photo-taking command is a trigger command used to initiate the image acquisition action. It can be a hardware trigger signal generated by the user pressing the physical photo-taking button on the device, or a software command generated by clicking the virtual photo-taking button on the screen. The photo-taking command itself does not control the recording; it only serves as a trigger source for the time anchor, used to mark the moment when image capture occurs. The target image is a static image file captured after responding to the photo-taking command. In this embodiment, it is a 1920×1080 resolution JPEG image, typically a road sign, doorplate, foreign language sign, or similar object photographed by the user. The photo-taking timestamp of the target image is used to mark the precise time stamp T0 at the moment when the target image was acquired, serving as the time anchor for establishing the association between the image and audio. The photo-taking timestamp can be used to locate and extract audio segments in the audio loop buffer and will also be carried in the multimodal data packet for cloud matching and local database record indexing.

[0017] The target audio data is a PCM format audio segment extracted from the audio loop buffer, centered on the photo's timestamp. The preset duration is a pre-configured duration, defaulting to 10 seconds before and after the photo's capture time, for a total duration of 20 seconds. This target audio data includes ambient sounds from the surrounding environment at the time of the photo's capture, typically including the user's reading of text on the image or asking for pronunciation, thus automatically binding the image content with the corresponding pronunciation.

[0018] Specifically, this method utilizes a device for continuous audio acquisition. The microphone continuously captures ambient sound, and the audio data is stored in an audio circular buffer. This circular buffer continuously stores audio data from the most recent period, eliminating the need to wait for a photo to be taken before recording begins. Upon receiving a photo-taking command, the device performs image acquisition, capturing a static target image and recording the moment the image is captured, generating a timestamp. Then, the device control unit directly reads an audio segment from the pre-buried audio circular buffer, centered on the timestamp and extending for a preset time before and after it, as the target audio data. This eliminates the need for manual recording by the user, automatically establishing a spatiotemporal correspondence between the image and surrounding audio, overcoming the cumbersome process of requiring separate photos and manual selection of recording segments in existing technologies.

[0019] S120. Recognize the target image to obtain first text data, and determine the first confidence level of each character in the first text data and the first average confidence level of the first text data as a whole; recognize the target audio data to obtain second text data, and determine the second confidence level of each character in the second text data and the second average confidence level of the second text data as a whole.

[0020] In this embodiment, the first text data is the text data output by the local OCR engine from the target image recognition. It is the text content parsed from the target image, such as the street address "Nishi-Shinjuku 2-B-1". The first text data originates from the visual image and represents the printed characters on the image. The second text data is the text data output by the local ASR engine from the target audio data speech recognition. It is the text obtained from the user's reading and environmental speech transcription in the audio, and can be kanji, kana, letters, numbers, etc., representing the characters corresponding to the pronunciation expressed by the speech.

[0021] The first confidence level is the independent confidence level of each character in the OCR output. For each character in the first text data, a confidence level value of 0-100% is output, representing the reliability of the local OCR engine in correctly recognizing that character. For example, the character "B" has a confidence level of 70%, and the character "1" has a confidence level of 95%. The second confidence level is the independent confidence level of each character in the ASR output. For each character in the second text data, a confidence level of 0-100% is output, representing the reliability of the ASR in correctly recognizing that character.

[0022] The first average confidence score is the overall confidence score of the entire first text data, obtained by weighting the first confidence scores of each character, and represents the overall reliability of the OCR recognition result for the entire text segment. The second average confidence score is the overall confidence score of the entire second text data, obtained by weighting the second confidence scores of each character, and represents the overall reliability of the entire speech-to-text result.

[0023] Among them, the confidence score at the character level is used for character-by-character comparison of conflict positions during subsequent cross-modal verification, and the average confidence score of the entire text is used to evaluate the overall reliability of the entire recognition result.

[0024] Specifically, the target image is subjected to optical character recognition using local OCR, and the first text data obtained from the recognition is output. Simultaneously, an independent first confidence score for each character is output. Then, a weighted average of the entire text is calculated based on the first confidence scores of each character to obtain the first average confidence score. The target video data is subjected to automatic speech recognition using local ASR, and the second text data obtained from the transcription is output. Simultaneously, an independent second confidence score for each character is output. Then, a weighted average of the entire text is calculated based on the second confidence scores of each character to obtain the second average confidence score.

[0025] In this embodiment, the captured images and automatically extracted audio are recognized in parallel in a local offline environment of the device. At the same time, the recognized text, character-level confidence and overall weighted average confidence of the text are output. Two layers of confidence index are distinguished: single character granularity and whole text granularity. This realizes the quantitative output of image visual information and audio speech information, lays a complete data foundation for image and audio multimodal cross-validation, and overcomes the defects of the existing technology that only outputs the recognized text and lacks hierarchical confidence information, thus failing to carry out local automatic error correction.

[0026] S130. Based on a preset phonetic-image mapping table, perform semantic-level equivalence comparison and character-level equivalence comparison on the first text data and the second text data to determine the semantic-level judgment result and the character-level judgment result of the first text data and the second text data.

[0027] In this embodiment, a preset phonetic-shape mapping table is used to store the mapping relationship between glyphs and corresponding phonetic text. The preset phonetic-shape mapping table includes at least the mapping relationship between Japanese kanji and kana, the mapping relationship between English numeral pronunciation and numerals, and the mapping relationship between Chinese pinyin and kanji.

[0028] The preset phonetic-graphic mapping table is a locally stored mapping table that stores the phonetic-graphic equivalence correspondences between characters and proper nouns in different languages. Character-level equivalence comparison aligns the first and second text data character by character, and then uses the preset phonetic-graphic mapping table to determine whether characters at each position are literally identical, phonetically and graphically equivalent, or conflicting. Semantic-level equivalence comparison, from the overall semantic dimension of the entire text, uses the vocabulary entries in the mapping table to determine whether the first and second text data point to the same semantic object.

[0029] The character-level determination result is the comparison status of each character position in the first and second text data, and may also include the set of conflicting character positions. The semantic-level determination result is the semantic equivalence comparison status of the first and second text data, used to characterize whether the two texts are semantically equivalent or semantically conflicting as a whole.

[0030] In this embodiment, character-level and semantic-level comparisons are used to distinguish between real conflicts and phonetic-graphic equivalence, resulting in semantic-level and character-level judgment results.

[0031] Specifically, the character-level equivalence comparison process is as follows: align the first text data with the second text data character sequences, extract the corresponding character at each position according to the character index. If the two characters are identical in meaning, the characters at that position are considered identical; if they are not identical, query the preset phonetic-shape mapping table. If an equivalence relationship exists between the two characters in the preset phonetic-shape mapping table, the characters at that position are considered phonetically and shape-equivalent; otherwise, the characters at that position are considered conflicting, and the index of the conflicting character is recorded. After traversing all character positions, the comparison status of each position and the set of conflicting positions are summarized to generate the character-level judgment result.

[0032] Specifically, the semantic equivalence comparison process is as follows: the first text data and the second text data are segmented into word units to obtain lexical units. The core proprietary entity words in the two texts are extracted, and the lexical equivalence entries in the preset phonetic-text mapping table are queried to determine the objective objects to which the two texts refer. When the core proprietary entities of the two texts refer to the same object after verification by the mapping table, they are determined to be semantically equivalent. When the core entities corresponding to the two texts are different and express different meanings, they are determined to be semantically conflicting, and the semantic-level judgment result is output.

[0033] In this embodiment, the simple direct string comparison method is abandoned. Instead, a local preset sound-shape mapping table is used to perform dual equivalence verification at both the character granularity and semantic granularity. This can identify recognition conflicts at the position of a single character, as well as equivalent scenarios where the characters have different shapes but the same pronunciation and semantics. This avoids misjudging correct sound-shape equivalence as a conflict. At the same time, it distinguishes between local character conflicts and overall semantic conflicts, providing a reliable basis for subsequent branch logic and ensuring the accuracy of local cross-modal verification.

[0034] S140. Based on the character-level determination result and the semantic-level determination result, determine whether there is a conflict between the first text data and the second text data, and obtain the conflict determination result.

[0035] In this embodiment, the conflict determination result is used to characterize whether there is a conflict between the first text data and the second text data. The conflict determination result is determined based on the character-level determination result and the semantic-level determination result.

[0036] Specifically, conflict determination can be completed by combining the dual verification results at the character granularity and semantic granularity. The semantic-level determination result is prioritized. When the semantic-level determination result indicates an overall semantic conflict, a "global semantic conflict" result is directly obtained, indicating that the first and second text data express completely different content, and local correction is not possible. When the semantic-level determination result indicates semantic equivalence, further judgment is made by combining the conflict position set in the character-level determination result. If the conflict position set is not empty, the conflict determination result is a local character conflict, representing that the overall meaning is consistent, but only some character position recognition results are contradictory, which can be corrected locally based on the confidence priority rule. If the conflict position set is empty, the conflict determination result is no conflict, representing that the two recognition results corroborate each other.

[0037] In this embodiment, instead of simply judging conflicts based on whether a single character is contradictory, a hierarchical conflict judgment is made by combining the overall semantic information. This distinguishes between local character conflicts that can be repaired locally and overall semantic conflicts that cannot be handled locally, thereby achieving hierarchical conflict diversion and improving the reliability of the entire multimodal data parsing process.

[0038] Optionally, based on the character-level determination result and the semantic-level determination result, it is determined whether there is a conflict between the first text data and the second text data, and a conflict determination result is obtained, including: If the semantic level determination result is equivalent and the character level determination result is equivalent, then the conflict determination result of the first text data and the second text data is determined to be non-conflicting. If the semantic level determination result is not equivalent, then the conflict determination result of the first text data and the second text data is determined to be a semantic conflict. If the character-level determination result is not equivalent, then the conflict determination result of the first text data and the second text data is determined to be a character conflict.

[0039] In this embodiment, the conflict determination result is divided into no conflict, semantic conflict, or character conflict.

[0040] Specifically, semantic-level judgment is performed first. When the semantic-level judgment result is not equivalent, it is directly judged as a semantic conflict, indicating that the two texts describe different objective objects, which is a conflict at the overall level. Only when the semantic-level judgment result is equivalent, is the character-level judgment result considered. When both semantic and character-level judgments are equivalent, it is judged as no conflict, meaning that the OCR and ASR recognition results can corroborate each other. When semantics are equivalent but character-level judgments are not equivalent, it is judged as a character conflict, that is, the overall meaning is consistent, but the recognition results of some character positions are contradictory.

[0041] In this embodiment, a hierarchical judgment is used to distinguish between overall semantic conflicts and local character conflicts, thereby achieving traffic splitting for different conflict types.

[0042] S150. Based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data.

[0043] In this embodiment, the local parsing result is the text parsing result generated based on the first text data and the second text data, according to the conflict determination result combined with the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level.

[0044] Specifically, different processing procedures are executed based on different conflict determination results. For example, in the case of semantic conflict, the original recognized text is retained and no local correction is performed; in the case of character conflict, the confidence of the corresponding characters is compared at the conflict position, and if the confidence difference condition is met, the low-confidence character is corrected with the high-confidence character; in the case of no conflict, the first text data is directly adopted and the generated local parsing result is obtained.

[0045] Optionally, based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data, including: If the conflict determination result is no conflict, and both the first average confidence level and the second average confidence level are greater than the preset confidence threshold, then the first text data is used as the local parsing result.

[0046] In this embodiment, the preset confidence threshold is a pre-set confidence threshold used to determine whether the text data recognition result is reliable.

[0047] If both conditions are met simultaneously: first, the conflict determination result is non-conflicting, meaning the first and second text data are semantically equivalent and all character positions are consistent or phonetically equivalent, with the two recognition results mutually corroborating each other; second, both the first and second average confidence scores are greater than the preset confidence threshold, indicating that the overall credibility of both OCR and ASR recognition results is high. When both conditions are met, the first text data from the target image is selected as the local parsing result. In practical applications, there is no need to upload it to the cloud for multimodal fusion parsing; the local parsing result can be directly used for subsequent educational mode display, saving network traffic and cloud computing resources.

[0048] In this embodiment, under the premise that the two sets of recognition results do not conflict and the overall confidence level is high enough, the image-side recognition result is directly adopted, which can realize local offline closed loop, avoid unnecessary cloud requests, and balance the use of response speed and resource consumption.

[0049] Optionally, based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data, further including: If the conflict determination result is a semantic conflict, then the first text data and the second text data are used together as the local parsing result.

[0050] In this embodiment, when the conflict determination result is a semantic conflict, it indicates that the first text data and the second text data refer to different objective objects in terms of overall semantics. This is a contradiction at the overall level, and the local system cannot distinguish the correctness of the two recognition results. Therefore, no local character correction processing is performed, and the original first and second text data are saved together as the local parsing result. In practical applications, this local parsing result can temporarily present the two recognized contents to the user locally, and simultaneously upload them to the cloud along with the target image and target audio data. This provides a reference for cloud-based multimodal fusion inference, and the cloud combines the original image and audio signal to complete joint parsing.

[0051] In this embodiment, when an overall semantic conflict occurs, no blind selection is made. The original recognition results of both paths are fully preserved. This ensures that the original results are available for viewing when the device is offline, and provides complete reference information for cloud-based reasoning, thus avoiding the risk of result distortion caused by local error correction.

[0052] Optionally, based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data, further including: If the conflict determination result is a character conflict, then the position of the conflicting character is determined, and the first conflicting character in the first text data and the second conflicting character in the second text data are determined according to the position of the conflicting character. Determine the first confidence level of the first conflicting character and the second confidence level of the second conflicting character, and determine the maximum confidence level between the first confidence level and the second confidence level; A reference character is determined from the first and second conflicting characters based on the maximum confidence level; The first conflicting character at the position of the conflicting character in the first text data is replaced according to the reference character, and a local parsing result is generated based on the replaced first text data.

[0053] In this embodiment, the conflicting character position is the index position of a character in the first text data and the second text data where characters are not equivalent. The first conflicting character is the character below the corresponding conflicting character in the first text data, and the second conflicting character is the character below the corresponding conflicting character in the second text data. For example, comparing the first text data and the second text data sequentially according to their character indices, the conflicting character position index is 5. The first conflicting character corresponding to this position is the character B, and the second conflicting character is the character 8.

[0054] Furthermore, extract the confidence level of each character at the conflict position, the first confidence level corresponding to the first conflicting character, and the second confidence level corresponding to the second conflicting character, and compare them to obtain the maximum confidence level between the two.

[0055] The reference character is either the first or second conflicting character corresponding to the maximum confidence level. If the first conflicting character has a higher confidence level, the reference character is the first conflicting character. If the second conflicting character has a higher confidence level, the reference character is the second conflicting character.

[0056] Furthermore, the reference character is used to replace the original first conflicting character at the conflicting character position in the first text data; characters in non-conflicting positions remain unchanged. After the replacement is completed, the modified first text data is used as the local parsing result. For example, if the first conflicting character B has a confidence level of 70% and the second conflicting character 8 has a confidence level of 88%, and the maximum confidence level is 88%, then the reference character is 8. The B at index 5 in the OCR result is replaced with 8, and the remaining characters remain unchanged. The text after the replacement is the local parsing result.

[0057] It should be noted that in practical applications, if the confidence difference is ≤15%, and the confidence of the two characters is close, then no replacement will be performed, and the original characters will be retained.

[0058] In this embodiment, for scenarios where the overall semantics are correct but only some characters are misrecognized, local correction is performed using single-character confidence. High-confidence recognition results are used to correct low-confidence characters, and local error correction is completed offline on the device without waiting for results to be returned from the cloud, which can improve the accuracy of parsing results in offline scenarios.

[0059] Optionally, based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, after generating a local parsing result based on the first text data and the second text data, the process includes: The target image, target audio data, and local parsing results are packaged into a multimodal data packet, and the multimodal data packet is uploaded to the cloud server. The target image and target audio data in the multimodal data packet are identified using a cloud server to obtain cloud-based parsing results; Receive the cloud parsing result sent by the cloud server, replace the local parsing result according to the cloud parsing result, and visualize the cloud parsing result.

[0060] In this embodiment, the multimodal data packet is a data packet that packages the target image, target audio data, and local parsing results. The cloud parsing result is the text parsing result obtained by using a cloud server to recognize the target image and target audio data in the multimodal data packet.

[0061] It should be noted that this step applies to situations where the conditions for cloud-free output are not met; the local parsing result is only a temporary result. The device encapsulates the original target image and target audio data along with the local parsing result into a multimodal data packet and uploads it to the cloud server. The uploaded local parsing result is only used as reference information for the cloud; the cloud will not directly accept the local parsing result, but will instead perform multimodal joint recognition inference based on the original target image and original target audio data to obtain a more accurate cloud parsing result.

[0062] Specifically, the cloud server can call the cloud multimodal fusion engine to perform cloud OCR recognition on the target image in the data packet and cloud ASR recognition on the target audio data. Then, it combines image features and audio features to perform multimodal joint reasoning fusion, overcoming the recognition defects caused by the limited computing power of local small models, and outputting a unified cloud parsing result. The cloud parsing result can be the original text, translation, and network address of the standard pronunciation audio of the recognized text book.

[0063] Furthermore, after receiving the cloud-based parsing results, the device replaces the original local parsing results with the cloud-based parsing results, completing the local data update and avoiding confusion caused by the coexistence of old local results and correct cloud-based results. The device also provides a visual output display of the cloud-based parsing results, specifically through the device's educational display mode, showing the original text, translation, pronunciation annotations, and other content to the user.

[0064] In this embodiment, a local-cloud collaborative working mechanism is constructed. The local output of the parsing results is prioritized to ensure response speed. Then, the original audio-visual data is uploaded and the cloud performs high-precision secondary inference. The authoritative cloud results replace the local results, thereby improving the reliability of the final output results.

[0065] The technical solution of this invention obtains two sets of text data based on image and audio recognition respectively, automatically binds image content with corresponding pronunciations, and simultaneously performs character-level and semantic-level equivalence comparisons of the two sets of text data using a preset sound-image mapping table. This accurately distinguishes scenarios with different conflict types and employs differentiated processing strategies for different conflict types, generating local parsing results based on confidence levels. For parsing results that do not meet the conditions for cloud-free processing, the device packages and uploads the original image, audio, and local parsing results to the cloud. The cloud-based multimodal fusion engine then re-infers the original audio-image data to generate authoritative cloud-based parsing results, which replace the local parsing results. This solves the technical shortcomings of traditional single-modal recognition, such as low accuracy, inability to automatically associate audio-image data, inability to distinguish between equivalence and real conflicts, and the inability to balance offline and high-precision processing. It achieves automatic association of audio-image data, allowing accurate text parsing results to be obtained simply by capturing an image.

[0066] Example 2 Figure 2 This is a schematic diagram of an image-speech multimodal data parsing device according to Embodiment 2 of the present invention. This embodiment is applicable to image-speech multimodal data parsing. The image-speech multimodal data parsing device can be implemented in hardware and / or software, and can be configured in any electronic device with network communication and computing capabilities. For example... Figure 2 As shown, the device includes: The data acquisition module 310 is used to acquire, in response to a photo capture command, a target image and a photo capture timestamp of the target image, as well as target audio data of a preset time length centered on the photo capture timestamp. The data recognition and confidence determination module 320 is used to recognize the target image to obtain first text data, determine the first confidence of each character in the first text data and the first average confidence of the first text data as a whole; and to recognize the target audio data to obtain second text data, determine the second confidence of each character in the second text data and the second average confidence of the second text data as a whole. The equivalence comparison module 330 is used to perform semantic-level equivalence comparison and character-level equivalence comparison on the first text data and the second text data based on a preset phonetic-graphic comparison mapping table, and to determine the semantic-level judgment result and the character-level judgment result of the first text data and the second text data. The conflict determination module 340 is used to determine whether there is a conflict between the first text data and the second text data based on the character-level determination result and the semantic-level determination result, and to obtain the conflict determination result. The parsing result generation module 350 is used to generate a local parsing result based on the conflict determination result, combined with the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, and based on the first text data and the second text data.

[0067] Optionally, based on the character-level determination result and the semantic-level determination result, it is determined whether there is a conflict between the first text data and the second text data, and a conflict determination result is obtained, including: If the semantic level determination result is equivalent and the character level determination result is equivalent, then the conflict determination result of the first text data and the second text data is determined to be non-conflicting. If the semantic level determination result is not equivalent, then the conflict determination result of the first text data and the second text data is determined to be a semantic conflict. If the character-level determination result is not equivalent, then the conflict determination result of the first text data and the second text data is determined to be a character conflict.

[0068] Optionally, based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data, including: If the conflict determination result is no conflict, and both the first average confidence level and the second average confidence level are greater than the preset confidence threshold, then the first text data is used as the local parsing result.

[0069] Optionally, based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data, further including: If the conflict determination result is a semantic conflict, then the first text data and the second text data are used together as the local parsing result.

[0070] Optionally, based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data, further including: If the conflict determination result is a character conflict, then the position of the conflicting character is determined, and the first conflicting character in the first text data and the second conflicting character in the second text data are determined according to the position of the conflicting character. Determine the first confidence level of the first conflicting character and the second confidence level of the second conflicting character, and determine the maximum confidence level between the first confidence level and the second confidence level; A reference character is determined from the first and second conflicting characters based on the maximum confidence level; The first conflicting character at the position of the conflicting character in the first text data is replaced according to the reference character, and a local parsing result is generated based on the replaced first text data.

[0071] Optionally, based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, after generating a local parsing result based on the first text data and the second text data, the process includes: The target image, target audio data, and local parsing results are packaged into a multimodal data packet, and the multimodal data packet is uploaded to the cloud server. The target image and target audio data in the multimodal data packet are identified using a cloud server to obtain cloud-based parsing results; Receive the cloud parsing result sent by the cloud server, replace the local parsing result according to the cloud parsing result, and visualize the cloud parsing result.

[0072] Optionally, the preset phonetic-shape mapping table is used to store the mapping relationship between the glyphs and the corresponding phonetic text. The preset phonetic-shape mapping table includes at least the mapping relationship between Japanese kanji and kana, the mapping relationship between English numeral pronunciation and numerals, and the mapping relationship between Chinese pinyin and kanji.

[0073] The technical solution of this invention obtains two sets of text data based on image and audio recognition respectively, automatically binds image content with corresponding pronunciations, and simultaneously performs character-level and semantic-level equivalence comparisons of the two sets of text data using a preset phoneme-image mapping table. This accurately distinguishes scenarios with different conflict types and employs differentiated processing strategies for different conflict types, generating local parsing results based on confidence levels. For parsing results that do not meet the conditions for cloud-free processing, the device packages and uploads the original image, audio, and local parsing results to the cloud. The cloud-based multimodal fusion engine then re-infers the original audio-image data to generate authoritative cloud-based parsing results, which replace the local parsing results. This achieves automatic association of audio-image data, allowing accurate text parsing results to be obtained simply by capturing an image.

[0074] The image and speech multimodal data parsing device provided in the embodiments of the present invention can execute the image and speech multimodal data parsing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0075] Example 3 Figure 3 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0076] like Figure 3 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0077] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0078] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as image and speech multimodal data parsing methods.

[0079] In some embodiments, the image-speech multimodal data parsing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the image-speech multimodal data parsing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the image-speech multimodal data parsing method by any other suitable means (e.g., by means of firmware).

[0080] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0081] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0082] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0083] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0084] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0085] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0086] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0087] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0088] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for parsing multimodal image and speech data, characterized in that, include: In response to a photo capture command, acquire the target image and the photo capture timestamp of the target image, as well as target audio data of a preset time length centered on the photo capture timestamp; The target image is identified to obtain first text data, and the first confidence level of each character in the first text data and the first average confidence level of the first text data as a whole are determined; the target audio data is identified to obtain second text data, and the second confidence level of each character in the second text data and the second average confidence level of the second text data as a whole are determined. Based on a preset phonetic-graphic mapping table, semantic-level equivalence comparison and character-level equivalence comparison are performed on the first text data and the second text data to determine the semantic-level judgment result and the character-level judgment result of the first text data and the second text data. Based on the character-level and semantic-level determination results, it is determined whether there is a conflict between the first text data and the second text data, and a conflict determination result is obtained. Based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data.

2. The method according to claim 1, characterized in that, Based on the character-level and semantic-level determination results, it is determined whether there is a conflict between the first text data and the second text data, and a conflict determination result is obtained, including: If the semantic level determination result is equivalent and the character level determination result is equivalent, then the conflict determination result of the first text data and the second text data is determined to be non-conflicting. If the semantic level determination result is not equivalent, then the conflict determination result of the first text data and the second text data is determined to be a semantic conflict. If the character-level determination result is not equivalent, then the conflict determination result of the first text data and the second text data is determined to be a character conflict.

3. The method according to claim 2, characterized in that, Based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data, including: If the conflict determination result is no conflict, and both the first average confidence level and the second average confidence level are greater than the preset confidence threshold, then the first text data is used as the local parsing result.

4. The method according to claim 2, characterized in that, Based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data, further including: If the conflict determination result is a semantic conflict, then the first text data and the second text data are used together as the local parsing result.

5. The method according to claim 2, characterized in that, Based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, a local parsing result is generated based on the first text data and the second text data, further including: If the conflict determination result is a character conflict, then the position of the conflicting character is determined, and the first conflicting character in the first text data and the second conflicting character in the second text data are determined according to the position of the conflicting character. Determine the first confidence level of the first conflicting character and the second confidence level of the second conflicting character, and determine the maximum confidence level between the first confidence level and the second confidence level; A reference character is determined from the first and second conflicting characters based on the maximum confidence level; The first conflicting character at the position of the conflicting character in the first text data is replaced according to the reference character, and a local parsing result is generated based on the replaced first text data.

6. The method according to claim 1, characterized in that, Based on the conflict determination result, and combining the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, after generating a local parsing result based on the first text data and the second text data, the following is included: The target image, target audio data, and local parsing results are packaged into a multimodal data packet, and the multimodal data packet is uploaded to the cloud server. The target image and target audio data in the multimodal data packet are identified using a cloud server to obtain cloud-based parsing results; Receive the cloud parsing result sent by the cloud server, replace the local parsing result according to the cloud parsing result, and visualize the cloud parsing result.

7. The method according to claim 1, characterized in that, The preset phonetic-shape mapping table is used to store the mapping relationship between the character shape and the corresponding phonetic text. The preset phonetic-shape mapping table includes at least the mapping relationship between Japanese kanji and kana, the mapping relationship between English number pronunciation and number, and the mapping relationship between Chinese pinyin and Chinese characters.

8. A multimodal image and speech data parsing device, characterized in that, include: The data acquisition module is used to respond to the photo capture command to acquire the target image and the photo capture timestamp of the target image, as well as the target audio data of a preset time length centered on the photo capture timestamp; The data recognition and confidence determination module is used to recognize the target image to obtain first text data, determine the first confidence of each character in the first text data and the first average confidence of the first text data as a whole; and to recognize the target audio data to obtain second text data, determine the second confidence of each character in the second text data and the second average confidence of the second text data as a whole. The equivalence comparison module is used to perform semantic-level equivalence comparison and character-level equivalence comparison on the first text data and the second text data based on a preset phonetic-graphic mapping table, and to determine the semantic-level judgment result and the character-level judgment result of the first text data and the second text data. The conflict determination module is used to determine whether there is a conflict between the first text data and the second text data based on the character-level determination result and the semantic-level determination result, and to obtain the conflict determination result. The parsing result generation module is used to generate a local parsing result based on the conflict determination result, combined with the first confidence level, the first average confidence level, the second confidence level, and the second average confidence level, and based on the first text data and the second text data.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image and speech multimodal data parsing method as described in any one of claims 1-7.

10. A storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the image-speech multimodal data parsing method as described in any one of claims 1-7.

Images