Multimodal material oriented user device interaction method and apparatus

Abstract

Embodiments of the present disclosure disclose a multi-modal material-oriented user terminal device interaction method and device. A specific embodiment of the method comprises: in response to receiving a case handling request, collecting corresponding multi-modal material data; using a heterogeneous parallel processor to perform local parallel processing on the multi-modal material data; and performing the following interaction steps: generating a matching result on the user terminal device side; in response to the matching result meeting a preset condition, using a cloud server to perform structured analysis and multi-round cross-validation processing, and pushing an audit result to the user terminal device; and in response to the matching result not meeting the preset condition and a corresponding number of interactions being lower than a preset threshold, performing the following steps: generating an incremental supplementary prompt message, and collecting corresponding supplementary material; and performing incremental fusion processing on the supplementary material. This embodiment can achieve efficient parallel audit and intelligent supplementation of multi-modal material, thereby saving computing power loss and interaction waiting time caused by repeated calculations.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to user-end device interaction methods and apparatus for multimodal materials. Background Technology

[0002] Currently, when reviewing government service requests through user-end devices, applicants are required to submit multimodal materials, including images, documents, and audio recordings. The typical method for reviewing these multimodal materials is as follows: all raw multimodal data collected by the user-end device is directly uploaded to a cloud server. The cloud server then sequentially parses and reviews the image, document, and audio data. If the review fails, the user is prompted to supplement all missing materials and re-upload all data for a full review by the cloud server.

[0003] However, when using the above methods for user device interaction, the following technical problems often arise: First, all multimodal raw data is directly uploaded to the cloud, posing a risk of leakage for sensitive personal information such as ID photos and voice recordings during transmission and cloud storage; Second, image, document, and voice data are processed serially in the cloud, making it impossible to utilize the idle heterogeneous computing resources of the user device for parallel preprocessing, resulting in high cloud computing power consumption and high response latency; Third, after each supplementary material submission, all submitted materials and historical review results must be fully recalculated, and the approved parts cannot be reused, resulting in a serious waste of computing power; Fourth, the supplementary prompts are fixed templates, listing all review conditions in full each time, lacking incremental guidance for the current actual gaps, requiring users to repeatedly submit and wait in multiple rounds of interaction, resulting in low interaction efficiency.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure propose user-end device interaction methods and apparatus for multimodal materials to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide a user-end device interaction method for multimodal materials, comprising: in response to receiving a service request initiated by the user-end device, collecting corresponding multimodal material data, wherein the multimodal material data includes image data, document data, and voice data; on the user-end device side, using a heterogeneous parallel processor to perform local parallel processing on the multimodal material data to generate a first encrypted interactive data stream; for the first encrypted interactive data stream, performing the following interaction steps: on the user-end device side, generating a matching result based on the first encrypted interactive data stream; in response to the matching result meeting preset conditions, using a cloud server to perform structured parsing and multi-round cross-validation processing to generate an audit result, and pushing the audit result to the user-end device; in response to the matching result not meeting preset conditions and the corresponding number of interactions being less than a preset threshold, performing the following steps: generating incremental supplementary prompt information through the cloud server, and collecting corresponding supplementary materials based on the incremental supplementary prompt information; on the user-end device side, using the heterogeneous parallel processor to perform incremental fusion processing on the supplementary materials to generate a second encrypted interactive data stream.

[0008] Secondly, some embodiments of this disclosure provide a user-end device interaction apparatus for multimodal materials, including: a collection unit configured to collect corresponding multimodal material data in response to receiving a service request initiated by a user-end device, wherein the multimodal material data includes image data, document data, and voice data; a generation unit configured to perform local parallel processing on the multimodal material data using a heterogeneous parallel processor on the user-end device side to generate a first encrypted interactive data stream; and an interaction unit configured to perform the following interaction steps on the first encrypted interactive data stream: on the user-end device side, based on the first encrypted data stream... The system generates a matching result from a encrypted interactive data stream. If the matching result meets preset conditions, it performs structured parsing and multi-round cross-validation on a cloud server to generate an audit result, and pushes the audit result to the user terminal device. If the matching result does not meet preset conditions and the corresponding number of interactions is below a preset threshold, the system performs the following steps: generates incremental supplementary prompts on the cloud server and collects corresponding supplementary materials based on the incremental supplementary prompts; on the user terminal device side, it performs incremental fusion processing on the supplementary materials using a heterogeneous parallel processor to generate a second encrypted interactive data stream.

[0009] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, such that when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0010] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0011] The above-described embodiments of this disclosure have the following beneficial effects: Through the user-end device interaction method for multimodal materials in some embodiments of this disclosure, efficient parallel review and intelligent supplementation of multimodal materials can be achieved, thereby saving computational power consumption and interaction waiting time caused by repetitive calculations. Specifically, the reasons for high computational power consumption and long interaction waiting time are: existing solutions typically centralize all review calculations in the cloud, and after each supplementation of materials, a full recalculation of all submitted materials and historical review results is required. Simultaneously, the review processes for different modal materials are executed serially, lacking a heterogeneous parallel processing and incremental update mechanism on the end-side, resulting in redundant computational power consumption and requiring users to repeatedly wait for the full re-review to complete in multiple rounds of interaction. Based on this, the user-end device interaction method for multimodal materials in some embodiments of this disclosure first, in response to receiving a service request initiated by the user-end device, collects corresponding multimodal material data, including image data, document data, and voice data. By collecting image, document, and voice data, full modal coverage of government service materials is achieved, avoiding repeated submissions due to missing material types, and improving the completeness of data collection and the first-time pass rate. Then, on the user-end device side, the multimodal material data is processed locally in parallel using a heterogeneous parallel processor to generate a first encrypted interactive data stream. Utilizing a heterogeneous parallel processor on the user-end side completes the parallel structured processing and encryption of the multimodal data, reducing the bandwidth consumption of the original data transmission, shortening the preprocessing waiting time, and ensuring data security before transmission. Next, for the first encrypted interactive data stream, the following interactive steps are executed: On the user-end device side, a matching result is generated based on the first encrypted interactive data stream. Rapid matching of personnel identities with the local database is completed on the user-end side, avoiding invalid data uploads to the cloud, reducing unnecessary network interactions and cloud computing power waste, and improving overall response speed. Secondly, in response to the matching result meeting preset conditions, structured parsing and multi-round cross-validation are performed using a cloud server to generate an audit result, and the audit result is pushed to the user-end device. Cloud computing power and a knowledge base are used to perform in-depth audits on the matched data to ensure the uniformity and accuracy of the audit standards, and the results are pushed to the user in real time, shortening user waiting time. Next, in response to the aforementioned matching result not meeting the preset conditions and the corresponding number of interactions being lower than the preset threshold, the following steps are executed: Incremental supplementary prompt information is generated through the aforementioned cloud server, and corresponding supplementary materials are collected based on the incremental supplementary prompt information. For matching failure scenarios, the cloud automatically generates accurate supplementary guidance and pushes it to the user's end, guiding the user to supplement as needed, avoiding blind supplementation, and reducing invalid interaction rounds. Finally, on the aforementioned user-end device side, incremental fusion processing is performed on the aforementioned supplementary materials using the aforementioned heterogeneous parallel processor to generate a second encrypted interactive data stream.On the user side, incremental fusion and encryption of supplementary materials are performed, processing only the newly added parts rather than all the data, which significantly reduces the amount of redundant calculations and transmission overhead. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of a user-end device interaction method for multimodal materials according to the present disclosure; Figure 2 This is a structural schematic diagram of some embodiments of a user terminal device interaction device for multimodal materials according to the present disclosure; Figure 3 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0016] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0017] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0018] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0019] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] refer to Figure 1 The diagram illustrates a flow 100 of some embodiments of a user terminal device interaction method for multimodal materials according to the present disclosure. This user terminal device interaction method for multimodal materials includes the following steps: Step 101: In response to receiving a service request initiated by the user terminal device, collect the corresponding multimodal material data.

[0021] In some embodiments, the executing entity (e.g., an electronic device) of the above-described user-end device interaction method for multimodal materials can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed in the hardware devices listed above. It can be implemented as multiple software programs or software modules to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0022] In other embodiments, the aforementioned executing entity may, in response to receiving a service request initiated by a user terminal device, collect corresponding multimodal material data, including image data, document data, and voice data. The user terminal device may be a terminal hardware device deployed at the government service site, directly interacting with the public, and possessing multimodal data collection and heterogeneous parallel computing capabilities. For example, the user terminal device may be a government service integrated self-service machine, a comprehensive window staff interaction terminal, or a smartphone running a government application. The service request may be an operation instruction initiated by the user through the user terminal device's interactive interface to initiate the processing of a specific government service item. For example, the service request may be an electronic request instruction carrying an item code and user identity information generated after the user clicks on the "New Application for Public Place Health Permit" item on a self-service terminal. The multimodal material data may be a collection of various types of materials with different data forms and encoding formats required during the government service item review process. The image data may be a visual information carrier existing in the form of static image files. For example, the image data may be photos of the front and back of an applicant's ID card, photos of the inner pages of a household registration book, or other document photos taken by the applicant through the user terminal device's camera. The aforementioned document data can be a text information carrier existing in the form of structured or semi-structured electronic files. For example, the aforementioned document data can be a social security payment certificate in PDF format, an application form in Word format, or an income statement in Excel format submitted by the applicant. The aforementioned voice data can be an acoustic information carrier existing in the form of an audio stream. For example, the aforementioned voice data can be a confirmation of intent to handle the matter or a supplementary explanation of the application matters recorded by the applicant through the microphone of the user terminal device.

[0023] In some optional implementations of certain embodiments, the execution entity may, in response to receiving a service request initiated by a user terminal device, collect corresponding multimodal material data, including image data, document data, and voice data, and may include the following steps: The first step, in response to a service request initiated by a user device, is to determine the material submission method specified in the request. This method can be an instruction specifying how various materials should be submitted to the user device. For example, it could be a combination of three submission methods automatically matched by the system based on the type of service: "on-site photo upload," "local file selection," or "real-time voice recording." In practice, firstly, the user selects the government service item they need to process through the user device's interface and submits the request. Upon receiving the request, the user device reads the item type code and user identification credentials from the request instruction. Then, the user device queries the item configuration information stored locally or in the cloud based on the item type code to obtain a list of material submission methods corresponding to that item. This list specifies which types of materials need to be collected and the collection methods for each type. Finally, the user device records this list of material submission methods as the basis for subsequently activating the data collection hardware. For example, if a user clicks on the "New Application for Public Place Hygiene Permit" item, the user device reads the material submission method list as "ID photo requires on-site shooting, application form requires local file selection, and service confirmation requires real-time voice recording."

[0024] The second step involves determining the multimodal material data type corresponding to the above-mentioned material submission method. This multimodal material data type includes image types, document types, and audio types. The multimodal material data type can be a material category identifier corresponding to different data formats and acquisition / processing paths. For example, the multimodal material data type can be image types, document types, and audio types, corresponding to image processing paths, document parsing paths, and speech recognition paths, respectively. The image type can be a tag used to identify materials that need to be collected in static image form. For example, the image type can correspond to ID card photos, household registration book photos, or ID photos. The document type can be a tag used to identify materials that need to be collected in electronic document file form. For example, the document type can correspond to PDF format certificates, Word format application forms, or Excel format data tables. The audio type can be a tag used to identify materials that need to be collected in audio stream form. The audio type can be a category of sound materials stored in waveform data form. For example, the audio type can correspond to MP3 format oral statement recordings or WAV format telephone consultation records. In practice, firstly, the user-end device parses the submission method information for each type of material based on the acquired submission method. Then, the user-end device categorizes and labels each material in the list according to its modal type: materials submitted via image acquisition are labeled as image types, materials submitted via file reading are labeled as document types, and materials submitted via voice recording are labeled as voice types. Finally, the user-end device generates a list of acquisition tasks containing these three types of labels and assigns a corresponding processing channel to each type.

[0025] The third step involves activating the corresponding type of acquisition hardware in the user terminal device based on the aforementioned multimodal material data types. This acquisition hardware includes an image sensor, a file reading interface, and a microphone array. The image sensor can be a hardware module responsible for converting optical signals into electrical signals; for example, it could be a high-definition camera CMOS sensor built into the self-service terminal. The file reading interface can be a hardware port responsible for reading file data from storage media; for example, it could be a USB interface or an SD card reader. The microphone array can be a multi-channel audio hardware module responsible for acquiring sound signals; for example, it could be a dual-microphone noise reduction array built into the self-service terminal. In practice, firstly, the user terminal device determines the type of hardware resource to be activated based on the acquisition task list generated in the second step. Then, for acquisition tasks marked as image type, the user terminal device sends a start command to the image sensor, activating the camera and adjusting it to photo mode; for acquisition tasks marked as document type, the user terminal device sends a start command to the file reading interface, activating the USB interface or card reader and establishing a file transfer channel; for acquisition tasks marked as audio type, the user terminal device sends a start command to the microphone array, activating the microphones and initializing the audio acquisition buffer. Finally, the user terminal device confirms that all the necessary hardware has been successfully started and is ready, and displays the corresponding operation guidance interface to the user.

[0026] The fourth step involves acquiring image data for the aforementioned image type, document data for the aforementioned document type, and voice data for the aforementioned voice type using the corresponding acquisition hardware. In practice, firstly, after all acquisition hardware is ready, the user-end device guides the user through the interactive interface to perform acquisition operations for each type of material. Then, the user captures and confirms image data using the image sensor; the user-end device receives the image data and stores it in a temporary buffer. The user selects and confirms document data through the file reading interface; the user-end device reads the document file and stores it in a temporary buffer. The user records and confirms voice data using the microphone array; the user-end device records the voice segment and stores it in a temporary buffer. Finally, after confirming that all types of material data have been successfully acquired and stored in the temporary buffer, the user-end device triggers data integrity and format correctness checks. Once the checks pass, the three types of material data are aggregated into a unified multimodal material dataset.

[0027] Step 102: On the user terminal device side, the multimodal material data is processed locally in parallel using a heterogeneous parallel processor to generate the first encrypted interactive data stream.

[0028] In some embodiments, the aforementioned execution entity can utilize a heterogeneous parallel processor on the user terminal device side to perform local parallel processing on the aforementioned multimodal material data to generate a first encrypted interactive data stream. The aforementioned user terminal device side can be a local execution environment for data acquisition and processing; for example, the user terminal device side can be the location of the computing unit inside a self-service all-in-one machine. The aforementioned heterogeneous parallel processor can be a chip integrating multiple types of computing cores; for example, the aforementioned heterogeneous parallel processor can be an embedded processor integrating GPU cores, CPU cores, and NPU cores. The aforementioned local parallel processing can be a processing method that executes multiple computing tasks simultaneously on the user terminal device; for example, the aforementioned local parallel processing can be simultaneously performing image OCR recognition and speech-to-text processing. The aforementioned first encrypted interactive data stream can be an encrypted and encapsulated initial interactive data sequence; for example, the aforementioned first encrypted interactive data stream can be a binary data packet containing encrypted personnel identity information and material structured data.

[0029] In addressing the technical challenges of the aforementioned background technologies, and considering the specific application scenario—the review of government service applications where a large number of users queue to use client devices, and these devices simultaneously process various forms of application materials such as ID card photos, scanned copies of household registration books, income certificates, and recorded voice statements—and require real-time feedback on review results, the following technical issues often arise: serial processing of multimodal data on the client device side is time-consuming, leading to excessively long user wait times; uploading raw materials without structured processing consumes significant network bandwidth; and the lack of end-to-end encryption during data transmission poses a risk of privacy breaches. To meet the following requirements of this application scenario—parallel accelerated processing, structured conversion, and encrypted encapsulation of multimodal materials on the client device side to reduce uploaded data volume, shorten processing latency, and ensure data security—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity may perform local parallel processing of the multimodal material data on the user terminal device side using a heterogeneous parallel processor to generate a first encrypted interactive data stream, which may include the following steps: The first step involves using the image parallel processing sub-thread within the heterogeneous parallel processor to perform optical character recognition (OCR) and image feature extraction on the image data, generating structured image data. This image parallel processing sub-thread can be a dedicated parallel computing thread within the heterogeneous parallel processor for processing image data; for example, it could be an image OCR recognition thread running on a GPU core. The optical character recognition (OCR) can be a technique for extracting text information from an image; for example, it could be extracting the name and ID number from an ID card photo. The structured image data can be data obtained by converting the image content into structured fields; for example, it could be fields such as name, ID number, and validity period stored in JSON format. In practice, firstly, the image parallel processing sub-thread in the heterogeneous parallel processor retrieves the ID card photo from the image data buffer. Then, it performs OCR on the photo to extract text information such as the name and ID number. Finally, image feature extraction is performed on the same photo to obtain facial contour features and document anti-counterfeiting features, which are then combined to generate structured image data. For example, the image processing thread on the GPU core reads the front photo of the applicant's ID card, identifies the name as "Zhang San" and the ID number as "*" through OCR, and simultaneously extracts facial feature vectors to generate JSON-formatted structured image data containing text fields and feature vectors.

[0030] The second step involves utilizing the file-parallel processing sub-thread within the aforementioned heterogeneous parallel processor to perform layout analysis and text extraction on the document data, generating structured document data. This file-parallel processing sub-thread can be a dedicated parallel computing thread within the heterogeneous parallel processor for processing document data; for example, it could be a PDF parsing thread running on a CPU core. The layout analysis and text extraction process can involve analyzing the document layout and extracting text content; for example, it could be extracting the amount, company name, and date from a PDF income certificate. The structured document data can be data obtained by converting the document content into structured fields; for example, it could be income amount, issuing company, and issuance date stored as key-value pairs. In practice, first, the file-parallel processing sub-thread retrieves the PDF income certificate file from the document data buffer. Then, it performs layout analysis on the file, identifying the positions of the title area, body area, and signature area. Finally, it extracts and categorizes the text content from each area to generate structured document data. For example, the document processing thread on the CPU core reads the income certificate in PDF format, analyzes the area containing the title "Income Certificate", the text "This is to certify that Comrade Zhang San's monthly income is 5,000 yuan" and the area with the red official seal, and extracts the amount "5,000 yuan" and the company name "XX Company" into the document's structured data.

[0031] The third step involves utilizing the speech parallel processing sub-thread within the aforementioned heterogeneous parallel processor to perform speech recognition and semantic parsing on the speech data, thereby generating structured speech data. This speech parallel processing sub-thread can be a dedicated parallel computing thread within the heterogeneous parallel processor for processing speech data; for example, it could be a speech recognition thread running on an NPU core. The speech recognition can be a technique that converts speech signals into text; for example, it can convert spoken content from a recording into text. The structured speech data can be data obtained by converting the speech content into structured fields; for example, it could be data stored as fields containing the speaker's name, the content of the statement, and the time of the statement. In practice, firstly, the speech parallel processing sub-thread acquires the MP3 recording file from the speech data buffer. Then, it performs speech recognition on the recording, converting the audio signal into text. Finally, it performs semantic parsing on the transcribed text to extract key information and generate structured speech data. For example, the speech processing thread on the NPU core reads a 30-second MP3 recording, transcribes it into "My name is Zhang San, there are three people in my family, and my monthly income is 5,000 yuan" through speech recognition, and then extracts "Name: Zhang San", "Family size: 3", and "Monthly income: 5,000 yuan" through semantic parsing and stores them in the speech structured data.

[0032] The fourth step involves performing timeline alignment and association mapping on the aforementioned image structured data, document structured data, and voice structured data to obtain multimodal fused structured data. The association mapping can be a process of establishing correspondences between different modalities of data; for example, it can establish a correspondence between the name in an ID photo and the self-proclaimed name in a voice statement. The multimodal fused structured data can be unified data after integrating multiple modalities of structured data; for example, it can be a comprehensive data record containing fields for personal identity, income verification, and voice statement. In practice, firstly, the image structured data, document structured data, and voice structured data generated in the first three steps are acquired. Then, based on the collection timestamp, the three types of structured data are timeline aligned. Finally, association mapping relationships between cross-modal fields are established to generate multimodal fused structured data. For example, the name "Zhang San" in the image data, document data, and voice data are consistently associated, merging the three into a single comprehensive data record containing all fields.

[0033] The fifth step involves extracting the basic personnel information embedded within the aforementioned multimodal fused structured data to generate multimodal fused structured data bound with identity identifiers. This basic personnel information can be fundamental attribute information identifying an individual, such as name, ID number, and registered address. The multimodal fused structured data bound with identity identifiers can be fused data with an attached unique identity tag; for example, it can be a fused data record tagged with an ID number. In practice, first, the personnel identity field is scanned from the multimodal fused structured data. Then, basic personnel information such as name and ID number are extracted as unique identifiers. Finally, this identifier is bound to the header of the multimodal fused structured data to generate multimodal fused structured data bound with identity identifiers. For example, the ID number "*" is extracted from the fused data as the primary key and written into the identity identifier field of the data packet header to generate tagged fused data.

[0034] The sixth step involves using the encryption operators built into the heterogeneous parallel processor to perform hierarchical encryption on the multimodal fusion structured data bound to identity identifiers, generating multiple encrypted single-unit data streams. These encryption operators can be hardware or software modules performing encryption operations; for example, the encryption operator could be the AES encryption engine built into the heterogeneous parallel processor. The hierarchical encryption process can employ different encryption strengths based on different security levels; for example, the hierarchical encryption process could use 256-bit encryption for ID numbers and 128-bit encryption for general materials. The multiple encrypted single-unit data streams can be multiple independent encrypted data sub-streams; for example, the multiple encrypted single-unit data streams could be encrypted image data streams, document data streams, and voice data streams, respectively. In practice, firstly, the encryption strategy is determined based on the sensitivity level of each field in the multimodal fusion structured data bound to identity identifiers. Then, the encryption operators built into the heterogeneous parallel processor are invoked to perform encryption processing with corresponding strengths on fields of different sensitivity levels. Finally, multiple independent encrypted single-unit data streams are output. For example, the ID number field is encrypted using AES-256, the income amount field is encrypted using AES-128, and the general statement content is encrypted using AES-64, generating three encrypted data substreams respectively.

[0035] Step 7: Perform time-series fusion and splicing processing on the aforementioned multi-channel encrypted single-unit data streams to generate the first encrypted interactive data stream. This time-series fusion and splicing can be a process of merging multiple data streams into a single continuous data stream according to their chronological order. For example, it can splice three encrypted sub-streams into a complete data packet based on their acquisition time. First, obtain the multi-channel encrypted single-unit data streams generated in Step 6. Then, arrange the encrypted sub-streams according to the original data acquisition time order. Finally, splice the arranged encrypted sub-streams into a continuous binary data stream to generate the first encrypted interactive data stream. For example, according to the chronological order of image acquisition time, document acquisition time, and voice acquisition time, splice the three encrypted sub-streams into a complete encrypted data packet, and add a start flag and length field.

[0036] The above-described steps, as an inventive point of this disclosure, solve the technical problems mentioned in the background art: "The serial processing of multimodal material data on the user-end device side is time-consuming, resulting in excessively long user waiting times; the raw materials are uploaded directly without structured processing, consuming a large amount of network bandwidth; and the material data lacks end-to-end encryption protection during transmission, posing a risk of privacy leakage." The reasons for these technical problems are as follows: In traditional solutions, multimodal materials are processed serially on the user-end device side, with different modal data queuing for processing sequentially, resulting in a long overall processing time; the raw materials are uploaded directly without structured compression, resulting in a large data volume; and the transmission process lacks end-to-end encryption protection. This invention utilizes a heterogeneous parallel processor to simultaneously process image, document, and audio data, and performs structured conversion and hierarchical encryption encapsulation locally, achieving parallel accelerated processing and secure transmission of multimodal materials, saving review processing time and network transmission bandwidth.

[0037] Step 103: For the first encrypted interactive data stream, perform the following interactive steps: Step 1031: On the user terminal device side, a matching result is generated based on the first encrypted interactive data stream.

[0038] In some embodiments, the aforementioned executing entity can generate a matching result on the user terminal device side based on the first encrypted interactive data stream. The matching result can be a determination conclusion after comparing a person's identity with a database record; for example, the matching result can be a Boolean value of "match successful" or "match failed." The aforementioned decryption and decapsulation processing can be a process of reverse engineering the encrypted data stream; for example, the decryption and decapsulation processing can be using an AES key to decrypt data packets and disassemble protocol frame headers.

[0039] In some optional implementations of certain embodiments, the execution entity may generate a matching result on the user terminal device side based on the first encrypted interactive data stream, which may include the following steps: The first step involves decrypting and decapsulating the first encrypted interactive data stream to generate multimodal fused structured data. This multimodal fused structured data can be a unified data record integrating image, document, and voice structured data. For example, it can be a comprehensive JSON object containing fields such as name, ID number, income amount, and voice statement. In practice, first, the binary data packet of the first encrypted interactive data stream is obtained. Then, an AES decryption operation is performed on the data packet using a preset decryption key to remove the encryption layer. Finally, the data frame header and trailer are disassembled according to a preset protocol format to reconstruct the multimodal fused structured data. For example, the received encrypted data packet is decrypted using a 256-bit AES key stored in the user's device security zone, and then the length and type fields in the frame header are disassembled according to the TLV format to obtain JSON-formatted multimodal fused structured data containing fields such as name, ID number, and income amount.

[0040] The second step is to extract personnel identification information from the aforementioned multimodal fusion structured data to generate a candidate identifier. This candidate identifier can be a unique identity index used to find the corresponding person in the database; for example, it could be an ID card number or social security number. In practice, first, the field list of the multimodal fusion structured data is scanned. Then, the field identified as "identity identifier" is located. Finally, the personnel identification information is read from this field to generate the candidate identifier. For example, the "id_card" field is searched in the JSON data, its value is read as "*", and this 18-digit ID card number is used as the candidate identifier.

[0041] The third step involves performing a fuzzy match query between the aforementioned personnel identifier and the locally cached personnel database to generate matching similarity information. The personnel identifier can be a unique identity index used to find the corresponding person in the database; for example, it could be an ID card number or social security number. The locally cached personnel database can be a structured collection of personnel information stored in the user's device memory; for example, it could be an SQLite database containing residents' names, ID card numbers, and registered addresses. The matching similarity information can be a numerical indicator reflecting the closeness between the identifier and the database record; for example, it could be a similarity score between 0 and 1. In practice, first, the locally cached personnel database index table on the user's device is loaded. Then, the personnel identifier is used as a query condition to perform a fuzzy match query on the database. Finally, the edit distance or cosine similarity between each candidate record and the identifier is calculated to generate matching similarity information. For example, a fuzzy match is performed between the ID number "*" and 10,000 locally cached personnel records. The edit distance algorithm calculates that the similarity with a certain record is 0.95, while the similarity with other records is all below 0.5.

[0042] The fourth step involves comparing the aforementioned matching similarity information with a preset matching threshold to generate a matching result. This preset matching threshold can be a critical similarity value used to determine whether a match is successful. For example, the preset matching threshold could be 0.85, meaning a similarity greater than or equal to 0.85 is considered a successful match. In practice, first, the preset matching threshold configuration parameter is read. Then, the matching similarity information generated in the third step is compared with this matching threshold. Finally, if the similarity is greater than or equal to the matching threshold, "match successful" is output; otherwise, "match failed" is output, generating a matching result. For example, if the preset matching threshold is 0.85, and a similarity of 0.95 is compared to it, 0.95 ≥ 0.85, therefore the generated matching result is "match successful".

[0043] Step 1032: In response to the matching result meeting the preset conditions, structured parsing and multi-round cross-validation are performed using the cloud server to generate the audit result and push the audit result to the aforementioned user terminal device.

[0044] In some embodiments, the execution entity may, in response to the matching result meeting preset conditions, utilize a cloud server to perform structured parsing and multi-round cross-validation processing to generate an audit result and push the audit result to the user terminal device. The preset conditions may be the matching result judgment criteria that trigger the cloud audit process; for example, the preset conditions may be a matching result of "successful match." The cloud server may be a high-performance computing node deployed in a remote data center. The structured parsing may be the process of converting unstructured data into normalized fields; for example, the structured parsing may split free text in fused data into fixed fields according to a predefined template. The multi-round cross-validation processing may be an audit method that alternately verifies data from different dimensions multiple times; for example, the multi-round cross-validation processing may first verify the completeness of the materials and then verify the compliance of the personnel qualifications. The audit result may be the final review conclusion of the application; for example, the audit result may be "approved," "disapproved," or "requires supplementary materials."

[0045] In some optional implementations of certain embodiments, the execution entity may, in response to the matching result meeting preset conditions, utilize a cloud server to perform structured parsing and multi-round cross-validation processing to generate an audit result, and push the audit result to the user terminal device, which may include the following steps: The first step involves uniformly decrypting and parsing the first encrypted interactive data stream to generate cloud-readable multimodal fused structured data. This unified decryption and parsing can involve using a unified key to globally decrypt and format-parse the encrypted data stream. For example, it could involve using the cloud master key to decrypt the entire encrypted data packet at once. The cloud-readable multimodal fused structured data can be standardized fused data that the cloud server can directly parse. For example, it could be a JSON-formatted data object conforming to the cloud API specifications. In practice, the cloud server first receives the first encrypted interactive data stream uploaded by the user device. Then, it performs a unified AES-256 decryption operation on the data stream using the cloud master key to remove the encryption layer. Finally, it parses the data frame according to the cloud standard protocol format to restore it to cloud-readable multimodal fused structured data. For example, after receiving the encrypted data packet, the cloud server decrypts it using the master key stored in the HSM hardware security module and then parses it according to the Protobuf protocol to obtain a standardized data object containing fields such as name, ID number, and income amount. The second step involves using a cloud-stored standard review knowledge base to extract the corresponding set of rules for determining material compliance. This cloud-stored standard review knowledge base can be a structured collection of review rules stored in the cloud. For example, it could be a vector database containing a list of materials required for low-income assistance reviews, qualification requirements, and policy basis. The set of rules for determining material compliance can be a set of rules for judging whether materials meet the requirements. For example, it could include rules such as "ID card must be valid" and "income certificate must be stamped with an official seal." In practice, firstly, based on the item type code in the application request, the corresponding item index is located in the cloud-stored standard review knowledge base. Then, all review rules for that item are retrieved from the knowledge base partition pointed to by the index. Finally, the retrieved rules are organized into a structured rule set. For example, based on the item code "DB001" (low-income assistance review), the corresponding material list rules, qualification requirement rules, and policy basis rules are retrieved from the vector knowledge base and organized into a JSON array containing 12 rules.

[0046] The third step involves processing the first round of compliance verification of the aforementioned material compliance judgment rule set, using the cloud-readable multimodal fusion structured data, to obtain the first verification index data. This first round of compliance verification can be an initial check of material completeness and format correctness; for example, it could check if photos of the front and back of an ID card have been uploaded and are clearly identifiable. The first verification index data can be a quantitative evaluation result generated after the first round of verification; for example, it could be "Material completeness score: 90 points, format correctness score: 85 points". In practice, firstly, the cloud-readable multimodal fusion structured data generated in the first step is obtained. Then, each rule in the material compliance judgment rule set extracted in the second step is iterated through. Finally, each field in the structured data is compared with the rules to generate the first verification index data. For example, checking if the structured data contains the "front photo of ID card" field and that this field is not empty, and checking if the "ID card validity period" field is later than the current date, generates a verification index of "Material completeness: 10 items passed / 2 items missing" after each comparison.

[0047] The fourth step involves using the cloud-based personnel file database to perform a cross-check of the first verification indicator data to obtain the second verification indicator data. This cross-checking can be a process of mutually verifying personnel information against the personnel file database; for example, it could compare the applicant's name with the name in the household registration system. The second verification indicator data can be the verification result generated after the cross-checking; for example, it could be "Name consistency: Pass, Household registration address consistency: Pass". In practice, first, the list of passing fields in the first verification indicator data is obtained. Then, the values ​​of these fields are compared field by field with the corresponding personnel file records in the cloud-based personnel file database. Finally, the number of consistent fields and the number of inconsistent fields are counted to generate the second verification indicator data. For example, the applicant's name "Zhang San" is compared with the name "Zhang San" in the household registration file, and the applicant's date of birth "1990-01-01" is compared with "1990-01-01" in the file, generating a verification index of "consistency comparison: 5 items consistent / 0 items inconsistent".

[0048] The fifth step involves performing multiple rounds of cyclical cross-validation based on the first and second verification indicator data to obtain the comprehensive compliance score. This multi-round cyclical cross-validation process can involve repeatedly performing verifications of different dimensions until convergence. For example, it could involve verifying materials first, then personnel, and if problems are found, backtracking to re-verify the materials. The comprehensive compliance score can be the overall score after combining the results of multiple rounds of verification; for example, it could be "Comprehensive Compliance Score: 92 points, Rating: Good". In practice, firstly, the initial verification round counter is set to 1. Then, based on the first and second verification indicator data, the first round of comprehensive scoring is performed. Next, if the score is lower than a preset threshold, backtracking to the third or fourth step to supplement the verification based on missing items, and incrementing the round counter by 1. Finally, the above process is repeated until the score stabilizes or the maximum number of rounds is reached, generating the comprehensive compliance score. For example, if the first round score is 85 points, and a missing item is found in the materials, the process goes back to the third step to re-verify the supplemented materials. The second round score is 92 points and does not change, resulting in an overall compliance score of 92 points.

[0049] Step 6 involves comparing the comprehensive compliance score data with the cloud-preset review level thresholds to generate a tiered review result. These cloud-preset review level thresholds can be cloud-defined score limits for passing or failing the review. For example, the thresholds could be "90 points or above is passing, 70-89 points are pending review, and below 70 points is failing." The tiered review process can determine the review level based on the score's range. For instance, a score of 92 falling into the "pass" range would be considered a passing review. In practice, first, the cloud-preset review level threshold configuration table is read. Then, the comprehensive compliance score data generated in step 5 is compared with the score lines in the threshold table. Finally, the score's range is determined, and the review result is generated. For example, if the threshold configuration is "pass ≥ 90 points, pending review 70-89 points, failing < 70 points," comparing 92 points with the threshold (92 ≥ 90) results in a "pass" review.

[0050] Step 7: Based on the above review results, generate a targeted push interaction command to push the review results to the aforementioned user terminal device. This targeted push interaction command can be a push command specifically generated for a particular user terminal device; for example, it can be a JSON command package containing the review results and instructions for the next step. In practice, firstly, based on the review results generated in Step 6, assemble the content template of the push message. Then, add the unique identifier and routing information of the target user terminal device to the message header. Finally, send the complete push command to the user terminal device through a secure channel.

[0051] Step 1033: In response to the matching result not meeting the preset conditions and the corresponding number of interactions being lower than the preset threshold, the following steps are executed: Step 10331 involves generating incremental supplementary prompts via a cloud server and collecting corresponding supplementary materials based on these prompts. The number of interactions can be the total count of all interactions performed since the user's device first initiated the application request. For example, the number of interactions could be 1 (first round of supplementation triggered by matching failure after initial submission) or 2 (second round of supplementation triggered if still not satisfied after supplementation). The preset threshold can be a pre-set numerical upper limit for restricting the maximum allowed number of interactions. When the number of interactions reaches this threshold, the loop terminates and a final conclusion is output. For example, the preset threshold could be 3, meaning a maximum of 3 rounds of supplementary interactions are allowed before the process terminates regardless of the matching result. The incremental supplementary prompts can be differentiated supplementary guidance generated only for the missing materials in the current interaction round, without repeatedly reminding users of already satisfied items. For example, the incremental supplementary prompt could be "ID photo approved, but insurance payment certificate (PDF format, stamped with official seal) still needs to be supplemented." The aforementioned supplementary materials can be compliant materials submitted by users based on the incremental supplementary prompts, which have passed quality inspection and format verification. For example, the aforementioned supplementary materials can be PDF format insurance payment certificates that meet the clarity standards and are formatted in compliance with regulations.

[0052] In some embodiments, the aforementioned executing entity may generate incremental supplementary prompt information through the aforementioned cloud server, and collect corresponding supplementary materials based on the aforementioned incremental supplementary prompt information.

[0053] In some optional implementations of certain embodiments, the execution entity may generate incremental supplementary prompt information through the cloud server and collect corresponding supplementary materials based on the incremental supplementary prompt information, which may include the following steps: The first step involves parsing the missing material field identifiers in the first encrypted interactive data stream using the aforementioned cloud server to generate a list of missing material types. These missing material field identifiers can be field codes in the pre-defined knowledge base that are marked as necessary but whose corresponding fields in the multimodal fusion structured data are empty or invalid. For example, the missing material field identifiers could be "MATERIAL_Social Security Payment Certificate" or "FIELD_Household Registration Address". The list of missing material types can be generated by categorizing and deduplicating the missing material field identifiers according to material modality (image / document / voice). For example, the list of missing material types could be {"Image": ["Back of ID Card"], "Document": ["Social Security Certificate"], "Voice": []}. In practice, firstly, the cloud server obtains the multimodal fusion structured data corresponding to the first encrypted interactive data stream. Then, it iterates through all fields in the data structure, checking each field value for emptiness or being marked as missing. Finally, all field identifiers with empty values ​​or marked as missing are collected and summarized to generate a list of missing material types.

[0054] The second step involves a comparative analysis of the missing material types list and the material requirements in the pre-defined knowledge base, generating differentiated supplementary guidance text. The pre-defined knowledge base can be a pre-built knowledge base organized by matter type and indexed using slices and vectorization, containing a complete list of required materials and format specifications for each matter. For example, the pre-defined knowledge base could be a structured knowledge base stating that "social security matters require submission of ID card photos, social security card photos, and proof of social security contributions (PDF format)." The material requirements can be the format specifications, content requirements, and validity periods stipulated in the pre-defined knowledge base for specific materials. For example, the material requirements could be "proof of social security contributions must be stamped with the company seal, in PDF format, and the file size cannot exceed 10MB." The comparative analysis can be a process of comparing the missing material types list with the complete list of material requirements in the pre-defined knowledge base item by item to identify specific missing or incomplete content. For example, the comparative analysis could identify a missing item if the "proof of social security contributions" field is empty but required by the knowledge base. The aforementioned differentiated supplementary guidance text can be user-oriented natural language description text generated based on the results of difference comparison analysis, clearly informing users of the specific content and specifications that need to be supplemented. For example, the differentiated supplementary guidance text could be, "You are missing proof of social security contributions; please upload a PDF file with an official seal." In practice, firstly, the cloud server obtains the list of missing material types generated in the first step and the detailed requirements for the corresponding missing materials in the preset knowledge base (such as format specifications, content requirements, and example descriptions). Then, a difference comparison analysis is performed between each item in the list of missing material types and the material requirements in the knowledge base to specifically identify the compliance difference type of each missing material ("completely missing," "inadequate clarity," "incorrect format," etc.). Finally, based on the difference analysis results, differentiated supplementary guidance text is generated for each missing material. For example, if the difference for the missing material "social security card photo" is "inadequate clarity," and the knowledge base requires "image resolution ≥ 1920×1080," the guidance text "The social security card photo you uploaded is not clear enough; please retake it to ensure the photo resolution reaches 1920×1080 pixels" is generated.

[0055] The third step involves performing hierarchical semantic packaging on the differentiated supplementary guidance text to generate incremental supplementary prompts. This hierarchical semantic packaging can be achieved by organizing the text hierarchically according to urgency and importance. For example, mandatory materials can be marked as "Important," and optional materials as "Optional." In practice, first, the differentiated supplementary guidance text generated in the second step is obtained. Then, the text content is hierarchically labeled according to the importance and urgency of the materials. Finally, the hierarchical text is reorganized and packaged according to priority to generate incremental supplementary prompts. For example, "back photo of ID card" is labeled as "Required - High Priority," and "Income Certificate" is labeled as "Required - Medium Priority," reorganized as "[Important] Please prioritize: Back photo of ID card; [Second] Please prioritize: Original income certificate."

[0056] The fourth step involves pushing the incremental supplementary prompt information to the user terminal device via an encrypted channel, triggering the corresponding acquisition hardware on the user terminal device to enter a standby state. The encrypted channel can be a secure data transmission path protected by encryption; for example, it can be an HTTPS connection based on the TLS protocol. The acquisition hardware corresponding to the user terminal device can be a hardware module within the user terminal device responsible for acquiring specific types of data; for example, it can be a camera, scanner, or microphone. The standby state can be a low-power, waiting-to-activate working mode where the acquisition hardware has completed initial configuration. In this state, the hardware can be quickly woken up and begin data acquisition; for example, the standby state could be a camera with preview enabled but not recording started, or a USB interface with power supplied but not transmitting data. In practice, firstly, the generated incremental supplementary prompt information is encapsulated into an encrypted message. Then, the message is pushed to the user terminal device via a pre-established TLS encrypted channel. Finally, after the push is complete, a wake-up signal is sent to the hardware management module of the user terminal device, triggering the corresponding acquisition hardware to enter a standby state.

[0057] Fifth, in response to the user terminal device receiving a supplementary operation instruction triggered by the user based on the incremental supplementary prompt information, the standby acquisition hardware is activated to execute the incremental acquisition task. The supplementary operation instruction can be a user operation signal triggered by the user through the user terminal device's interactive interface to activate the acquisition hardware for data acquisition. For example, the supplementary operation instruction can be a hardware invocation instruction generated by clicking the "Start Shooting" button or the "Select File" button. The incremental acquisition task can be a targeted acquisition task initiated only for material types listed in the missing material type list, without re-acquiring already passed materials. For example, the incremental acquisition task can be to only take a photo of the back of the ID card without re-acquiring the already passed front photo. In practice, firstly, the user terminal device displays the incremental supplementary prompt information on the screen and shows the "Start Supplement" button. Then, the user clicks the "Start Supplement" button, and the user terminal device generates a supplementary operation instruction. Finally, the user terminal device forwards the instruction to the hardware management module, activating the standby acquisition hardware to execute the incremental acquisition task. For example, when a user sees the prompt "Please supplement a photo of the back of your ID card" on the self-service terminal screen and clicks "Start Supplementing", the terminal immediately activates the camera and pops up a shooting interface, guiding the user to place the back of their ID card in the shooting area.

[0058] Step 6: Perform resolution and signal-to-noise ratio (SNR) detection processing on the newly acquired material data from the incremental acquisition task to generate image quality score data. The newly acquired material data can be raw data obtained after the user performs the incremental acquisition task, which has not yet undergone quality testing and compliance verification. For example, the newly acquired material data can be a freshly taken, uncompressed image of the back of an ID card. The image quality score data can be a quality score of 0-100 points generated by weighting the resolution and SNR according to a preset scoring model. For example, the image quality score data can be 85 points or 42 points. The resolution and SNR detection processing can be a process of quantitatively evaluating image sharpness and noise level. For example, the resolution and SNR detection processing can be the calculation of the image's PPI value and SNR dB value. First, obtain the newly acquired material data file returned by the incremental acquisition task. Then, call the image processing algorithm to calculate the resolution (PPI value) and signal-to-noise ratio (SNR value) of the file. Finally, weighted sum of the two indicators to generate image quality score data. For example, if a newly taken JPEG file of the back of an ID card is obtained, and the resolution is calculated to be 350 PPI and the signal-to-noise ratio is 32 dB, the comprehensive score calculated with a weight of 0.5:0.5 is (350 / 400×100)×0.5+(32 / 40×100)×0.5=43.75+40=83.75 points.

[0059] Step 7: In response to the image quality score data falling below a preset quality threshold, the acquisition hardware of the user terminal device automatically triggers a re-acquisition command to obtain optimized new material data. The preset quality threshold can be a pre-set minimum quality score threshold used to determine whether the new material data reaches the minimum quality score required for format verification. For example, the preset quality threshold could be 60 points; materials scoring below 60 points require re-acquisition. The re-acquisition command can be a control command automatically generated by the user terminal device when the image quality score data falls below the preset quality threshold, requiring the re-acquisition of the material data. For example, the re-acquisition command could be a guiding command such as "Photo is blurry, please retake and ensure the ID card is completely within the frame." The optimized new material data can be new material data re-acquisitioned after the user responds to the re-acquisition command, with a quality score not lower than the preset quality threshold. For example, the optimized new material data could be a clear ID card photo retaken after adjusting the lighting. In practice, first, the preset quality threshold configuration parameters are read. Then, the image quality score data generated in step 6 is compared with this threshold. Finally, if the score is below the threshold, a re-sampling command is automatically sent to the acquisition hardware, instructing the user to re-acquire and obtain optimized new material data. For example, if the preset threshold is 80 points and the current score is 83.75 points, which is above the threshold, re-sampling will not be triggered. If the score is 75 points, which is below 80 points, a prompt will automatically pop up saying "The photo quality is poor. Please retake the photo, ensuring sufficient light and clear focus," and the camera will be reactivated.

[0060] Step 8 involves performing format compliance checks and metadata integrity checks on the optimized new material data to generate supplementary materials. The format compliance check can involve verifying whether the file type, file size, and encoding format of the new material data conform to the material format requirements in the preset knowledge base. For example, this could involve checking whether the file is in JPG format and whether its size is between 100KB and 10MB. The metadata integrity check can involve verifying whether metadata fields such as the creation time and acquisition device identifier of the new material data are complete and untampered with. For example, this could involve checking whether the EXIF ​​information of the photo contains the shooting time and device model and whether it has been modified. In practice, first, qualified new material data is acquired (this could be data from the initial acquisition or optimized data from a re-acquisition). Then, the file format is checked to ensure it meets preset requirements (e.g., JPEG, PNG, PDF) and the file size is within the allowed range. Finally, the file metadata is checked for completeness (e.g., shooting time, device model, GPS location). Supplementary materials are generated only after all checks are passed.

[0061] As an example, in response to the matching result not meeting the preset conditions and the corresponding number of interactions not being less than the preset threshold, an audit termination message is generated, and the audit termination message is pushed to the user terminal device to inform the user that the application has been terminated due to multiple failed supplements.

[0062] Step 10332: On the user terminal device side, incremental fusion processing is performed on the supplementary material using a heterogeneous parallel processor to generate a second encrypted interactive data stream.

[0063] In some embodiments, the execution entity can perform incremental fusion processing on the supplementary materials using the heterogeneous parallel processor on the user terminal device side to generate a second encrypted interactive data stream. The incremental fusion processing can be a method of incrementally merging new data with existing data; for example, it can merge only the newly taken photo of the back of the ID card with existing front photos. The second encrypted interactive data stream can be a new interactive data sequence after incremental fusion and encryption; for example, it can be a complete data packet containing the unchanged portion of the original data and newly supplemented encrypted data blocks.

[0064] In addressing the technical challenges of the aforementioned background technologies, and considering the specific application scenario—the review of government service applications—where users initially submit multimodal materials but find some materials missing or inadequate after review, requiring supplementary submissions that must be integrated with existing materials, often presents the following technical problems: Traditional solutions require the supplementary materials to be reprocessed and transmitted along with the original materials, leading to significant redundant calculations and wasted bandwidth; data overlap or conflicts may exist between the supplementary and original materials. Given the specific requirements of this application scenario—incremental processing of supplementary materials on the user's device side, handling only the newly added data rather than all data, while ensuring seamless integration and version traceability of supplementary data with historical data—we have decided to adopt the following solution: In some optional implementations of certain embodiments, the execution entity may perform incremental fusion processing on the supplementary material using the heterogeneous parallel processor on the user terminal device side to generate a second encrypted interactive data stream, which may include the following steps: The first step involves modal type identification and classification of the supplementary materials to obtain supplementary image data, supplementary document data, and supplementary audio data. Modal type identification can be a process of determining the information format category of the data; for example, it can determine whether a file is an image, document, or audio. Classification can be a process of assigning data to the corresponding processing queue according to modal type; for example, it can assign JPEG files to the image queue and PDF files to the document queue. The supplementary image data can be user-submitted image materials; for example, it could be a newly taken JPEG photo of the back of an ID card. The supplementary document data can be user-submitted document materials; for example, it could be a newly uploaded PDF income certificate. The supplementary audio data can be user-submitted audio materials; for example, it could be a newly recorded MP3 audio recording explaining the situation. In practice, first, the user-submitted supplementary materials are obtained. Then, the magic number header and extension of the supplementary materials are read to determine the actual file format type. Finally, the files are assigned to the corresponding modal category queues according to their format type, resulting in supplementary image data, supplementary document data, and supplementary audio data.

[0065] The second step involves utilizing idle computing threads within the aforementioned heterogeneous parallel processor to perform parallel structured processing on the supplementary image data, supplementary document data, and supplementary audio data, respectively, to generate supplementary image structured data, supplementary document structured data, and supplementary audio structured data. The idle computing threads can be currently unused computing threads within the heterogeneous parallel processor; for example, they can be CUDA core threads in a GPU that have completed their preceding tasks and are currently idle. The parallel structured processing can simultaneously perform structured transformation operations on multiple modalities of data; for example, it can simultaneously perform OCR on images, text extraction on documents, and transcription on audio. The supplementary image structured data can be data obtained by converting the supplementary image content into structured fields; for example, it can be a JSON object containing fields such as the validity period on the back of an ID card and the issuing authority. The supplementary document structured data can be data obtained by converting the supplementary document content into structured fields; for example, it can be key-value pairs containing fields such as income amount, company name, and issuance date. The aforementioned supplementary structured audio data can be data obtained by converting the supplementary audio content into structured fields. For example, the supplementary structured audio data can be a structured record containing fields such as the speaker's name and a summary of the statement content. In practice, firstly, the current load status of each computational sub-thread in the heterogeneous parallel processor is queried. Then, the supplementary image data is allocated to idle GPU sub-threads, the supplementary document data is allocated to idle CPU sub-threads, and the supplementary audio data is allocated to idle NPU sub-threads. Finally, the three sub-threads execute their respective structured processing in parallel to generate the corresponding structured data. For example, the idle GPU sub-thread performs OCR recognition on the back of an ID card photo to extract the validity period and issuing authority; the idle CPU sub-thread performs layout analysis and text extraction on a PDF income certificate to extract the amount and company name; and the idle NPU sub-thread performs speech transcription and semantic parsing on an MP3 recording to extract the statement content. All three processes are completed in parallel.

[0066] The third step involves performing timestamp alignment and overlapping area deduplication on the supplementary image structured data, supplementary document structured data, and supplementary voice structured data, along with the historical structured data corresponding to the first encrypted interactive data stream, to obtain incrementally fused structured data. The historical structured data corresponding to the first encrypted interactive data stream can be a copy of the structured data generated and archived during the initial submission. For example, the historical structured data can be JSON data containing the front information of an ID card generated during the initial submission. Timestamp alignment can be a process of matching different batches of data along a timeline according to the data collection time. For example, timestamp alignment can align the collection time of the supplementary materials with the collection time of the original materials. Overlapping area deduplication can be a process of identifying and removing duplicate content between the old and new data. For example, overlapping area deduplication can be performed when the supplementary materials contain the same ID card number field as the original materials. The incrementally fused structured data can be the fused data containing only the newly added and changed parts. For example, the incrementally fused structured data can be simplified data containing newly added information from the back of the ID card but not existing information from the front. In practice, firstly, the aforementioned supplementary image structured data, supplementary document structured data, and supplementary voice structured data are acquired. Then, using the acquisition timestamp as a reference, the supplementary structured data is time-aligned with the historical structured data corresponding to the first encrypted interactive data stream. Finally, the aligned data is scanned to check for overlapping areas with identical field names and values; if found, the duplicates are removed to obtain incrementally fused structured data.

[0067] The fourth step involves appending the version identifier and update log information of the current interaction round to the aforementioned incrementally fused structured data to generate incrementally fused structured data with a version number. The version identifier of the current interaction round can be a version number indicating which supplementary interaction it is; for example, "V2.0" could represent the second interaction version. The update log information can be a historical record of the content and time of this supplementary operation; for example, the update log information could be a log entry for "2026-05-29 15:00:00 Supplementing ID card back photo". The incrementally fused structured data with a version number can be fused data with the version identifier and log appended; for example, the incrementally fused structured data with a version number can be a JSON data packet containing the version number "V2.0" and the update log. In practice, first, the incrementally fused structured data is obtained. Then, the counter value of the current interaction round is read to generate a version identifier string. Finally, the version identifier and update log information including operation time and operation type are appended to the header of the incremental fused structured data to generate incremental fused structured data with version number.

[0068] The fifth step involves using the encryption operator built into the heterogeneous parallel processor to perform incremental encryption on the version-numbered incrementally fused structured data, generating an incrementally encrypted data block. This incremental encryption can be performed on only the newly added data portion, rather than re-encrypting the entire dataset. For example, it could involve performing AES encryption only on the newly added data on the back of the ID card. The incrementally encrypted data block can be an independent data segment containing only the newly encrypted data; for example, it could be an AES-256 encrypted data block of the information on the back of the ID card. In practice, the version-numbered incrementally fused structured data generated in the fourth step is obtained. Then, the encryption operator built into the heterogeneous parallel processor is invoked, using the same key system as the first encryption. Finally, AES-256 encryption is performed only on the newly added structured data portion to generate an incrementally encrypted data block. For example, using the same AES-256 key as the first encryption, encryption is performed only on the JSON fragments of the two newly added fields, "Validity Period" and "Issuing Authority," generating an independent encrypted data block, while the version identifier and update log are retained in plaintext for traceability.

[0069] The sixth step involves concatenating and recombining the incremental encrypted data block with the unchanged portion of the first encrypted interactive data stream to generate the second encrypted interactive data stream. The unchanged portion of the first encrypted interactive data stream can be encrypted data segments from the initial data submission that do not require updating; for example, it could be an encrypted segment of the front image of an ID card. In practice, firstly, the incremental fused structured data with a version number is obtained. Then, the built-in encryption operator of the heterogeneous parallel processor is invoked, using the same key system as the first encryption. Finally, AES-256 encryption is performed only on the newly added structured data portion to generate an incremental encrypted data block. For example, using the same AES-256 key as the first encryption, encryption is performed only on the JSON fragments of the two newly added fields, "Validity Period" and "Issuing Authority," generating a separate encrypted data block, while the version identifier and update log are retained in plaintext for traceability.

[0070] The above-described steps, as an inventive point of this disclosure, solve the technical problem mentioned in the background art: "In traditional solutions, supplementary materials need to be reprocessed and transmitted as a whole with the original materials, resulting in a large amount of redundant calculations and serious waste of transmission bandwidth; there may be data overlap or conflict between supplementary materials and original materials." The reasons for the above technical problems are as follows: In traditional solutions, each time supplementary materials are added, all data needs to be restructured and encrypted before transmission, making it impossible to distinguish between new and historical parts; the lack of effective version control and deduplication mechanisms leads to data redundancy and conflicts. The inventive point of this invention uses a heterogeneous parallel processor to perform incremental parallel processing of supplementary materials, aligns and deduplicates the data with historical data using timestamps, and performs incremental encryption after adding a version identifier, thereby achieving efficient incremental processing and secure transmission of supplementary materials, saving computing resources and network transmission bandwidth. In some embodiments, the aforementioned execution entity may also perform the following steps: The first step is to identify the second encrypted interactive data stream as the first encrypted interactive data stream and continue executing the above interactive steps. In practice, firstly, the second encrypted interactive data stream is obtained. Then, the memory pointer of the second encrypted interactive data stream is assigned to the variable storage location of the first encrypted interactive data stream, overwriting the original data of the first encrypted interactive data stream. Finally, the execution jump is made to the starting point of the above interactive steps, and the updated first encrypted interactive data stream is used to re-execute the interactive steps such as matching result generation, review, or supplementary prompts.

[0071] The above-described embodiments of this disclosure have the following beneficial effects: Through the user-end device interaction method for multimodal materials in some embodiments of this disclosure, efficient parallel review and intelligent supplementation of multimodal materials can be achieved, thereby saving computational power consumption and interaction waiting time caused by repetitive calculations. Specifically, the reasons for high computational power consumption and long interaction waiting time are: existing solutions typically centralize all review calculations in the cloud, and after each supplementation of materials, a full recalculation of all submitted materials and historical review results is required. Simultaneously, the review processes for different modal materials are executed serially, lacking a heterogeneous parallel processing and incremental update mechanism on the end-side, resulting in redundant computational power consumption and requiring users to repeatedly wait for the full re-review to complete in multiple rounds of interaction. Based on this, the user-end device interaction method for multimodal materials in some embodiments of this disclosure first, in response to receiving a service request initiated by the user-end device, collects corresponding multimodal material data, including image data, document data, and voice data. By collecting image, document, and voice data, full modal coverage of government service materials is achieved, avoiding repeated submissions due to missing material types, and improving the completeness of data collection and the first-time pass rate. Then, on the user-end device side, the multimodal material data is processed locally in parallel using a heterogeneous parallel processor to generate a first encrypted interactive data stream. Utilizing a heterogeneous parallel processor on the user-end side completes the parallel structured processing and encryption of the multimodal data, reducing the bandwidth consumption of the original data transmission, shortening the preprocessing waiting time, and ensuring data security before transmission. Next, for the first encrypted interactive data stream, the following interactive steps are executed: On the user-end device side, a matching result is generated based on the first encrypted interactive data stream. Rapid matching of personnel identities with the local database is completed on the user-end side, avoiding invalid data uploads to the cloud, reducing unnecessary network interactions and cloud computing power waste, and improving overall response speed. Secondly, in response to the matching result meeting preset conditions, structured parsing and multi-round cross-validation are performed using a cloud server to generate an audit result, and the audit result is pushed to the user-end device. Cloud computing power and a knowledge base are used to perform in-depth audits on the matched data to ensure the uniformity and accuracy of the audit standards, and the results are pushed to the user in real time, shortening user waiting time. Next, in response to the aforementioned matching result not meeting the preset conditions and the corresponding number of interactions being lower than the preset threshold, the following steps are executed: Incremental supplementary prompt information is generated through the aforementioned cloud server, and corresponding supplementary materials are collected based on the incremental supplementary prompt information. For matching failure scenarios, the cloud automatically generates accurate supplementary guidance and pushes it to the user's end, guiding the user to supplement as needed, avoiding blind supplementation, and reducing invalid interaction rounds. Finally, on the aforementioned user-end device side, incremental fusion processing is performed on the aforementioned supplementary materials using the aforementioned heterogeneous parallel processor to generate a second encrypted interactive data stream.On the user side, incremental fusion and encryption of supplementary materials are performed, processing only the newly added parts rather than all the data, which significantly reduces the amount of redundant calculations and transmission overhead.

[0072] Further reference Figure 2 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a user terminal device interaction device for multimodal materials. These device embodiments are similar to... Figure 1 Corresponding to the method embodiments shown, this user terminal device interaction device for multimodal materials can be specifically applied to various electronic devices.

[0073] like Figure 2 As shown, a user terminal device interaction device 200 for multimodal materials includes: a data acquisition unit 201, a generation unit 202, and an interaction unit 203. The data acquisition unit 201 is configured to: in response to receiving a service request initiated by the user terminal device, acquire corresponding multimodal material data, including image data, document data, and voice data. The generation unit 202 is configured to: on the user terminal device side, utilize a heterogeneous parallel processor to perform local parallel processing on the multimodal material data to generate a first encrypted interactive data stream. The interaction unit 203 is configured to perform the following interaction steps for the first encrypted interaction data stream: on the user terminal device side, generate a matching result based on the first encrypted interaction data stream; in response to the matching result meeting preset conditions, perform structured parsing and multi-round cross-validation processing using a cloud server to generate an audit result, and push the audit result to the user terminal device; in response to the matching result not meeting preset conditions and the corresponding number of interactions being lower than a preset threshold, perform the following steps: generate incremental supplementary prompt information through the cloud server, and collect corresponding supplementary materials based on the incremental supplementary prompt information; on the user terminal device side, perform incremental fusion processing on the supplementary materials using the heterogeneous parallel processor to generate a second encrypted interaction data stream.

[0074] It is understandable that the units described in the user terminal device interaction device 200 for multimodal materials are related to the reference. Figure 1 The steps in the described method correspond accordingly. Therefore, the operations, features, and beneficial effects described above for the method are also applicable to the user terminal device interaction device 200 for multimodal materials and the units contained therein, and will not be repeated here.

[0075] The following is for reference. Figure 3 It shows a schematic diagram of the structure of an electronic device (e.g., an electronic device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0076] like Figure 3 As shown, the electronic device 300 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the electronic device 300. The processing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0077] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 An electronic device 300 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 3 Each box shown can represent a device or multiple devices as needed.

[0078] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, it performs the functions defined in the methods of some embodiments of this disclosure.

[0079] It should be noted that, in some embodiments of this disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0080] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0081] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs. When the electronic device executes the aforementioned one or more programs, the electronic device causes the following actions: In response to receiving a service request initiated by a user terminal device, it collects corresponding multimodal material data, including image data, document data, and voice data; on the user terminal device side, it performs local parallel processing on the multimodal material data using a heterogeneous parallel processor to generate a first encrypted interactive data stream; for the first encrypted interactive data stream, it performs the following interaction steps: on the user terminal device side, it generates a matching result based on the first encrypted interactive data stream; in response to the matching result meeting preset conditions, it performs structured parsing and multi-round cross-validation processing using a cloud server to generate an audit result and pushes the audit result to the user terminal device; in response to the matching result not meeting preset conditions and the corresponding number of interactions being lower than a preset threshold, it performs the following steps: it generates incremental supplementary prompt information through the cloud server and collects corresponding supplementary materials based on the incremental supplementary prompt information; on the user terminal device side, it performs incremental fusion processing on the supplementary materials using the heterogeneous parallel processor to generate a second encrypted interactive data stream.

[0082] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0084] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a generation unit, and an interaction unit. The names of these units do not necessarily limit the unit itself; for example, the acquisition unit, generation unit, and interaction unit may also be described as "a unit that acquires corresponding multimodal material data in response to receiving a service request initiated by a user terminal device."

[0085] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0086] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A user terminal device interaction method for multimodal materials, comprising: In response to receiving a service request initiated by a user terminal device, the corresponding multimodal material data is collected, including image data, document data, and voice data; On the user terminal device side, the multimodal material data is processed locally in parallel using a heterogeneous parallel processor to generate a first encrypted interactive data stream; For the first encrypted interactive data stream, perform the following interactive steps: On the user terminal device side, a matching result is generated based on the first encrypted interactive data stream; In response to the matching result meeting the preset conditions, the cloud server is used to perform structured parsing and multi-round cross-validation to generate the review result and push the review result to the user terminal device; In response to the matching result not meeting the preset conditions and the corresponding number of interactions being less than a preset threshold, the following steps are executed: The cloud server generates incremental replenishment prompts, and the corresponding supplementary materials are collected based on the incremental replenishment prompts. On the user terminal device side, the heterogeneous parallel processor performs incremental fusion processing on the supplementary material to generate a second encrypted interactive data stream.

2. The method according to claim 1, wherein, The method further includes: The second encrypted interactive data stream is identified as the first encrypted interactive data stream, and the interactive steps are continued.

3. The method according to claim 1, wherein, The step of responding to a service request initiated by a user terminal device and collecting corresponding multimodal material data includes: In response to receiving a service request initiated by a user terminal device, determine the material submission method in the service request; Based on the material submission method, the multimodal material data type corresponding to the application request is determined, and the multimodal material data type includes image type, document type and voice type; Based on the type of multimodal material data, the corresponding type of acquisition hardware in the user terminal device is activated, the acquisition hardware including an image sensor, a file reading interface and a microphone array; The corresponding acquisition hardware is used to acquire image data for the image type, document data for the document type, and voice data for the voice type, respectively.

4. The method according to claim 1, wherein, On the user terminal device side, generating a matching result based on the first encrypted interactive data stream includes: The first encrypted interactive data stream is decrypted and decapsulated to generate multimodal fused structured data; Personnel identification information is extracted from the multimodal fused structured data to generate personnel identifiers to be matched; The identifier of the person to be matched is matched with the locally cached personnel database to generate matching similarity information. The matching similarity information is compared with a preset matching threshold to generate a matching result.

5. The method according to claim 1, wherein, The response that the matching result meets preset conditions includes performing structured parsing and multi-round cross-validation processing using a cloud server to generate an audit result, and pushing the audit result to the user terminal device, including: The first encrypted interactive data stream is uniformly decrypted and parsed to generate cloud-readable multimodal fused structured data; Utilize the cloud-stored standard review knowledge base to extract the corresponding set of rules for determining material compliance; Based on the aforementioned material compliance judgment rule set, the cloud-readable multimodal fused structured data undergoes a first round of compliance verification processing to obtain the first verification index data; Using the cloud-based personnel file database, cross-compare the personnel information consistency data of the first verification indicator data to obtain the second verification indicator data. Based on the first verification index data and the second verification index data, multiple rounds of cyclic cross-validation are performed to obtain comprehensive compliance score data. The comprehensive compliance score data is compared with the cloud-preset audit level thresholds to determine the level and generate audit results. Based on the review results, a targeted push interaction instruction is generated to push the review results to the user's terminal device.

6. The method according to claim 1, wherein, The process of generating incremental supplementation prompts via the cloud server and collecting corresponding supplementary materials based on the incremental supplementation prompts includes: The cloud server parses the missing material field identifiers in the first encrypted interactive data stream to generate a list of missing material types; Based on the comparison analysis between the missing material type list and the material requirements in the preset item knowledge base, a differentiated supplementary guidance text is generated; The differentiated supplementary guidance text is subjected to hierarchical semantic packaging processing to generate incremental supplementary prompt information; The incremental supplementary prompt information is pushed to the user terminal device through an encrypted channel, and the corresponding acquisition hardware of the user terminal device is triggered to enter the standby state. In response to the user terminal device receiving a supplementary operation instruction triggered by the user based on the incremental supplementary prompt information, the standby acquisition hardware is activated to execute the incremental acquisition task; The newly acquired material data from the incremental acquisition task is processed for resolution and signal-to-noise ratio detection to generate image quality score data. In response to the image quality score data being lower than a preset quality threshold, a re-sampling instruction is automatically triggered by the acquisition hardware of the user terminal device to obtain optimized new material data. The optimized new material data is subjected to format compliance verification and metadata integrity checks to generate supplementary materials.

7. A user terminal device interaction device for multimodal materials, comprising: The acquisition unit is configured to acquire corresponding multimodal material data in response to receiving a service request initiated by a user terminal device. The multimodal material data includes image data, document data, and voice data. The generation unit is configured to perform local parallel processing on the multimodal material data using a heterogeneous parallel processor on the user terminal device side to generate a first encrypted interactive data stream. The interaction unit is configured to perform the following interaction steps in response to the first encrypted interaction data stream: On the user terminal device side, a matching result is generated based on the first encrypted interactive data stream; In response to the matching result meeting the preset conditions, the cloud server is used to perform structured parsing and multi-round cross-validation to generate the review result and push the review result to the user terminal device; In response to the matching result not meeting the preset conditions and the corresponding number of interactions being less than a preset threshold, the following steps are executed: The cloud server generates incremental replenishment prompts, and the corresponding supplementary materials are collected based on the incremental replenishment prompts. On the user terminal device side, the heterogeneous parallel processor performs incremental fusion processing on the supplementary material to generate a second encrypted interactive data stream.

8. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images