Information processing method, apparatus, device, storage medium, and program product

Abstract

The application provides an information processing method, device, equipment, storage medium and program product, and relates to the field of financial technology or other related fields. The method comprises the following steps: acquiring multi-source business information of a target customer, the multi-source business information being business data collected based on different collection strategies, and the multi-source business information comprising video data, image data and text data; performing frame extraction processing on the video data to obtain a plurality of first images, and performing conversion processing on audio information in the video data to obtain corresponding text information, a first parameter of the first image being greater than a first threshold; performing fusion processing on the plurality of first images, the text information, the image data and the text data according to a first model to obtain fusion information; and generating a quality inspection report corresponding to the target customer based on the fusion information, the quality inspection report being used to indicate a consistency degree between the video data, the image data and the text data. The accuracy of information processing is improved.

Description

Technical Field

[0001] This application relates to the field of financial technology or other related fields, and in particular to an information processing method, apparatus, device, storage medium and program product. Background Technology

[0002] In some scenarios, it is necessary to conduct due diligence on relevant clients to assess their compliance risks. For example, in financial business scenarios, a standardized due diligence report can be generated by collecting various business materials provided by clients to meet compliance review requirements.

[0003] In related technologies, due diligence reports are usually generated manually. For example, when due diligence is required on a client, staff need to manually collect key information from on-site video recordings, image materials submitted by the client, and filled-out text documents. Furthermore, they need to perform various processing operations, such as comparing each item of the collected information, before they can obtain the client's due diligence report.

[0004] However, the above methods require manual collection and organization of information from multiple sources of business materials based on experience, resulting in low accuracy of information processing. Summary of the Invention

[0005] This application provides an information processing method, apparatus, device, storage medium, and program product to solve the technical problem of low accuracy in information processing.

[0006] In a first aspect, this application provides an information processing method, including:

[0007] Obtain multi-source business registration information of target customers. Multi-source business registration information is business data collected based on different collection strategies. Multi-source business registration information includes video data, image data, and text data.

[0008] The video data is processed by frame extraction to obtain multiple first images, and the audio information in the video data is converted to obtain the corresponding text information. The first parameter of the first image is greater than the first threshold. The first parameter is used to indicate the degree of influence of the frame in which the first image is located on the video data.

[0009] Based on the first model, multiple first images, text information, image data, and text data are fused to obtain fused information.

[0010] Based on the fused information, a quality inspection report is generated for the target customer. The quality inspection report is used to indicate the degree of consistency between video data, image data, and text data.

[0011] Secondly, this application provides an information processing apparatus, comprising: an acquisition module, a processing module, a fusion module, and a generation module, wherein,

[0012] The acquisition module is used to acquire multi-source business information of target customers. Multi-source business information is business data collected based on different acquisition strategies, including video data, image data, and text data.

[0013] The processing module is used to perform frame extraction processing on video data to obtain multiple first images, and to perform conversion processing on the audio information in the video data to obtain corresponding text information. The first parameter of the first image is greater than a first threshold. The first parameter is used to indicate the degree of influence of the frame in which the first image is located on the video data.

[0014] The fusion module is used to perform fusion processing on multiple first images, text information, image data and text data according to the first model to obtain fused information;

[0015] The generation module is used to generate a quality inspection report for the target customer based on the fused information. The quality inspection report is used to indicate the degree of consistency between video data, image data and text data.

[0016] Thirdly, embodiments of this application provide an information processing method, including: at least one processor and a memory; the memory stores computer execution instructions; at least one processor executes the computer execution instructions stored in the memory, causing at least one processor to perform the information processing method as described in the first aspect above and any one of the first aspects.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the information processing method described in the first aspect and any one of the first aspects.

[0018] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements the information processing method involved in the first aspect and any one of the first aspects.

[0019] The information processing method, apparatus, equipment, storage medium, and program products provided in this application acquire multi-source business information of a target customer. This multi-source business information comprises business data collected based on different acquisition strategies, including video data, image data, and text data. The method involves frame extraction processing of the video data to obtain multiple first images, and conversion processing of the audio information in the video data to obtain corresponding text information. A first parameter of each first image is greater than a first threshold, indicating the degree of influence of the frame containing the first image on the video data. Based on a first model, the multiple first images, text information, image data, and text data are fused to obtain fused information. Based on the fused information, a quality inspection report corresponding to the target customer is generated. This quality inspection report indicates the degree of consistency between the video data, image data, and text data. In this method, electronic equipment can automatically extract key information from multi-source heterogeneous data and achieve cross-modal consistency verification through multimodal fusion, thereby quickly generating a standardized and objective audit report without the need for manual item-by-item comparison. This method effectively avoids problems such as low efficiency and information bias caused by subjective differences in manual processes, significantly improves the processing efficiency and accuracy of due diligence data, unifies the review standards, reduces operational risks and compliance risks, and is applicable to business materials from different sources and in different formats, demonstrating good generalization ability and practicality. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0021] Figure 1 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0022] Figure 2 A flowchart illustrating an information processing method provided in an embodiment of this application;

[0023] Figure 3 A schematic diagram illustrating the process of generating fused information provided in the embodiments of this application;

[0024] Figure 4 A schematic diagram illustrating the process of generating a quality inspection report as provided in this application embodiment;

[0025] Figure 5 A flowchart illustrating another information processing method provided in an embodiment of this application;

[0026] Figure 6 A flowchart illustrating another information processing method provided in an embodiment of this application;

[0027] Figure 7This is a schematic diagram of the structure of an information processing device provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0029] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant regions, have taken necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse.

[0032] Furthermore, the technical solution involved in this application, which involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.) and the use of artificial intelligence technology for automated decision-making, and makes decisions that have a significant impact on personal rights based on the results of automated decision-making, provides users with corresponding operation entry points for users to choose to agree to or reject the results of automated decision-making; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0033] It should be noted that the information processing methods, apparatus, equipment, storage media and program products provided in this application can be used in the field of financial technology, or in any field other than financial technology. The application fields of the information processing methods, apparatus, equipment, storage media and program products in this application are not limited.

[0034] To facilitate understanding, the following will be combined with... Figure 1 The application scenarios applicable to the embodiments of this application will be described.

[0035] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application. Please refer to [link / reference]. Figure 1 This includes terminal equipment and electronic equipment.

[0036] Users can input multi-source business information to be processed into the terminal device. The terminal device can send the multi-source business information input by the user to the electronic device. The electronic device can process the multi-source business information. After the electronic device obtains the quality inspection report corresponding to the multi-source business information, the electronic device sends the processing result, i.e., the user's quality inspection report, to the terminal device.

[0037] Terminal devices can be hardware with information input and display functions, such as mobile phones, tablets, and laptops. Target applications can be installed on these devices, allowing users to input information; for example, a target application could be a banking application. Electronic devices can be those with on-device computing capabilities, such as servers.

[0038] In related technologies, due diligence reports are typically generated manually. For example, when conducting due diligence on a client, staff need to manually collect key information from on-site video recordings, images submitted by the client, and completed text documents. Furthermore, they need to perform various processing operations, such as comparing each item, to obtain the client's due diligence report. However, this method relies on manual experience to collect and process information from multiple sources, leading to lower accuracy in information processing.

[0039] To address the aforementioned technical issues, this application embodiment obtains multi-source business information of the target customer, extracts keyframe images from video data, converts audio information in the video into text information, and then performs multimodal fusion processing on the keyframe images, audio text, image data, and text data according to a first model to generate a quality inspection report indicating the degree of consistency between the video, images, and text. In this method, the electronic device automatically extracts keyframes and audio text from the video, achieving efficient preprocessing of unstructured data. By utilizing the first model for cross-modal feature alignment and fusion inference, it can accurately identify semantic contradictions or inconsistencies between information from different sources, effectively avoiding problems such as low manual efficiency and information bias caused by subjective differences, significantly improving the processing efficiency and accuracy of due diligence generation. Furthermore, by integrating multi-dimensional data, real-time cross-validation of multi-source data is achieved, eliminating the need for extensive manual information collection, organization, and item-by-item verification. This reduces labor costs, simplifies business processes, and lowers operational and compliance risks. Moreover, this method is applicable to business materials of different formats and sources, enhancing the versatility and scalability of the solution. It significantly improves information processing efficiency, accuracy, and system reliability while enhancing the accuracy of consistency judgments.

[0040] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0041] Figure 2 This is a flowchart illustrating an information processing method provided in an embodiment of this application. Please refer to [link / reference]. Figure 2 As shown, the method may include the following steps:

[0042] S201. Obtain multi-source business registration information of target customers.

[0043] The execution subject of this application embodiment can be an electronic device or an information processing device installed in an electronic device. The information processing device can be implemented by software or by a combination of software and hardware.

[0044] Among them, multi-source business information refers to business data collected based on different collection strategies. That is, it can be understood as acquiring information reflecting the business status, compliance risks and business intentions of the same target customer from three different data sources: video, images and text.

[0045] Multi-source business information includes video data, image data, and text data.

[0046] Target customers can refer to corporate clients who require due diligence, such as enterprises, sole proprietorships, or other types of organizations.

[0047] Video data can refer to continuous video recordings of target customers collected by staff, and its content should include key information such as the target customer appearing on camera to state their business situation.

[0048] Image data can refer to images taken of static scenes such as the business environment of a target customer. For example, image data may include the business information of the target customer.

[0049] Text data can refer to documents related to the business operations information of the target customer. For example, text data may include the address information of the business operated by the target customer.

[0050] Data collection strategy refers to the specific methods used to collect various types of business information. Different data modalities employ different data collection strategies due to their different physical forms and business acquisition methods.

[0051] For example, video data can be collected using the following strategy: Staff members, equipped with mobile devices that can record audio and video (e.g., tablets or dedicated recording equipment), are at the client's actual business location. Following pre-defined operating procedures, they guide the client to verbally describe their business operations and other relevant information to the camera, recording the entire process continuously to create an unedited raw video file. During the collection process, it is essential to ensure sufficient lighting, stable video, clear audio, and that the client's statements are synchronized with the video presentation in time.

[0052] Image data can be collected using the following strategy: staff will use digital cameras or mobile devices with high-resolution cameras to independently photograph the client's business environment and other commercial information, according to the due diligence checklist. Each photograph should ensure that the subject is clear, the text is legible, there are no obstructions, and there are no signs of editing. After shooting, the images should be named and stored according to category, and submitted to the electronic device as independent image files.

[0053] Text data can be collected using the following strategy: Clients fill out structured documents related to their business registration information using the provided standard forms or online filing system, and prepare various supporting materials (e.g., address information, contact information, etc.). After confirmation, all text materials are submitted by scanning them into PDFs, taking photos as images, or directly uploading electronic documents. Upon receiving the documents, the system converts them into a parsable text format for subsequent processing.

[0054] In some embodiments, the electronic device can receive data packets uploaded from a front-end acquisition terminal (e.g., a mobile recording device, digital camera, scanner, or online form filling system) via a communication interface. Specifically, video data can be uploaded to the electronic device via a mobile terminal; image data can be uploaded by the user in batches or individually; and text data can be uploaded and submitted to the electronic device via the user's terminal device. The electronic device performs format verification, integrity checks, and metadata extraction on the received data, and stores it in a data directory corresponding to the unique identifier of the target customer, thus completing the acquisition of multi-source business information.

[0055] S202. Perform frame extraction processing on the video data to obtain multiple first images, and perform conversion processing on the audio information in the video data to obtain the corresponding text information.

[0056] Wherein, the first parameter of the first image is greater than the first threshold.

[0057] The first parameter is used to indicate the degree of influence of the frame containing the first image on the video data.

[0058] The first image can refer to a keyframe image automatically extracted from video data that contains significant changes in the scene. These keyframes represent moments in the video that are rich in information, such as the appearance of address information.

[0059] Audio information can refer to sound signals extracted from video data. For example, audio information can include the content of a customer's spoken statements, ambient sounds, etc.

[0060] Text information can refer to the text content converted from audio information through speech recognition technology, such as the text representation of key fields like the company address information spoken by a customer.

[0061] The first threshold is used to determine whether the difference between two adjacent frames is significant enough. When the first parameter is greater than this threshold, it indicates that a significant scene change has occurred, and the frame should be retained as a keyframe.

[0062] In some embodiments, an electronic device may perform frame extraction processing on video data to obtain multiple first images based on the following method: acquiring the total number of frames and video duration of the video data; determining the target number of extracted frames based on preset frame information, the total number of frames, and the video duration, and determining the sampling interval based on the target number of extracted frames; performing frame sampling on the video data according to the sampling interval to obtain multiple sampled frames; and determining multiple first images based on the multiple sampled frames.

[0063] Total frames can refer to the total number of image frames contained in the video data, denoted as . .

[0064] Video duration refers to the total playback time of a video from start to finish, measured in seconds. Its calculation method can be expressed as:

[0065]

[0066] in, This indicates the video's frame rate (frames per second).

[0067] The preset frame information can refer to a pre-configured set of parameters used to control the keyframe extraction density. The frame information can include the expected number of frames per minute (denoted as ). ) and the optional maximum total number of frames (denoted as ).

[0068] in, This indicates the desired number of keyframes to extract from each minute of video. A hard limit is set to prevent extremely long videos from generating too many keyframes.

[0069] The target number of extracted frames can refer to the final number of keyframes expected to be extracted from the video, denoted as This value is used to balance information integrity and computational efficiency, ensuring that neither important images are missed nor redundancy is generated.

[0070] The sampling interval is used to determine how many frames to sample when performing equal-interval sampling on a video, thus avoiding the enormous computational burden of frame-by-frame processing. The sampling interval is denoted as... .

[0071] A sampled frame refers to a candidate frame extracted from a video according to a sampling interval. These sampled frames have not yet undergone difference filtering and require further processing before they can become the final first image.

[0072] In some embodiments, to adaptively control the number of keyframes and ensure that videos of different durations and content can extract keyframes of appropriate density, avoiding oversampling of short videos or omission of information in long videos due to a fixed number, the electronic device can calculate a preliminary expected number of frames based on the video duration and the desired frame rate per minute. This expected number of frames is then compared with the total number of frames and a preset maximum total number of frames, and the minimum of these three values ​​is taken as the final target number of extracted frames. The relevant formula can be expressed as:

[0073]

[0074] in, This indicates that the video duration has been converted from seconds to minutes. This indicates the pre-configured expected number of frames to be extracted per minute (e.g., 2 frames / minute). Indicates the total number of frames in the video. This indicates the maximum possible total number of frames (this is ignored if not specified). In this way, the target number of frames to be extracted meets the business requirements for extraction density, and will not exceed the total number of frames in the video or the preset limit.

[0075] In some embodiments, to avoid frame-by-frame processing, equally spaced sampling can be used. The sampling step size is derived by working backward from the number of target extracted frames, ensuring that the number of sampled frames is twice the number of target frames, thus reserving sufficient redundant samples for subsequent difference screening. That is, the sampling interval can be calculated from the total number of frames and the number of target extracted frames, and the relevant formula can be expressed as:

[0076]

[0077] in, The first term indicates that the minimum sampling interval is 1 to avoid sampling anomalies caused by an interval of 0. The second term indicates the interval value that makes the number of candidate frames obtained by sampling approximately twice the number of target extracted frames.

[0078] In this way, by using coarse sampling at equal intervals, the computational cost of comparing differences in subsequent frames is reduced from the total number of frames to the number of candidate frames (twice the number of target extracted frames). This provides sufficient selection space for subsequent filtering based on difference scores without significantly increasing computational power consumption. Subsequently, the effective frames with the highest difference can be accurately selected from a large number of sampled frames. This avoids the high computational power consumption caused by traversing frame by frame and solves the problem of missing key scenes caused by sparse sampling, thus greatly improving the accuracy of key frame filtering.

[0079] In some embodiments, the electronic device can extract frames sequentially from the 0th frame (or the first frame) of the video, starting with the sequence number 0, the sampling interval, 2×sampling interval, 3×sampling interval, and so on, until the end of the video. Each extracted frame is stored as a sampled frame. For example, if the sampling interval is 100, then frame 0, frame 100, frame 200, and so on are extracted. These sampled frames together constitute the sampled frame set. Since the sampling interval may not be an integer, the sequence number is rounded down or an integer index is used during actual processing to ensure that all frame sequence numbers are within a valid range.

[0080] In some embodiments, the electronic device may determine a plurality of first images based on a plurality of sampling frames in the following manner: performing grayscale processing on the plurality of sampling frames to obtain grayscale images corresponding to each sampling frame; for any sampling frame, determining the grayscale difference between the grayscale images of the current sampling frame and the previous sampling frame, and determining a first parameter based on the grayscale difference, the first parameter being used to indicate the degree of difference between adjacent sampling frames; among the plurality of sampling frames, determining the sampling frames whose first parameter is greater than a first threshold as candidate sampling frames; and determining a plurality of first images based on the plurality of candidate sampling frames.

[0081] A grayscale image refers to an image where the original color sampled frame (usually RGB three-channel) is converted into a single-channel luminance image, with each pixel having a grayscale value ranging from 0 to 255. Grayscale conversion reduces the amount of data while preserving the structural information used to compare differences between frames.

[0082] Gray-scale difference can refer to the absolute value of the difference between the gray-scale values ​​of corresponding pixels in two gray-scale images.

[0083] The first parameter refers to the value obtained by averaging all elements in the grayscale difference matrix. This value quantifies the overall visual difference between the current sampled frame and the previous sampled frame. The value of the first parameter ranges from 0 to 255, with a larger value indicating a more significant difference between the two frames.

[0084] Candidate sampling frames refer to those frames in the sampling frame set whose first parameter exceeds a first threshold after calculation. Candidate sampling frames are initially selected candidate frames that may contain important visual information, and they need to be further sorted and selected to become the final first image (keyframe).

[0085] In some embodiments, the electronic device can convert the current sampling frame and the previous sampling frame into grayscale images, and then calculate the absolute value of the difference between the grayscale values ​​at the same position in the two grayscale images pixel by pixel, resulting in an absolute difference matrix with the same size as the image. Each element in this matrix is ​​the grayscale difference value of the corresponding pixel. For example, if the value at position (x,y) in the current frame's grayscale image is 128, and the value at the same position in the previous frame is 100, then the grayscale difference value at that position is 28.

[0086] In some embodiments, after obtaining the grayscale difference matrix, the electronic device sums the grayscale differences of all pixels in the matrix and then divides the sum by the total number of pixels in the image (width × height) to obtain the average value, which is the first parameter. This first parameter reflects the intensity of the average pixel change between two sampling frames; the more drastic the change, the larger the first parameter.

[0087] In some embodiments, the first threshold can be set to a fixed value. However, the complexity of different video content varies greatly. For dynamic videos with frequent scene changes, customer movement, or extensive device display, a fixed threshold may lead to the omission of keyframes; while for long periods of static footage, a fixed threshold may retain too many redundant frames. To overcome these problems, the first threshold can also be dynamically adjusted.

[0088] Specifically, before calculating the first parameter of the sampled frame, the electronic device can first analyze the complexity of the video content. Complexity analysis can be performed from two dimensions: temporal features and spatial features. Temporal features include inter-frame motion vectors (calculated by optical flow) or scene switching frequency; spatial features include image edge density (e.g., the proportion of non-zero pixels after edge detection) or texture complexity (e.g., the contrast of the gray-level co-occurrence matrix).

[0089] Subsequently, the electronic device dynamically calculates a first threshold applicable to the current video based on the overall or local complexity of the video. For example, an initial threshold (e.g., 30) is set, and the initial threshold is scaled according to a normalized complexity factor, for example, first threshold = initial threshold × (1 - k × complexity), where k is an adjustment coefficient, and the complexity is normalized to the [0,1] interval. When the video complexity is high, the threshold is lowered to retain more candidate frames; when the video complexity is low, the threshold is raised to reduce redundant frames.

[0090] Furthermore, electronic devices can employ segmented processing, dividing the video into multiple independent segments based on scene transition points, calculating the complexity of each segment separately, and independently determining the first threshold for that segment. In this way, appropriate thresholds are applied to dynamic and static segments within the same video, effectively avoiding information omissions or redundancy caused by a single global threshold.

[0091] In this way, dynamic thresholds can be used to adapt to the complexity of different video content, reducing redundant frame processing while retaining key scenes and significantly reducing computational resource consumption; the segmented processing strategy ensures that key frames of both dynamic and static scenes are retained, improving the integrity of information extracted from key frames.

[0092] In some embodiments, after obtaining candidate sampling frames, the number of candidate sampling frames may be greater than the desired number of target extraction frames. The electronic device can sort the candidate sampling frames in descending order according to a first parameter, so that the frames with the most significant differences are placed first. Then, the top few candidate sampling frames from the sorted list are taken as the initially selected frames, an output path is generated, and it is saved as the first image.

[0093] If a maximum total number of frames is preset, and the number of initially selected frames exceeds this maximum, the initially selected frames are uniformly downsampled according to their original temporal order in the video, ensuring that the final number of frames does not exceed the maximum total number of frames. Otherwise, the initially selected frames are directly used as the final multiple first images. In this way, the electronic device can extract the set of keyframes with the richest information and most significant visual changes from the video, ensuring the integrity of key content while controlling the total number of keyframes and avoiding redundancy.

[0094] S203. Based on the first model, multiple first images, text information, image data and text data are fused to obtain fused information.

[0095] The first model can refer to a pre-trained multimodal machine learning model. In other words, the first model can be an artificial intelligence model built on a deep learning architecture that can simultaneously process, understand, and associate multiple modal information, including but not limited to text, images, speech, and video. Its core function is to transform unstructured image data into feature vectors that can be analyzed in a structured way, so as to achieve semantic alignment and joint reasoning of cross-modal information.

[0096] Information fusion can refer to the structured results output by the first model after comprehensive analysis and feature fusion of the input multi-source data.

[0097] In some embodiments, electronic devices may not rely on a complete end-to-end multimodal large model, but instead use independent lightweight models. Specifically, the first model may include multiple sub-models. For example, a text recognition model may extract text from an image, a speech recognition model may convert audio, and a text classification model may extract key fields. The extracted structured text fields are then subjected to string matching or semantic similarity calculation based on word vectors. Finally, all comparison results are aggregated into a single structured data set as fused information. This approach is suitable for scenarios with high real-time requirements and limited hardware resources.

[0098] S204. Based on the integrated information, generate the quality inspection report corresponding to the target customer.

[0099] The quality inspection report is used to indicate the degree of consistency between video data, image data, and text data.

[0100] A quality inspection report can be a structured electronic document used to record the review conclusions of multi-source business registration information for target customers. This report may include comparison results of key information (e.g., whether the business address matches), descriptions of specific inconsistencies in fields, and final business processing recommendations (e.g., approve, require manual review, or reject). The report can be output in a format tailored to actual needs, facilitating storage, display, or integration with other business systems.

[0101] In some embodiments, after acquiring the integrated information, the electronic device extracts core characterization information from multi-source business data in batches, including key assessment information such as content characteristics, information matching status, and overall consistency characteristics of various data. Subsequently, it calls a preset standardized report generation template, which defines the fixed framework, display dimensions, content format, and output specifications of the quality inspection report, adapting to the general business needs of commercial information verification. The electronic device fills the extracted information into the corresponding modules of the report template, automatically assembling, formatting, and standardizing the report content. Finally, it performs compliance verification on the assembled report content, removing invalid and redundant information, standardizing the report's expression format, and ultimately generating a complete and standardized quality inspection report for the target customer.

[0102] In this embodiment, by acquiring multi-source business information of the target customer, keyframe images are obtained from video data through frame extraction processing. Audio information in the video is converted into text information. Then, based on a first model, multimodal fusion processing is performed on the keyframe images, audio text, image data, and text data to generate a quality inspection report indicating the degree of consistency between the video, images, and text. In this method, the electronic device achieves efficient preprocessing of unstructured data by automatically extracting keyframes and audio text from the video. By utilizing the first model for cross-modal feature alignment and fusion inference, it can accurately identify semantic contradictions or inconsistencies between information from different sources, effectively avoiding problems such as low manual efficiency and information bias caused by subjective differences, significantly improving the processing efficiency and accuracy of due diligence. Furthermore, by integrating multi-dimensional data, real-time cross-validation of multi-source data is achieved, eliminating the need for extensive manual information collection, organization, and item-by-item verification. This reduces labor costs, simplifies business processes, and lowers operational and compliance risks. Moreover, this method is applicable to business materials of different formats and sources, enhancing the versatility and scalability of the solution. It significantly improves information processing efficiency, accuracy, and system reliability while enhancing the accuracy of consistency judgments.

[0103] exist Figure 2 Based on the embodiments shown, the following, in conjunction with Figure 3 The process of fusing multiple first images, text information, image data, and text data to obtain fused information in the above information processing method is explained in detail.

[0104] Figure 3 This is a schematic diagram illustrating the process of generating fused information according to an embodiment of this application. Please refer to... Figure 3 ,include:

[0105] S301. Input multiple first images, text information, image data and text data into the first model to obtain multimodal feature information that integrates video data, image data and text data.

[0106] Feature information refers to the vectorized representation obtained by the first model after encoding the input data. These vectors are located in the model's latent space and can capture the semantic features and cross-modal correlation information of the original data. Feature information can be a multidimensional array (feature vector) or a sequence of feature vectors.

[0107] In some embodiments, the electronic device may input multiple first images, text information, image data, and text data into a first model in the following manner to obtain multimodal feature information that integrates video data, image data, and text data: The first model performs visual encoding processing on multiple first images and image data to obtain first feature information corresponding to each image, and performs semantic encoding processing on text information and text data to obtain second feature information corresponding to each text information and third feature information corresponding to each text data; it then performs feature alignment processing on multiple first feature information, multiple second feature information, and multiple third feature information to obtain a multimodal feature sequence; finally, it concatenates the multimodal feature sequence and performs cross-modal semantic fusion on the concatenated multimodal feature sequence to obtain multiple feature information.

[0108] The first feature information can refer to the sequence of feature vectors obtained by the visual encoder of the first model after encoding each input image (including keyframes extracted from videos and photos of business premises, qualification documents, etc. uploaded by target customers). Each first feature information corresponds to one image and is usually composed of feature vectors from multiple image patches, preserving the local structure and global semantics of the image.

[0109] The second feature information can refer to the sequence of feature vectors obtained by the semantic encoder of the first model after encoding the text information transcribed from the video audio. This feature information captures semantic information in the customer's spoken content, such as the business address.

[0110] The third feature information can refer to the sequence of feature vectors obtained by the semantic encoder of the first model after encoding the text materials submitted by the customer. This feature information reflects the key fields and overall semantics of the customer's written declaration.

[0111] A multimodal feature sequence can refer to a unified sequence formed by aligning the first, second, and third feature information to place them in the same vector space. In this sequence, the feature vectors of different modalities are arranged in a certain order, and modality type labels or positional encodings may be added to facilitate subsequent cross-modal fusion.

[0112] In some embodiments, for the first feature information, the electronic device scales multiple first images and image data to a preset input size of the first model, and then sequentially inputs them into the visual encoder of the first model. The visual encoder divides each image into image blocks of a fixed size. Each image block is linearly projected and positionally encoded to form an initial feature vector. Then, a multi-layer self-attention mechanism is used to extract local and global features within the image, and finally outputs a feature vector sequence (each sequence element corresponds to an image block) as the first feature information of the image.

[0113] For text information (audio transcription results) and text data (customer-submitted materials), the electronic device first performs word segmentation and standardization on the text, and then inputs the processed text into the semantic encoder of the first model. The semantic encoder can adopt a Transformer-based language model, converting each word or subword into a corresponding feature vector, and fusing contextual information through a multi-layer self-attention mechanism to output a sequence of feature vectors (each vector corresponding to a word or subword) as the second or third feature information.

[0114] In some embodiments, after obtaining the first, second, and third feature information, since their feature spaces are not yet aligned due to their respective origins from the visual and semantic encoders, direct comparison or concatenation is invalid. Therefore, the first model internally includes one or more learnable projection layers (e.g., a linear transformation network or a more complex mapping module). The electronic device inputs the first, second, and third feature information into their respective projection layers, mapping them into a shared semantic space of the same dimension, such that semantically similar cross-modal features are close in distance within that space. After mapping, the electronic device concatenates the obtained aligned feature vectors into a long sequence according to a preset order (e.g., first all image features, then audio text features, and finally customer text features), and adds modal identifier encoding and position encoding to each position, forming a multimodal feature sequence.

[0115] In some embodiments, the electronic device inputs the concatenated multimodal feature sequence into the Transformer network (i.e., the cross-modal fusion layer) of the first model. This Transformer network consists of multiple stacked self-attention layers and feedforward layers. Through the self-attention mechanism, each feature vector in the sequence can pay attention to feature vectors at all other positions in the sequence, enabling deep fusion of information from different modalities. After processing by multiple Transformer layers, each feature vector in the sequence contains contextual information from both visual and textual modalities. The electronic device can use the entire output sequence as multiple feature information (each position corresponding to a fused feature vector), or it can extract specific positions in the sequence as global fusion features. These outputs are collectively referred to as multiple feature information and are used for subsequent cross-validation.

[0116] S302. Perform cross-validation on multiple feature information to obtain the validation results.

[0117] The verification results are used to verify the matching degree between video data, image data, and text data. In other words, the verification results can be understood as a conclusion drawn after judging the consistency of information from different sources, such as whether addresses match, company names are consistent, or identity information is consistent.

[0118] In some embodiments, the electronic device may perform cross-validation processing on multiple feature information to obtain a verification result based on the following method: calculating the similarity between the feature information corresponding to the audio information in the video data and the feature information corresponding to the text data to obtain a first verification result; matching the feature information corresponding to multiple first images and image data with the feature information corresponding to the text data to obtain a second verification result; matching the feature information corresponding to multiple first images and image data, the feature information corresponding to the audio information in the video data, and the feature information corresponding to the text data to obtain a third verification result; and determining the first verification result, the second verification result, and the third verification result as the verification result.

[0119] The first verification result can refer to the output of the voice-to-text consistency verification, used to indicate whether there are any contradictions between the content of the customer's oral statements and the written application materials. For example, whether the business address extracted from the video / audio matches the registered address filled in by the customer in the written materials.

[0120] The second verification result can refer to the output of image-text consistency verification, which indicates whether the visual information recognized by the image (e.g., house number) matches the corresponding field in the customer's text material. For example, whether the house number on the house number plate matches the business address number in the text material.

[0121] The third verification result can refer to the output of the three-way cross-verification of image, voice, and text, which is used to jointly compare whether the business information in the image, voice, and text belongs to the same entity. For example, whether the identification information shown in the video, the identification information in the audio information, and the identity information filled in the text materials are consistent.

[0122] In some embodiments, the electronic device extracts a fused representation of the second feature information corresponding to the video audio-to-text transcription (e.g., a pooled vector or classification label vector of the entire text sequence) and a fused representation of the third feature information corresponding to the customer's text data from the multiple feature information obtained in S301. Then, the semantic similarity between these two feature vectors is calculated, for example, using cosine similarity or dot product similarity. If the similarity is greater than or equal to a preset threshold (e.g., 80%), the speech and text information are determined to be consistent, and the first verification result is passed; otherwise, it is determined to be failed, and the first verification result is either address information inconsistency or speech-text inconsistency, and the specific mismatch field is recorded.

[0123] In some embodiments, the electronic device extracts a fused representation of visual feature information corresponding to the first image and customer image data from multiple feature information sources (e.g., performing global average pooling on the feature sequence of each image to obtain an image-level feature vector), and a fused representation of third feature information corresponding to the customer text data. Then, cross-modal matching is performed according to task requirements. One approach is to calculate the cosine similarity between image features and text features; if it exceeds a threshold, a match is determined. Another approach is to utilize the cross-modal attention weights within the first model to locate the region in the image most relevant to the text description (e.g., when recognizing the text of a house number, the model gives a higher attention score to the house number region in the image); if the attention score is higher than a threshold, a match is determined. If the house number matches the number in the text address, and the company logo in the image matches the company name in the text, the second verification result is passed; otherwise, it is marked as abnormal business information or image-text inconsistency.

[0124] In some embodiments, electronic devices extract visual features from video keyframes, features from audio-to-text transcription, and features filled in from textual materials. One approach is to directly calculate the pairwise similarity between these three feature vectors and take the minimum value as the overall consistency score. Another approach is to use the multimodal attention mechanism of the first model to construct a cross-attention matrix containing images, audio-text, and documents, and directly output a triplet consistency score through forward computation of the model. If the three features are close to each other in the shared semantic space, the third verification result is passed; otherwise, it is marked as inconsistent information among the three parties.

[0125] S303. The feature information and the verification results are determined as fused information.

[0126] Fusion information refers to the comprehensive data structure formed by combining the multimodal feature information obtained in the aforementioned steps (i.e., the multiple feature information output by S301) with the cross-validation results (i.e., the first, second, and third validation results output by S302). This fusion information includes both the deep semantic representation of the original data and the consistency judgment conclusion obtained after reasoning about these representations, and serves as the direct basis for generating the subsequent quality inspection report.

[0127] In this embodiment, video keyframes, audio-transcribed text, customer-uploaded images, and customer-submitted text data are input into a first model. Visual and semantic encoding are performed respectively to obtain multimodal feature information. This information is then fused through feature alignment, sequence concatenation, and cross-modal semantic fusion to generate fused feature information. Based on this, cross-validation is performed on audio and text, image and text, and image-audio-text pairs to obtain validation results. Finally, the feature information and validation results are determined as the fused information. In this method, the electronic device utilizes the first model to achieve automatic alignment and deep fusion of heterogeneous multi-source data. The three-way cross-validation mechanism accurately identifies semantic contradictions or inconsistencies between voice statements, image content, and application materials, effectively avoiding the problems of low efficiency, strong subjectivity, and high false negative rates associated with manual comparison. This significantly improves the accuracy and efficiency of multi-source information consistency judgment in due diligence. Furthermore, this solution eliminates the need for extensive manual verification, unifies review standards, reduces operational risks and compliance hazards, and simplifies business processes by reducing reliance on repetitive manual intervention through an end-to-end feature extraction and verification process. This method is applicable to business materials of different formats and sources, and has good generalization ability and scalability. While improving the accuracy of cross-modal consistency judgment, it significantly improves the automation level of information processing and system reliability.

[0128] exist Figure 3 Based on the embodiments shown, the following, in conjunction with Figure 4 The process of generating quality inspection reports for target customers based on integrated information is explained in detail.

[0129] Figure 4 This is a schematic diagram illustrating the process of generating a quality inspection report as provided in an embodiment of this application. Please refer to [link / reference]. Figure 4 ,include:

[0130] S401. Based on the preset knowledge base, the fused information is verified to obtain the verification result.

[0131] The preset knowledge base is used to store business information of multiple customers.

[0132] The verification result refers to the conclusion of authenticity verification obtained by comparing key fields in the fused information with data in a preset knowledge base. The verification result can include whether it passes or fails, as well as a description of the specific mismatched fields.

[0133] In some embodiments, the electronic device extracts key fields related to customer identity and business qualifications from the fused information, such as company name and registered address obtained from image recognition or text parsing. The electronic device accesses an external, pre-defined knowledge base through a preset interface, using these key fields as query conditions, and compares each data item in the knowledge base. After all comparisons are completed, the electronic device summarizes the comparison results. If all key fields match the knowledge base, the verification result is considered passed; if any field does not match or there is no corresponding record in the knowledge base, the verification result is considered failed, and the specific mismatched field is recorded. This verification result serves as a supplement to the fused information and is used for subsequent report generation.

[0134] S402. When the verification result indicates that the verification failed, and / or the check result indicates that the check failed, identify the abnormal information in the multi-source business information, mark the abnormal information as an abnormal state, and generate a quality inspection report.

[0135] Abnormal information can refer to fields or materials in multi-source business information that are inconsistent with internal cross-validation standards or external knowledge bases. For example, contradictory address information such as voice address and text address.

[0136] An abnormal status refers to an identifier added to abnormal information, used to indicate a problem with that information in the quality inspection report. Abnormal statuses can include specific types such as address discrepancies, name mismatches, and inconsistencies in business registration information, or they can be uniformly marked as requiring review or abnormal.

[0137] In some embodiments, the electronic device first obtains the verification results in S302 (including the first, second, and third verification results) and the verification results in S401. Based on the combination of the two, abnormal information is determined, specifically including three cases.

[0138] Specifically, scenario one involves a verification result indicating verification failed, while a validation result indicates validation passed. For example, internal multimodal cross-validation found an inconsistency between the voice address and the image address (verification failed), but the external knowledge base confirmed that the address entered by the customer is real and consistent with the business registration (validation passed). In this case, the anomaly is an internal cross-modal contradiction, specifically an inconsistency between the voice and image / text fields. The electronic device marks these fields as abnormal and notes in the report that there is an internal comparison anomaly, recommending manual review.

[0139] Scenario two involves a verification result indicating that verification passed, but a validation result indicating that validation failed. For example, internal cross-validation shows that all source information is consistent (verification passed), but the external knowledge base finds that the information provided by the customer does not exist (validation failed). In this case, the anomaly is an external authenticity anomaly, specifically a key field that does not match the knowledge base. The electronic device marks these fields as an anomaly and notes in the report that the external verification failed.

[0140] Scenario 3 is where the verification result indicates that the verification failed, and the validation result also indicates that the validation failed. For example, internal multimodal comparison finds an address contradiction, while the external knowledge base also confirms that the address does not exist or does not match the business registration. In this case, the abnormal information includes internal contradictory fields and external inconsistent fields. The electronic device marks all relevant fields as abnormal and suggests relevant information in the report.

[0141] After identifying the anomalies, the electronic device generates a quality inspection report. The report includes at least the overall audit conclusion, a list of anomalies (each anomaly item includes a field name, anomaly type, and anomaly status), scoring data for the anomalies (scored according to the anomaly level and preset scoring criteria), and an index of the corresponding original materials (e.g., pointing to specific keyframe images or text paragraphs). The quality inspection report can be in a structured or visual format.

[0142] S403. When the verification result indicates that the verification has passed, and the validation result indicates that the validation has passed, mark the abnormal information as normal and generate a quality inspection report.

[0143] In some embodiments, when the verification result indicates that the verification is passed (i.e., the first verification result, the second verification result, and the third verification result are all passed) and the validation result indicates that the validation is passed (i.e., all key fields match the preset knowledge base), it indicates that the target customer's multi-source business information has no abnormalities in terms of both internal cross-modal consistency and external authenticity.

[0144] At this point, the electronic device marks all fields in the multi-source business registration information as normal, without generating any abnormal information. Then, the electronic device generates a quality inspection report, which includes the overall audit conclusion (e.g., audit passed), consistency score, the pass status of each verification item and check item, and final business recommendations (e.g., subsequent business can be processed). This report can be directly used for automated decision-making or archiving.

[0145] In this embodiment, by acquiring fused information and verifying it according to a preset knowledge base, a verification result is obtained. When the verification result indicates that the verification failed and / or the verification result indicates that the verification failed, the abnormal information in the multi-source business information is marked as an abnormal state and a quality inspection report is generated. When both the verification result and the verification result indicate that the verification passed, the abnormal information is marked as a normal state and a quality inspection report is generated. In the above method, the electronic device can combine internal multimodal cross-validation with an external knowledge base to perform dual verification of the customer's key information, effectively avoiding the problem of failing to detect false information by relying solely on internal comparison, and significantly improving the authenticity verification capability of due diligence. By introducing a preset knowledge base for automatic verification, there is no need for manual querying of external systems one by one, reducing the time cost and operational risks of cross-system data verification, and it can capture abnormal information inconsistent with authoritative data in real time, realizing pre-warning and in-process interception. This method is applicable to business materials from different sources and in different formats, enhancing the risk control capability and audit reliability of the solution. On the basis of improving the accuracy of cross-modal consistency judgment, it further improves the automation level and compliance risk management capability of due diligence.

[0146] Figure 5 This is a flowchart illustrating another information processing method provided in an embodiment of this application. Please refer to... Figure 5 As shown, the method may include the following steps:

[0147] S501. Obtain multi-source business registration information of target customers.

[0148] Among them, multi-source business information can refer to due diligence data and multi-source business information corresponding to target customers. It is business verification data collected and aggregated at the business site based on differentiated collection strategies. Specifically, it includes three types of heterogeneous original data: video data, image data, and text data. It is the original data foundation for carrying out multimodal information verification and compliance judgment.

[0149] S502, Multi-source data preprocessing and structured extraction.

[0150] Specifically, structured extraction includes adaptive keyframe extraction from video data, speech translation and structured information extraction, content recognition of image data, and parsing of text materials.

[0151] Specifically, the processing of audio information in video data can be achieved by using a speech recognition model to accurately translate the audio stream from the video into text. The translated text is then input into a large language model, which, through designed prompt word templates, guides the model to extract structured information, such as the business address information of the target customer.

[0152] The image data processing can be done by inputting the image data into a multimodal large-scale visual encoder to automatically identify multiple key fields in the image, or by directly performing image recognition algorithms to obtain the corresponding data.

[0153] The process of parsing text data involves performing natural language processing on the text material to extract key fields and form a structured information set on the text side.

[0154] S503, multimodal cross-validation and fusion.

[0155] In this process, the structured information extracted from video, image, and text in step S502 is uniformly vectorized and represented, and cross-modal semantic alignment and fusion reasoning are performed using a multimodal large model.

[0156] S504. Combine the consistency score of the reasoning output to generate a quality inspection report.

[0157] A quality inspection report can refer to a document that evaluates the relevant business qualifications of a target customer.

[0158] In some embodiments, the electronic device acquires the consistency score (e.g., a value from 0 to 100) of the multimodal large model output in S503 and the inconsistency indications (e.g., address inconsistency, etc.) of each verification item. The electronic device compares the consistency score with a preset scoring threshold: if the score is greater than or equal to the high scoring threshold (e.g., 90 points), the review conclusion is determined to be passed; if the score is between the low scoring threshold and the high scoring threshold (e.g., 60 to 89 points), the review conclusion is determined to be that manual review is required, and the inconsistency indication is attached to the report as a review point; if the score is less than or equal to the low scoring threshold (e.g., below 60 points), the review conclusion is determined to be rejected.

[0159] Then, the electronic device fills the template with the audit conclusions, consistency scores, inconsistencies in each verification item, and corresponding original material indexes (e.g., keyframe numbers, text paragraph positions) according to the preset report template, generating a quality inspection report in a structured or visual format. This report can be used directly for business decisions or archiving.

[0160] The following is combined with Figure 6 The working process of this plan will be explained in detail.

[0161] Figure 6 This is a flowchart illustrating another information processing method provided in an embodiment of this application. Please refer to... Figure 6 ,

[0162] The process begins by acquiring multi-source business data from the target customer, including video, image, and text data. For video data, keyframes are extracted to obtain multiple keyframe images, and effective content recognition is performed on these images. Simultaneously, the audio in the video is transcribed into text, and key fields are extracted. For image data, effective content recognition is performed directly, and field information is extracted. For text data, key fields are extracted, and the core content of the customer's completed business application forms and declarations is obtained. Then, all the effective information and key fields extracted from video keyframes, speech transcription, image recognition, and text parsing are input into a multimodal large-scale model for cross-modal consistency comparison and fusion inference, completing multimodal cross-validation. Finally, a quality inspection report is generated based on the cross-validation results, and the process ends.

[0163] In this embodiment, an adaptive sampling mechanism is used to intelligently extract keyframes from video. Other recognition models are used collaboratively to complete speech recognition and text translation. A large language model extracts structured information, and a multimodal large model is used to automatically cross-validate and verify the consistency of due diligence data (video, images, and text). This effectively solves the problems of low efficiency, high error rate, and difficulty in processing unstructured data in existing technologies. Specifically, this solution extracts video and audio content and combines it with a large language model to extract structured information, automatically verifying the consistency between video statements and customer declarations, effectively ensuring the authenticity and completeness of information. Simultaneously, a multimodal large model accurately identifies visual elements such as company information in images and automatically cross-validates them with address data submitted by the customer, significantly reducing the error rate of manual verification. This upgrades the traditional manual review mode to intelligent quality inspection. Relying on the collaborative analysis of multimodal data, it promotes the transformation of compliance review towards intelligence and automation, comprehensively improving the accuracy and timeliness of the review. In the above method, electronic devices automatically process videos, images, and texts, cross-compare key information in the content, and generate quality inspection reports, which significantly improves the efficiency of the review process; eliminates the subjectivity and fatigue of manual review, reduces the overall error rate of document verification, and unifies the review standards; at the same time, it automatically identifies various compliance and risks such as missing evidence chains and contradictory information, realizes pre-warning and in-process interception, and strengthens the institution's compliance bottom line.

[0164] Figure 7 This is a schematic diagram of the structure of an information processing device provided in an embodiment of this application. Please refer to... Figure 7 The information processing device 70 includes: an acquisition module 71, a processing module 72, a fusion module 73, and a generation module 74, wherein...

[0165] The acquisition module 71 is used to acquire multi-source business information of the target customer. The multi-source business information is business data collected based on different acquisition strategies, including video data, image data and text data.

[0166] The processing module 72 is used to perform frame extraction processing on the video data to obtain multiple first images, and to perform conversion processing on the audio information in the video data to obtain corresponding text information. The first parameter of the first image is greater than a first threshold. The first parameter is used to indicate the degree of influence of the frame in which the first image is located on the video data.

[0167] The fusion module 73 is used to perform fusion processing on multiple first images, text information, image data and text data according to the first model to obtain fused information;

[0168] The generation module 74 is used to generate a quality inspection report for the target customer based on the fused information. The quality inspection report is used to indicate the degree of consistency between video data, image data and text data.

[0169] The information processing apparatus provided in this application embodiment can execute the method shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0170] In one possible implementation, the processing module 72 is specifically used for:

[0171] Obtain the total number of frames and video duration from the video data;

[0172] Based on the preset frame information, total number of frames, and video duration, determine the target number of frames to be extracted, and determine the sampling interval based on the target number of frames to be extracted;

[0173] Based on the sampling interval, the video data is sampled frame by frame to obtain multiple sampled frames;

[0174] Multiple first images are determined based on multiple sampled frames.

[0175] In one possible implementation, the processing module 72 is specifically used for:

[0176] Multiple sampling frames are processed to obtain grayscale images corresponding to each sampling frame;

[0177] For any given sampling frame, determine the grayscale difference between the current sampling frame and the previous sampling frame, and determine the first parameter based on the grayscale difference. The first parameter is used to indicate the degree of difference between adjacent sampling frames.

[0178] Among multiple sampling frames, the sampling frames whose first parameter is greater than the first threshold are determined as candidate sampling frames;

[0179] Multiple first images are determined based on multiple candidate sampling frames.

[0180] In one possible implementation, the fusion module 73 is specifically used for:

[0181] Multiple first images, text information, image data, and text data are input into the first model to obtain multimodal feature information that integrates video data, image data, and text data;

[0182] Cross-validation is performed on multiple feature information to obtain the validation result, which is used to verify the matching degree between video data, image data and text data.

[0183] The feature information and the verification results are identified as the fused information.

[0184] In one possible implementation, the fusion module 73 is specifically used for:

[0185] The first model performs visual encoding on multiple first images and image data to obtain the first feature information corresponding to each image, and performs semantic encoding on text information and text data to obtain the second feature information corresponding to each text information and the third feature information corresponding to each text data.

[0186] Feature alignment processing is performed on multiple first feature information, multiple second feature information, and multiple third feature information to obtain a multimodal feature sequence;

[0187] The multimodal feature sequences are concatenated, and cross-modal semantic fusion is performed on the concatenated multimodal feature sequences to obtain multiple feature information.

[0188] In one possible implementation, the fusion module 73 is specifically used for:

[0189] The similarity between the feature information corresponding to the audio information in the video data and the feature information corresponding to the text data is calculated to obtain the first verification result.

[0190] The feature information corresponding to multiple first images and image data is matched with the feature information corresponding to text data to obtain a second verification result;

[0191] The feature information corresponding to multiple first images and image data, the feature information corresponding to audio information in video data, and the feature information corresponding to text data are matched to obtain the third verification result;

[0192] The first verification result, the second verification result, and the third verification result are determined as the verification results.

[0193] In one possible implementation, the generation module 74 is further configured to:

[0194] Based on a pre-set knowledge base, the fused information is verified to obtain the verification result. The pre-set knowledge base is used to store the business registration information of multiple customers.

[0195] When the verification result indicates that the verification failed, and / or the check result indicates that the check failed, identify the abnormal information in the multi-source business information, mark the abnormal information as an abnormal state, and generate a quality inspection report;

[0196] When both the verification result and the validation result indicate that the validation has passed, the abnormal information is marked as normal and a quality inspection report is generated.

[0197] The information processing apparatus provided in this application embodiment can execute the method shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0198] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device 80 may include: a transceiver 81, a processor 82, and a memory 83.

[0199] Processor 82 executes computer execution instructions stored in memory, causing processor 82 to perform the scheme in the above embodiments. Processor 82 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0200] The memory 83 is connected to the processor 82 via the system bus and completes communication between them. The memory 83 is used to store computer program instructions.

[0201] Transceiver 81 can be used to obtain the task to be run and its configuration information.

[0202] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0203] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0204] This application also provides a chip for executing instructions, which is used to execute the information processing method described in the above embodiments.

[0205] This application also provides a computer-readable storage medium storing computer instructions, which, when executed on a computer, cause the computer to perform the technical solution of the information processing method described in the above embodiments.

[0206] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the technical solution of the information processing method in the above embodiments.

[0207] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0208] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0209] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0210] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0211] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0212] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0213] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0214] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0215] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An information processing method, characterized in that, include: Obtain multi-source business registration information of target customers. The multi-source business registration information is business data collected based on different collection strategies, including video data, image data, and text data. The video data is subjected to frame extraction processing to obtain multiple first images, and the audio information in the video data is converted to obtain corresponding text information. The first parameter of the first image is greater than a first threshold. The first parameter is used to indicate the degree of influence of the frame in which the first image is located on the video data. According to the first model, the plurality of first images, the text information, the image data, and the text data are fused to obtain fused information; Based on the fused information, a quality inspection report corresponding to the target customer is generated. The quality inspection report is used to indicate the degree of consistency between the video data, the image data, and the text data.

2. The method according to claim 1, characterized in that, The video data is subjected to frame extraction processing to obtain multiple first images, including: Obtain the total number of frames and video duration of the video data; Based on the preset frame information, the total number of frames, and the video duration, the target number of frames to be extracted is determined, and the sampling interval is determined based on the target number of frames to be extracted. According to the sampling interval, the video data is sampled frame by frame to obtain multiple sampled frames; Based on the multiple sampling frames, multiple first images are determined.

3. The method according to claim 2, characterized in that, Based on the multiple sampling frames, multiple first images are determined, including: The multiple sampling frames are processed to obtain grayscale images corresponding to each sampling frame; For any given sampling frame, determine the grayscale difference between the current sampling frame and the previous sampling frame, and determine a first parameter based on the grayscale difference. The first parameter is used to indicate the degree of difference between adjacent sampling frames. Among the plurality of sampling frames, the sampling frames in which the first parameter is greater than the first threshold are determined as candidate sampling frames; Multiple first images are determined based on multiple candidate sampling frames.

4. The method according to claim 1, characterized in that, According to the first model, the plurality of first images, the text information, the image data, and the text data are fused to obtain fused information, including: The multiple first images, the text information, the image data, and the text data are input into the first model to obtain multimodal feature information that fuses the video data, image data, and text data; Cross-validation is performed on multiple feature information to obtain a validation result, which is used to verify the matching degree between the video data, the image data, and the text data. The feature information and the verification result are determined as fused information.

5. The method according to claim 4, characterized in that, The plurality of first images, the text information, the image data, and the text data are input into the first model to obtain multimodal feature information that fuses the video data, image data, and text data, including: The first model is used to perform visual encoding processing on the plurality of first images and the image data to obtain the first feature information corresponding to each image, and semantic encoding processing is performed on the text information and the text data to obtain the second feature information corresponding to each text information and the third feature information corresponding to each text data. Feature alignment processing is performed on multiple first feature information, multiple second feature information, and multiple third feature information to obtain a multimodal feature sequence; The multimodal feature sequences are concatenated, and cross-modal semantic fusion is performed on the concatenated multimodal feature sequences to obtain multiple feature information.

6. The method according to claim 5, characterized in that, Cross-validation is performed on multiple feature information to obtain validation results, including: The similarity between the feature information corresponding to the audio information in the video data and the feature information corresponding to the text data is calculated to obtain the first verification result. The feature information corresponding to the plurality of first images and the image data is matched with the feature information corresponding to the text data to obtain a second verification result; The feature information corresponding to the plurality of first images and the image data, the feature information corresponding to the audio information in the video data, and the feature information corresponding to the text data are matched to obtain a third verification result; The first verification result, the second verification result, and the third verification result are determined as the verification result.

7. The method according to claim 6, characterized in that, Based on the fused information, a quality inspection report corresponding to the target customer is generated, including: The fused information is verified based on a preset knowledge base to obtain a verification result. The preset knowledge base is used to store business registration information of multiple customers. When the verification result indicates that the verification failed, and / or when the inspection result indicates that the inspection failed, the abnormal information existing in the multi-source business information is identified, the abnormal information is marked as an abnormal state and a quality inspection report is generated; When the verification result indicates that the verification has passed, the abnormal information is marked as normal and a quality inspection report is generated.

8. An information processing device, characterized in that, include: The module consists of an acquisition module, a processing module, a fusion module, and a generation module. The acquisition module is used to acquire multi-source business information of the target customer. The multi-source business information is business data collected based on different acquisition strategies, and includes video data, image data and text data. The processing module is used to perform frame extraction processing on the video data to obtain multiple first images, and to perform conversion processing on the audio information in the video data to obtain corresponding text information. The first parameter of the first image is greater than a first threshold, and the first parameter is used to indicate the degree of influence of the frame in which the first image is located on the video data. The fusion module is used to perform fusion processing on the plurality of first images, the text information, the image data and the text data according to the first model to obtain fused information; The generation module is used to generate a quality inspection report corresponding to the target customer based on the fusion information. The quality inspection report is used to indicate the degree of consistency between the video data, the image data, and the text data.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images