Merchant inspection information processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a merchant inspection information processing method and device, a storage medium and electronic equipment, and relates to the field of financial science and technology, and the method comprises the steps: obtaining merchant inspection information which at least comprises an external environment image, an internal environment image and internal voice information of a merchant; predicting the type of the merchant according to the feature information in the external environment image to obtain a first type; predicting the type of the merchant according to the in-store environment image and the in-store voice information to obtain a second type; under the condition that the predicted first category is the same as the predicted second category, determining that the target category of the merchant is the first category; under the condition that the predicted first category is different from the predicted second category, determining a target category of the merchant from the first category and the second category according to the transaction flow information of the merchant; and submitting the determined target category of the merchant to an auditing system. The technical problem of low auditing efficiency caused by manual auditing of merchant inspection information in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of financial technology, and more specifically, to a method, apparatus, storage medium, and electronic device for processing merchant inspection information. Background Technology

[0002] With the widespread adoption of payment services, banks and other financial institutions need to conduct regular inspections and onboarding reviews of their partner merchants to identify potential risks such as cash-out schemes and fraud. Currently, this process relies primarily on manual labor in the relevant technologies.

[0003] For example, auditors need to review all materials submitted by merchants, including but not limited to business licenses, identification documents, photos of the store's interior and exterior environment, and transaction records. Auditors must rely on their personal experience to manually compare and judge the authenticity, consistency, and compliance of the materials. With the growth of acquiring business, the number of merchants awaiting review has surged, making the existing highly manual processing model inadequate and inefficient. On the one hand, auditors need to process massive amounts of heterogeneous data, including long text transaction records and multiple images; manual processing speed cannot keep up with the demands of business growth, resulting in a backlog. On the other hand, manual review is limited by individual attention and experience differences, easily leading to omissions and misjudgments in the screening process, making it difficult to guarantee the accuracy and consistency of the review results.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method, apparatus, storage medium, and electronic device for processing merchant inspection information, so as to at least solve the technical problem of low review efficiency caused by manual review of merchant inspection information in the prior art.

[0006] According to one aspect of the embodiments of this application, a method for processing merchant inspection information is provided, comprising: acquiring merchant inspection information, wherein the merchant inspection information includes at least images of the merchant's external environment, images of the merchant's internal environment, and voice information of the merchant's internal environment; predicting the merchant's category based on feature information in the external environment images to obtain a first category; predicting the merchant's category based on the images of the merchant's internal environment and the voice information of the merchant's internal environment to obtain a second category; determining the merchant's target category as the first category when the predicted first category and the second category are the same; determining the merchant's target category from the first category and the second category based on the merchant's transaction flow information when the predicted first category and the second category are different; and submitting the determined target category of the merchant to an approval system.

[0007] Optionally, the method further includes: performing a first type of image preprocessing on the external environment image to obtain a target external environment image, and then predicting the merchant's category based on the feature information extracted from the target external environment image to obtain a first category; performing a second type of image preprocessing on the internal environment image to obtain a target internal environment image, and then predicting the merchant's category based on the feature information extracted from the target internal environment image and internal voice information to obtain a second category, wherein the first type of image preprocessing and the second type of image preprocessing are different processing operations.

[0008] Optionally, the first type of image preprocessing includes at least one of the following operations: a first operation for expanding the dynamic range of an outdoor environment image by merging multiple outdoor environment images with different exposures, wherein the dynamic range is the ratio between the brightest and darkest pixel values ​​of the outdoor environment image; a second operation for adjusting the contrast of at least one region of the outdoor environment image based on the exposure information of each region in the outdoor environment image; a third operation for eliminating halo and specular reflection information in the outdoor environment image; and a fourth operation for adjusting the color saturation of the outdoor environment image based on the weather conditions at the time the outdoor environment image was captured.

[0009] Optionally, the second type of image preprocessing includes at least one of the following operations: color correction operation, used to calibrate image colors by adjusting white balance; background simplification operation, used to determine the outline of the target object and the background area of ​​the target object by semantic segmentation, and then blur or delete elements in the background area; noise reduction operation, used to reduce image noise in the store environment image when the brightness of the store environment image is detected to be lower than a preset threshold.

[0010] Optionally, the merchant's category is predicted based on the feature information extracted from the target store's external environment image to obtain a first category, including: extracting textual and non-textual features from the target store's external environment image; determining a first probability distribution of the merchant belonging to N categories based on the textual features in the target store's external environment image using a short text prediction model or a long text prediction model, wherein the input feature length of the short text prediction model is shorter than the input feature length of the long text prediction model, and N is an integer greater than 1; determining a second probability distribution of the merchant belonging to N categories based on the non-textual features in the target store's external environment image using an image prediction model; performing a weighted calculation on the first and second probability distributions, and then determining the category with the highest weighted sum of probabilities as the first category based on the weighted calculation result.

[0011] Optionally, based on the feature information extracted from the target store environment image and the in-store voice information, the merchant category is predicted to obtain the second category, including: extracting textual and non-textual features from the target store environment image; extracting semantic features from the in-store voice information; determining the third probability distribution of the merchant belonging to each of the N categories based on the textual features in the target store environment image using a short text prediction model or a long text prediction model; determining the fourth probability distribution of the merchant belonging to each of the N categories based on the non-textual features in the target store environment image using an image prediction model; determining the fifth probability distribution of the merchant belonging to each of the N categories based on the semantic features using a semantic prediction model; weighting the third, fourth, and fifth probability distributions, and then determining the category with the highest weighted sum of probabilities as the second category based on the weighted calculation results.

[0012] Optionally, the method further includes: when the feature length of the detected text feature is less than a preset length, selecting a short text prediction model to process the text feature; when the feature length of the detected text feature is greater than or equal to a preset length, selecting a long text prediction model to process the text feature.

[0013] Optionally, if the first category and the second category are different, the merchant's target category is determined from the first category and the second category based on the merchant's transaction flow information, including: extracting transaction behavior features from the transaction flow information if the first category and the second category are different, wherein the transaction behavior features include at least: transaction time distribution information, transaction amount information and transaction frequency information; and determining the merchant's target category from the first category and the second category based on the transaction behavior features.

[0014] Optionally, determining the merchant's target category from the first and second categories based on transaction behavior characteristics includes: generating initial prompt words based on the first and second categories, wherein the initial prompt words are used to construct the judgment and selection logic regarding the first and second categories; updating the initial prompt words to target prompt words based on transaction behavior characteristics, wherein the transaction behavior characteristics serve as the judgment basis for the judgment and selection logic in the target prompt words; and inputting the target prompt words into a large language model, determining the merchant's target category from the first and second categories based on the prior knowledge of the large language model during the model training phase.

[0015] According to another aspect of the embodiments of this application, a merchant inspection information processing apparatus is also provided, comprising: an acquisition unit for acquiring merchant inspection information, wherein the merchant inspection information includes at least an external environment image, an internal environment image, and internal voice information of the merchant; a first prediction unit for predicting the merchant's category based on feature information in the external environment image to obtain a first category; a second prediction unit for predicting the merchant's category based on the internal environment image and internal voice information to obtain a second category; a first determination unit for determining the merchant's target category as the first category when the predicted first category and the second category are the same; a second determination unit for determining the merchant's target category from the first category and the second category based on the merchant's transaction flow information when the predicted first category and the second category are different; and an information submission unit for submitting the determined merchant's target category to an auditing system.

[0016] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described method for processing merchant inspection information.

[0017] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described method for processing merchant inspection information.

[0018] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the above-described method for processing merchant inspection information.

[0019] In this embodiment, the merchant inspection processing system first acquires merchant inspection information, which includes at least images of the merchant's external environment, images of the merchant's internal environment, and voice information from within the store. Then, based on feature information from the external environment images, the system predicts the merchant's category to obtain a first category; based on the internal environment images and voice information, it predicts the merchant's category to obtain a second category. If the predicted first and second categories are the same, the merchant's target category is determined as the first category; if the predicted first and second categories are different, the merchant's target category is determined from the first and second categories based on the merchant's transaction history information, and the determined target category is submitted to the review system.

[0020] As described above, the merchant inspection processing system automatically acquires merchant inspection information and predicts the first and second categories based on this information. This enables the automatic processing and preliminary classification of massive and heterogeneous merchant inspection information, reducing the workload of auditors in the arduous and repetitive tasks of information review and initial category screening. Furthermore, when the first and second categories are the same, the system can quickly determine the merchant's target category, forming an efficient review channel. When the first and second categories differ, the system does not simply rely on manual intervention but automatically incorporates objective transaction flow information to determine the merchant's target category. This simulates and surpasses the process of human judgment based on limited experience, allowing for deeper automated analysis when information contradictions are discovered. This reduces the number of cases requiring manual review and directs human effort to truly complex and high-risk situations. Finally, by automatically submitting the determined merchant target category to the review system, a smooth connection can be achieved between the intelligent processing results and subsequent manual review stages. Auditors can directly obtain the system's predicted target category as a key reference, eliminating the need to analyze original materials from scratch and improving the efficiency of their final review. In this way, while ensuring or even improving the accuracy of the review, the degree and burden of human involvement can be significantly reduced, thereby solving the technical problem of low review efficiency caused by manual review of merchant inspection information in existing technologies. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 This is a schematic diagram of an optional method for processing merchant inspection information according to an embodiment of this application;

[0023] Figure 2 This is an optional overall processing flowchart of merchant inspection information according to an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of a multi-level review mechanism in an optional merchant inspection and review process according to an embodiment of this application;

[0025] Figure 4 This is a front-end flowchart of an optional merchant submitting qualifications and inspection materials in the merchant inspection and processing system according to an embodiment of this application;

[0026] Figure 5 This is an optional flowchart illustrating intelligent analysis of merchant inspection based on a multimodal large model, according to an embodiment of this application.

[0027] Figure 6 This is a schematic diagram of the internal hardware structure of an optional merchant inspection and processing system according to an embodiment of this application;

[0028] Figure 7 This is a schematic diagram of an optional merchant inspection information processing device according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] According to an embodiment of this application, a method embodiment for processing merchant inspection information is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] It should be noted that the information collected in this application (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of this data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse. For example, interfaces are set up between this system and relevant users or organizations, providing users with corresponding access points to choose to agree to or refuse automated decision-making results; if the user chooses to refuse, the process proceeds to the expert decision-making stage.

[0033] According to the embodiments of this application, a merchant inspection processing system can be used as the execution subject of the merchant inspection information processing method of this application embodiment. The system can be a software system or an embedded system combining software and hardware. Of course, the method execution subject in the embodiments of this application can also be other forms of execution subject, such as devices, equipment, etc. It should be known by those skilled in the art that this application does not particularly limit the specific form of the method execution subject.

[0034] Figure 1 This is a method for processing merchant inspection information according to embodiments of this application, such as... Figure 1 As shown, the method includes the following steps:

[0035] Step S101: Obtain merchant inspection information, which includes at least images of the merchant's external environment, images of the merchant's internal environment, and voice information from inside the store.

[0036] Optionally, merchant inspection information can refer to a multimodal data set that is actively submitted by the merchant or passively collected by the system during the merchant inspection and review process, and is used to characterize the merchant's physical business status and environmental characteristics.

[0037] Optionally, exterior environment images can refer to data characterizing the external visual features of a business's premises, and may include digital photographs or video frames of storefront signs, building exteriors, and streetscapes. Exterior environment images are used to provide visual evidence for determining the business's industry and scale.

[0038] Optionally, in-store environment images can refer to data that characterizes the internal layout and operational status of a merchant's business premises. For example, in-store environment images can include digital photos of shelf displays, equipment installations, and interior decorations. In-store environment images can verify and supplement external image information from an internal perspective, forming a spatial consistency check.

[0039] Optionally, in-store voice information can refer to data representing the background acoustic environment within the merchant's business premises. It can be an audio file collected in the store that includes ambient sounds, conversations, or background music. In-store voice information can provide non-visual auxiliary judgment criteria.

[0040] Optionally, methods for obtaining merchant inspection information may include using a data collection module deployed on the merchant's smart terminal to guide the merchant to take and upload images of the store's interior and exterior environment according to a preset process, while simultaneously recording and uploading audio of the store's environment. Alternatively, methods may include: setting up a standardized data receiving interface in the system backend to receive upload requests from the terminal, performing format verification, virus scanning, and temporary storage on the uploaded multimedia files, and parsing out structured inspection information metadata.

[0041] By acquiring merchant inspection information, the system can build a more comprehensive and interference-resistant initial profile of the merchant than a single data source, which helps to overcome the risk of misjudgment caused by falsified or missing information from a single channel.

[0042] Step S102: Predict the merchant's category based on the feature information in the external environment image to obtain the first category.

[0043] Optionally, feature information can refer to numerical vectors or structured descriptions extracted from the original outdoor environment image data that can effectively characterize the essential attributes of the image content. For example, deep semantic features extracted through convolutional neural networks, or texture, color, and shape features extracted through image processing algorithms.

[0044] Alternatively, predicting a merchant's category can refer to a classification task. For example, predicting a merchant's category can be a process of using a pre-trained machine learning or deep learning model to map input image features to a predefined set of category labels.

[0045] Optionally, the first category can refer to the merchant classification label with the highest probability value output by the prediction model based on a single data source—images of the store's external environment. This is the first preliminary judgment conclusion formed by the system. The prediction model for the merchant category can be, but is not limited to, a deep neural network. The output layer of the deep neural network corresponds to the probability distribution of N merchant categories, and the category label corresponding to the maximum value in the probability distribution is taken as the first category output.

[0046] By automating and initially automating the traditional process of judging the type of merchant by observing storefront signs based on human experience, we can quickly and in batches screen and classify the external image of a large number of merchants, generating a preliminary review reference based on images of the store's external environment. This can provide a preliminary, quantifiable basis for judgment in subsequent review processes and reduce the burden of manual initial review.

[0047] Step S103: Based on the in-store environment images and in-store voice information, predict the merchant's category to obtain the second category.

[0048] Optionally, the second category can refer to the merchant classification label with the highest probability value output by the predictive model, based on the fusion analysis of two data sources: in-store environmental images and in-store voice information. The second category is a second preliminary judgment conclusion formed by the system, independent of the first.

[0049] By introducing a second category, cross-validation can be achieved between the second and first categories to detect consistency between in-store and out-of-store information. When the predictions of the second and first categories are consistent, the confidence level of the system's judgment is enhanced; when the predictions of the second and first categories are inconsistent, potential risks can be effectively exposed, such as discrepancies between the storefront and internal operations, or false packaging, thereby triggering a more in-depth review process and improving the system's accuracy in identifying complex fraud scenarios.

[0050] Step S104: If the predicted first category and second category are the same, determine the merchant's target category as the first category.

[0051] Optionally, the target category can refer to a unique merchant classification identifier that is finally confirmed and prepared to be submitted to the business review system after the system's multi-path prediction and conflict arbitration.

[0052] Optionally, the system compares the first category with the second category. This comparison can be achieved through exact string matching or category code equivalence checks. The system determines that the first and second categories corroborate each other and the first category is acceptable only if the first and second categories are completely identical. In this case, the system does not trigger other complex arbitration processes such as transaction log information; instead, it directly assigns the agreed-upon category (the first category) as the merchant's final target category, improving processing efficiency and saving computational resources. The system can simulate the reasonable logic of rapid acceptance based on evidence consistency, providing a fast review channel for a large number of merchants with accurate information and compliant operations, reducing unnecessary redundant calculations. In the scenario of processing massive amounts of data during merchant inspections, the system can effectively distribute resources, allowing system resources to focus on handling complex steps with inconsistent predictions, potential risks, or fraud, thus optimizing the overall system response speed and resource allocation efficiency.

[0053] Step S105: If the predicted first category and second category are different, determine the merchant's target category from the first category and the second category based on the merchant's transaction flow information.

[0054] Optionally, transaction log information can refer to the collection of all transaction records generated by a merchant through the payment system within a certain period of time. Each transaction record may include structured fields such as timestamp, transaction amount, and transaction type.

[0055] Optionally, if the first and second categories differ, it indicates that a reliable judgment cannot be made solely based on the surface environmental information submitted by the merchant. In this case, the system retrieves the merchant's transaction history information to extract key behavioral characteristics, such as transaction time distribution patterns, typical transaction amount ranges, and transaction frequency. These behavioral characteristics are strongly correlated with the typical operating models of merchants in different industries; for example, transactions in the catering industry are concentrated during lunch and dinner hours, while retail transactions are more dispersed. Using objective behavioral characteristics as a basis, the system can analyze them through preset rule models or prompt-based large language models to determine which candidate category best matches the current transaction behavior pattern, thereby determining the merchant's final target category. This enhances the system's ability to identify complex risks and combat fraud, deepening the review logic from comparing static, easily forged images and voice data to analyzing dynamic, continuous behavioral data, thus improving the accuracy of overall merchant inspections and the depth of risk control.

[0056] Step S106: Submit the identified target categories of merchants to the review system.

[0057] Optionally, the auditing system can refer to a back-end management system used internally by a bank or payment institution to handle business processes such as merchant onboarding, inspection, and risk monitoring. The auditing system can be a downstream business platform that receives instructions, stores results, and drives workflows. Submission to the auditing system can include real-time synchronous submission via application programming interface (API) calls and asynchronous reliable delivery via message middleware.

[0058] Optionally, the system can bind the final determined target category with the merchant's unique identifier and encapsulate it into a structured data message. By calling the standardized data interface pre-provided by the auditing system, the message is proactively pushed to the auditing system's backend service using network communication protocols. After receiving and parsing the data message, the auditing system can write the target category as a key field into the corresponding merchant's audit database record. In this way, the system can achieve a closed-loop business process from automated intelligent processing to manual review and decision-making. This allows downstream auditors to directly access and use the results of complex multimodal analysis and intelligent judgment as a reference, reducing information silos and the workload of auditors in the arduous initial information screening and classification work. Auditors can then focus on higher-risk value judgments, thereby improving the overall collaborative efficiency of merchant inspection operations.

[0059] It should be noted that merchant risk assessments can include cash-out risk, fraud risk, and arbitrage risk. Cash-out risk mainly manifests as collusion between merchants and cardholders or other third parties, or merchants themselves using bank cards or other payment tools to obtain funds through virtual transactions, inflated prices, cash refunds, etc. Fraud risk includes merchants' illegal operations, intentional fraud, and acts of protecting or assisting criminals in using their business premises and facilities for fraudulent transactions. Arbitrage risk refers to merchants failing to genuinely utilize preferential resources for bank customers, instead colluding with cardholders to conduct fictitious transactions, using bank-invested preferential or rights resources to arbitrage and obtain cash gains, points rewards, withdrawal fees, etc.

[0060] Figure 3 A schematic diagram of a multi-level review mechanism in a merchant inspection and review process is shown. Optionally, an interactive process between the merchant side and the review side applying the above-mentioned merchant inspection information processing method may include: the merchant side process may first involve terminal registration and login. Subsequently, under the system guidance, the merchant submits qualification documents and uploads inspection materials, which at least include required images of the external environment, images of the internal environment, and in-store audio information. This merchant inspection information can serve as input for the system's intelligent processing. After uploading, the merchant can view the review status and results online. The review side process can be conducted within the review system. Reviewers can view the generated merchant inspection report through the review system interface. The generation of this merchant inspection report depends on the merchant inspection information processing method: the system first acquires the merchant inspection information, then predicts the first and second categories based on this information, and, if necessary, combines transaction flow information for judgment, ultimately determining the merchant's target category and generating a report along with other analysis results. The reviewer conducts a preliminary review based on this report. The preliminary review decision and the report can then be submitted to the merchant inspection and audit supervisor and other management personnel for a second review, forming a double review mechanism. This reduces the workload of traditional door-to-door material collection through online and intelligent methods.

[0061] Figure 4This document illustrates a front-end flowchart of a merchant submitting qualifications and inspection materials in a merchant inspection processing system. Merchants first need to select their entity type and fill in the text information and upload images for qualification information. The text information can include the company name, registered address, legal representative's name, and legal representative's ID number. Images can include the front and back of the ID card, business license, and payment QR code. The system then calls a map interface to obtain the merchant's current location and compares it with the registered address. If the distance between the merchant's current location and the registered address is less than or equal to 1 kilometer, the location verification is considered successful. After successful verification, the merchant can continue to upload inspection materials, including three types of images: exterior photos, interior photos, and equipment photos. All submitted images undergo image-aware hash deduplication to prevent duplicate or forged materials. Finally, the approved data is encrypted and stored in the database, completing the entire submission process. This front-end flowchart for submitting qualifications and inspection materials demonstrates the automation and security design of merchant inspection information collection and preliminary verification. The structured collection of text and image information provides a data foundation for subsequent prediction of merchant categories using short or long text prediction models. Location verification via map API and image deduplication mechanism based on perceptual hashing algorithms jointly ensure the authenticity and uniqueness of merchant inspection information, improving the reliability of data acquisition.

[0062] Optionally, merchants can interact with the system through browsers or applications on mobile phones or computers to submit information, thus providing a data source for the backend system to process merchant inspection information. This information submission process is the front-end interaction for obtaining merchant inspection information. First, merchants can complete dual authentication login by entering their account, password, and SMS verification code on the login interface. After logging in, merchants enter the qualification verification interface, where they fill in and upload the corresponding legal identity and business qualification documents according to their type, such as images of the legal representative's ID card and business license. After completing the qualification verification, merchants enter the inspection data upload interface. On this interface, the merchant terminal performs key operations: 1) calling the location service to obtain current location information; 2) taking photos through the terminal camera or selecting and uploading on-site photos from the album. These on-site photos must include at least the required images of the exterior and interior environments of the store. Simultaneously, the system can guide merchants to record and upload a voice message inside the store. The merchant terminal packages the above location information, images, and voice files and sends them to the server via the network. Subsequently, merchants can check the processing status on the review and query interface. If the review is rejected, merchants can re-enter the inspection data upload interface based on the feedback analysis and suggestions to perform a new round of operations, such as uploading on-site photos. This example illustrates how merchants can submit image data, voice data, and location-related auxiliary data that constitute merchant inspection information through a standardized front-end process, thereby improving the standardization of input data.

[0063] Optionally, after receiving the merchant's target category and related analysis report processed by the merchant inspection method, the review system can support manual review decisions according to the following process: After logging in, the reviewer can view the merchant list in the pending review list, which integrates the merchant's target category, location information, etc. After clicking "Review," the review details page can centrally display the complete inspection report generated by the large language model. The analysis in this inspection report includes the merchant's target category. Based on this report, the reviewer makes a preliminary review decision to approve or reject the application. After the preliminary review is passed, the process enters the final review stage of the merchant inspection review supervisor. The supervisor reviews the application and makes a final decision, completing the review loop. In addition, based on the risk score in the inspection report, the system automatically sets differentiated expiration dates for the next inspection for merchants with different risks. For example, a shorter expiration date is set for high-risk merchants, and a longer expiration date is set for low-risk, high-quality merchants, thereby achieving dynamic and differentiated risk management.

[0064] Optionally, before the initial review by junior auditors, target prompts can be used to call a large language model to generate a smart report on merchant inspections. The inspection results for different merchants should be scored out of 100 points to determine whether the merchants have passed the inspection access criteria, and the corresponding merchant inspection report should be printed out.

[0065] The prompt text can begin with "You are a senior bank merchant inspection specialist. Please rigorously review the following merchant information and generate a merchant inspection report":

[0066] A. Basic Merchant Information.

[0067] B. Audit requirements include:

[0068] All images must first pass the following two basic checks:

[0069] 1. Image relevance: Does the image content roughly match the type name?

[0070] 2. Image clarity: Whether the image is clear and recognizable.

[0071] If any of the above conditions are not met, the corresponding review item for the image will be deemed unacceptable. The reason should be either "irrelevant to the image" or "image is unclear". If the image is approved, the reason must also be stated.

[0072] C. Detailed review steps include:

[0073] 1. Legal entity identity verification:

[0074] Regarding the consistency between name and ID number:

[0075] Use a photo of the front of your ID card;

[0076] Check whether the name of the legal representative is completely consistent with the name in the photo on the front of the ID card;

[0077] Check whether the legal representative's ID number is completely consistent with the citizen's ID number shown in the front image of the ID card.

[0078] ID card validity period:

[0079] Use a photo of the back of your ID card;

[0080] Check whether the expiration date is within the expiration period.

[0081] 2. Merchant information verification:

[0082] Name consistency:

[0083] Use images with the merchant's name;

[0084] Check if the merchant name matches the name set in the image.

[0085] Address consistency:

[0086] Use images with the merchant's address;

[0087] Check whether the merchant's address matches the address in the image containing the merchant's address.

[0088] Consistency of legal entities:

[0089] Use images containing legal entity information;

[0090] Check whether the name of the legal representative matches the name of the legal representative in the picture.

[0091] Validity period:

[0092] Use the image on the business license;

[0093] Check if the business term is within the validity period (the current date must be within the validity period of the business term);

[0094] 2. Review of other materials:

[0095] Receipt:

[0096] Check if it bears the logo of "XX Bank";

[0097] Storefront sign:

[0098] Check whether the name on the storefront matches the name in the merchant's basic information.

[0099] Store interior conditions:

[0100] Check whether the store environment is suitable for the type of business;

[0101] Determine the merchant type: Based on a comprehensive consideration of the store's exterior signage, interior conditions, and business scope on the business license, determine which type the merchant belongs to.

[0102] Equipment status:

[0103] Check that the device has a POS machine installed and is not damaged.

[0104] After completing the above steps, the merchant's bank statement data must be strictly reviewed according to the following requirements:

[0105] D. Audit requirements include:

[0106] "Supermarket-specific bank statement notification text":

[0107] Based on the 0th and 90th percentiles of all transaction amounts, the range in which the amount occurs most frequently is calculated. The reference value for large transactions is set to 5 times the upper limit of the range. Transactions exceeding the reference value for large transactions are considered to be large abnormal transactions. Reasons must be given for both approval and disapproval. Please strictly follow the following four rules to give the review conclusion.

[0108] First point: Are 90% of the transaction amounts between 0 and 1000? If so, then proceed.

[0109] Second point: Are 90% of the trading time evenly distributed within the time range of 06:00-24:00, rather than concentrated in a certain period of time? If so, it passes.

[0110] Third point: Whether the number of transactions during non-business hours (00:00-06:00 the next day) is greater than 0; if it is, the transaction will not be approved.

[0111] Fourth point: Whether the number of large transactions is greater than 0; if it is, it will not be approved.

[0112] Fifth point: Whether the number of transactions in high-risk countries such as North Korea, Iran, and Myanmar is greater than 0; if it is, the application will not be approved.

[0113] In addition to bank statement prompts specific to supermarkets, this application also includes specialized prompts for identifying establishments such as restaurants, wholesale markets, state-owned enterprises, small and medium-sized companies, financial fraud companies, and casinos.

[0114] E. Output format:

[0115] Please output the results in JSON format (JavaScriptObjectNotationFormat), which should include the following fields:

[0116] Subject: "Pass / Fail";

[0117] Final conclusion: "Recommendation approved / Recommendation rejected (please refer to the reasons for the merchant's rejection and the transaction record review to make a summary)";

[0118] Report format suggestions: Accurately list the specific problems and their causes in bullet points, without redundancy. For issues related to the workflow, summarize them into one point and include it as the last point.

[0119] Risk level: Low risk / Medium risk / High risk;

[0120] Scoring Rules: Legal entity verification involves two items; passing both adds 8 points to each. Merchant information verification involves four items; passing all four adds 8 points to each. Other material verification involves four items; passing all four adds 8 points to each. Bank statement verification adds 20 points, for a maximum of 100 points. A score below 75 indicates a high-risk merchant requiring increased inspection frequency. A score between 75 and 90 indicates a medium-risk merchant requiring targeted checks on specific deduction items; passing these checks allows the merchant to pass inspection directly. A score above 90 indicates a low-risk merchant; adjusting inspection frequency for these high-quality merchants improves user experience and retention rates.

[0121] Figure 2 A flowchart illustrating the overall processing flow of merchant inspection information is provided. Figure 2 The following embodiments will be described:

[0122] In an optional embodiment, the method for processing merchant inspection information further includes: performing a first type of image preprocessing on the external environment image to obtain a target external environment image, and then predicting the merchant's category based on the feature information extracted from the target external environment image to obtain a first category; performing a second type of image preprocessing on the internal environment image to obtain a target internal environment image, and then predicting the merchant's category based on the feature information extracted from the target internal environment image and internal voice information to obtain a second category, wherein the first type of image preprocessing and the second type of image preprocessing are different processing operations.

[0123] Optionally, the first type of image preprocessing can refer to a set of image enhancement and correction operations applied to images of the store's exterior environment. Since images of the store's exterior environment are usually taken under natural light, they are easily affected by changes in lighting and weather conditions. The first type of image preprocessing can effectively suppress overexposure and shadows, expand the dynamic range of the image, and reduce atmospheric interference, thereby obtaining a target image of the store's exterior environment with clearer details and more realistic colors, which is beneficial for accurately extracting exterior feature information.

[0124] Optionally, the second type of image preprocessing can refer to another set of image optimization operations applied to images of the store's interior environment. Addressing issues such as uneven lighting, color cast, cluttered backgrounds, and low light noise that may be caused by indoor artificial lighting, the second type of image preprocessing can perform color calibration, background simplification, and noise suppression, thereby generating a target store interior environment image with a prominent subject and uniform lighting. This facilitates the extraction of internal feature information that better reflects the actual operating conditions.

[0125] Optionally, the first type of image preprocessing may include: fusing multiple images with different exposures using high dynamic range imaging technology; applying local tone mapping algorithms to optimize highlight and shadow details respectively; using filtering algorithms to reduce glare and specular reflection; and correcting image degradation caused by weather factors such as fog and rain based on a physical model.

[0126] Optionally, the second type of image preprocessing may include: performing an automatic white balance algorithm to correct the color temperature; using deep learning-based semantic segmentation technology to identify and blur irrelevant backgrounds; employing contrast-limited adaptive histogram equalization to improve local illumination; and using deep neural networks for image noise reduction.

[0127] By performing differentiated preprocessing of the first type and the second type of image preprocessing, the quality of the external and internal environment images can be improved in a targeted manner, making the feature information extracted from the target external and internal environment images more representative, thereby improving the reliability of the first and second category predictions.

[0128] In one optional embodiment, the first type of image preprocessing includes at least one of the following operations: a first operation for expanding the dynamic range of an outdoor environment image by merging multiple outdoor environment images with different exposures, wherein the dynamic range is the ratio between the brightest pixel value and the darkest pixel value of the outdoor environment image; a second operation for adjusting the contrast of at least one region of the outdoor environment image based on the exposure information of each region in the outdoor environment image; a third operation for eliminating halo and specular reflection information in the outdoor environment image; and a fourth operation for adjusting the color saturation of the outdoor environment image based on the weather conditions at the time the outdoor environment image was captured.

[0129] Optionally, outdoor scenes often have high brightness contrast, making it easy for a single image to lose details in either the bright or dark areas. The first operation merges multiple images of the outdoor environment at different exposure levels, such as underexposed, normal, and overexposed, to create a single high dynamic range image. This can simultaneously preserve the signage text under strong light and the details of the shop in shadow, making the image content more complete. The first operation can be implemented using a high dynamic range imaging algorithm.

[0130] Optionally, to address the possibility of uneven exposure in different areas of the outdoor environment image, the second operation can independently enhance or suppress the contrast of at least one overly dark or overly bright area based on the exposure information of each area, such as the local brightness histogram. This can equalize the visual performance of the entire image and improve the recognizability of key areas. The implementation of the second operation can include adaptive histogram equalization or retinal cortex theory algorithms (Retina+CortexTheoryAlgorithm, Retinex algorithm).

[0131] Optionally, strong light shining directly onto the surface of glass, metal, or other objects can produce halos and specular reflections, creating areas of high-light interference that obscure crucial information. The third operation can detect and reduce these high-light interference areas, which helps to restore the true information such as shop signs and interior views obscured by strong light, reducing misjudgments. The third operation can be implemented using neural networks based on polarized light principles or image inpainting.

[0132] Optionally, weather conditions such as fog and haze can cause images to appear dull and reduce color saturation. The fourth operation can specifically enhance color saturation based on weather models or image statistical features. This fourth operation can restore the vibrancy and realism of image colors, making features such as shop sign colors more prominent. The implementation methods for the fourth operation may include atmospheric scattering model correction or color mapping techniques.

[0133] By sequentially or selectively applying the first, second, third, and fourth operations, the system can significantly improve the quality of images of the external environment, which is beneficial for accurately extracting feature information and predicting merchant categories.

[0134] In one optional embodiment, the second type of image preprocessing includes at least one of the following operations: a color correction operation, used to calibrate the image color by adjusting the white balance; a background simplification operation, used to determine the outline of the target object and the background area of ​​the target object by semantic segmentation, and then blur or delete the elements of the background area; and a noise reduction operation, used to reduce image noise in the store environment image when the brightness of the store environment image is detected to be lower than a preset threshold.

[0135] Optionally, the color correction operation can adjust the white balance, which can correct the overall color deviation of the image caused by indoor artificial light sources such as incandescent lamps and fluorescent lamps. This can make the image colors closer to the visual effect under standard white light, which helps to ensure the authenticity of the characteristics of product colors, decoration color schemes and other features in the target store environment image, and avoid color distortion that may mislead category judgment.

[0136] Optionally, background simplification can utilize semantic segmentation to perform pixel-level classification of the image, determining the outline of the target object (such as shelves, cash registers, merchandise display areas, and other business-related entities) and distinguishing the background areas of the target object (such as irrelevant customers, wall decorations, and piles of clutter). Then, by blurring or deleting elements in these background areas, background simplification can highlight visual subjects directly related to the merchant's business category, reduce the interference of irrelevant information on the feature extraction model, and improve the model's focusing ability.

[0137] Optionally, images captured in low-light indoor environments are prone to significant image noise, primarily generated by the sensor under high sensitivity. Noise reduction can be achieved by first detecting whether the brightness of the indoor environment image is below a preset threshold. Once determined to be a low-light image, the noise reduction process can smooth the image, suppress noise, and improve visual quality while helping to more clearly preserve the edges and texture details of the target object. Noise reduction can be implemented using non-local mean denoising and deep learning-based noise reduction networks.

[0138] In one optional embodiment, the merchant category is predicted based on feature information extracted from the target store's external environment image to obtain a first category. This includes: the merchant inspection processing system can extract textual and non-textual features from the target store's external environment image; using a short text prediction model or a long text prediction model, based on the textual features in the target store's external environment image, determine a first probability distribution in which the merchant belongs to N categories, where the input feature length of the short text prediction model is shorter than the input feature length of the long text prediction model, and N is an integer greater than 1; using an image prediction model, based on the non-textual features in the target store's external environment image, determine a second probability distribution in which the merchant belongs to N categories; weight the first probability distribution and the second probability distribution, and then determine the category with the highest weighted sum of probabilities as the first category based on the weighted calculation result.

[0139] Optionally, textual features can refer to the digital representation of text content extracted from an image using optical character recognition technology, such as word vectors or sentence embeddings. By extracting textual features, semantic information in an image can be transformed into computable data. Non-textual features can refer to visual element features unrelated to text extracted from an image using visual models such as convolutional neural networks, such as the color, shape, material, and overall architectural style of a sign. Extracting non-textual features can capture visual style information that cannot be directly described by text.

[0140] Optionally, the system can intelligently select either a short text prediction model (suitable for short store names) or a long text prediction model (suitable for long texts containing business scope or slogans) based on the length of the extracted text features. Both short and long text prediction models output a first probability distribution, which can refer to the probability value of a merchant belonging to each of N categories such as catering, retail, and services. For example, for a convenience store, the short text model can directly give a high probability of retail; for a merchant whose sign says professional computer repair, mobile phone sales, or accessories retail, the long text model can better understand the complex business and can give a high probability of IT services or retail. The adaptive selection of the model can improve the processing accuracy of texts with different complexities.

[0141] Optionally, the image prediction model can be used to process non-textual features in the target store's external environment image and output an independent second probability distribution, which can provide visual predictions in parallel with text predictions and help identify risk points where text and vision are inconsistent.

[0142] Optionally, the system performs a weighted calculation on the first and second probability distributions. For example, it can assign a higher weight to the first probability distribution corresponding to clear and unambiguous text features, while increasing the weight of the second probability distribution corresponding to visual features when the text is blurry or misleading. Finally, the category with the highest weighted sum of probabilities is determined as the first category.

[0143] Weighted fusion enables the complementarity and cross-validation of textual and visual information. When merchants attempt fraud by forging signboard text, the significant discrepancy between visual features and textual predictions can alter the probability distribution after fusion, allowing for accurate identification of high-risk true categories after weighted calculation.

[0144] In one optional embodiment, the merchant category is predicted based on feature information extracted from the target store environment image and in-store voice information to obtain a second category, including: extracting textual and non-textual features from the target store environment image; extracting semantic features from the in-store voice information; determining a third probability distribution in which the merchant belongs to N categories based on the textual features in the target store environment image using a short text prediction model or a long text prediction model; determining a fourth probability distribution in which the merchant belongs to N categories based on the non-textual features in the target store environment image using an image prediction model; determining a fifth probability distribution in which the merchant belongs to N categories based on the semantic features using a semantic prediction model; weighting the third, fourth, and fifth probability distributions, and then determining the category with the highest weighted sum of probabilities as the second category based on the weighted calculation results.

[0145] Optionally, textual features in the target store environment image can be derived from in-store posters, price lists, product labels, etc., while non-textual features in the target store environment image capture visual elements such as store decoration, table and chair layout, and product display style.

[0146] Optionally, semantic features can refer to the business-related conceptual information contained in the in-store voice information, such as extracting keywords or themes like QR code ordering, membership discounts, and food delivery from background dialogues or broadcasts through voice recognition and natural language understanding technologies.

[0147] Optionally, for the in-store environment, the system can perform parallel predictions based on textual features, non-textual features, and semantic features to determine the third, fourth, and fifth probability distributions. By weighting the third, fourth, and fifth probability distributions and determining the category with the highest weighted sum of probabilities as the second category based on the weighted calculation results, cross-modal information fusion can be achieved. When one modality feature among textual, non-textual, and semantic features is deceptive, features from other modalities can be effectively corrected in the weighted fusion, thereby improving the system's accuracy in identifying the true merchant type in complex and deliberately disguised scenarios.

[0148] In an optional embodiment, the method further includes: when the feature length of the detected text feature is less than a preset length, selecting a short text prediction model to process the text feature; and when the feature length of the detected text feature is greater than or equal to the preset length, selecting a long text prediction model to process the text feature.

[0149] Optionally, the preset length can refer to a pre-defined threshold for the number of characters or words used to define the boundary between short and long text. For example, the preset length can be set to 20 characters. The feature length can refer to the length of the character sequence or word sequence corresponding to the text content identified and extracted from the target store's external environment image or target store's internal environment image.

[0150] Optionally, when the detected text feature length is less than the preset length, such as for short store names or slogans, the system can call a short text prediction model with relatively low computational cost and fast inference speed to quickly complete the classification. When the detected text feature length is greater than or equal to the preset length, such as for longer descriptions including business scope and advertising slogans, the system can call a long text prediction model that is better at understanding long-distance dependencies and complex semantics. This helps ensure a full understanding and accurate classification of complex text information, improves the accuracy of processing long-flow data, and also improves the accuracy of risk detection for flow data, resulting in more accurate risk reports. Setting a preset length helps to optimize the allocation of computing resources and accurately match model performance. Given the diverse functions of the inspection system, which needs to process text, images, and long-flow information, a combination of short and long text prediction models is used to process relevant image and text data, improving the accuracy of merchant inspection reports.

[0151] Optionally, for images in the acquired merchant inspection information, the system can first extract image and text information from the images, and then use a targeted text processing model to recognize the text information. This processing is based on a multi-layer bidirectional Transformer encoder architecture, which may include N Transformer blocks, each consisting of a self-attention mechanism and a feedforward neural network. The model combines large-scale text pre-training with fine-tuning by downstream merchant inspection tasks, and calls upon the merchant inspection database of the bank system to improve recognition accuracy. Specifically, the processing of text features can employ a hierarchical learning rate strategy: the first layer converts Chinese information into machine-recognizable encoding, allocating approximately 10% of the computing power; the second layer processes text frequency, speech-to-text conversion, and professional inspection input information, performing pruning and attention head optimization, allocating approximately 30% of the computing power; the third layer compares merchant text information with expert input information, allocating approximately 60% of the computing power to improve output accuracy. The system can also introduce a dynamic computing power allocation mechanism to adaptively adjust resource allocation according to the inspection task load.

[0152] Optionally, the short text prediction model is suitable for processing textual information with short feature lengths, such as structured text like merchant names, addresses, and legal representative names. This short text prediction model can employ a lightweight Transformer architecture, capturing local semantic relationships through a self-attention mechanism, and can be invoked when the input feature length is less than a preset length, outputting the probability distribution of the merchant belonging to N categories. The long text prediction model is suitable for processing complex text with longer feature lengths, such as multi-paragraph information like transaction details and business instructions. This long text prediction model can employ a deep or long-sequence optimized Transformer structure, effectively modeling long-distance dependencies, and is activated when the text feature length is greater than or equal to a preset length, outputting the probability distribution of the corresponding category. Both the short and long text prediction models are part of the text processing module, collaborating with image and semantic prediction models to complete multimodal feature fusion and classification.

[0153] Optionally, for image information processing, this application introduces a residual connection structure to alleviate the gradient vanishing and exploding problems. Each residual block contains N convolutional layers and uses short-circuit connections. Image processing is mainly used to assist in verifying text classification results; therefore, an attention mechanism is adopted to make the model focus on the key visual features suggested by the text. Pruning, quantization, and knowledge distillation techniques are used to compress the model size, improving inference efficiency and recognition accuracy. Furthermore, the system can enhance training samples through automatic data augmentation strategies to improve the model's generalization ability.

[0154] Optionally, in terms of bank transaction processing, the system obtains transaction data from the bank's data lake for preprocessing and structuring. The feature extraction module includes a text input module, a numeric feature module, and a structured data module: the text input module contains character-level, word-level, and sequence-level encoders to process information such as account numbers, transaction descriptions, and time series, respectively; the numeric feature module includes a financial normalization layer, a statistical feature extraction layer, and a time feature encoding layer to uniformly represent and statistically analyze amounts, channels, and time distributions; the structured data module integrates the above features for subsequent analysis. The system can also include an anomaly detection and risk assessment module to identify abnormal transaction behavior through outlier detection.

[0155] Finally, the multi-task output layer includes a classification task header, a regression task header, and a sequence generation header, which respectively realize transaction type classification, risk score regression, and inspection report summary generation, and output a structured merchant inspection report including risk level, abnormal behavior analysis, and inspection suggestions, thereby completing the processing flow from multimodal information input to intelligent inspection decision-making.

[0156] In one optional embodiment, when the first category and the second category are different, determining the merchant's target category from the first category and the second category based on the merchant's transaction flow information includes: extracting transaction behavior features from the transaction flow information when the first category and the second category are different, wherein the transaction behavior features include at least: transaction time distribution information, transaction amount information, and transaction frequency information; and determining the merchant's target category from the first category and the second category based on the transaction behavior features.

[0157] Optionally, transaction behavior characteristics can refer to quantitative indicators extracted from merchants' transaction flow information that reflect their business models and behavioral patterns. Transaction time distribution information can describe the concentration or dispersion of transactions at different times of the day. Transaction amount information may include the distribution of single transaction amounts, large transaction thresholds, and frequency. Transaction frequency information can refer to the number of transactions per unit of time. Extracting transaction behavior characteristics from transaction flow information can transform raw, discrete transaction records into structured, computable feature vectors.

[0158] Optionally, when the first and second categories differ, transaction behavior features are extracted from the transaction flow information. This transforms the original flow records into discriminative feature vectors that quantify the merchant's true business model, facilitating targeted and effective information extraction from massive transaction data. Determining the merchant's target category from the first and second categories based on transaction behavior features allows for the identification of contradictions and potential deceptions in the surface information of the external and internal environments. For example, if a merchant's external sign indicates a bookstore while the internal environment is an office, and the transaction time distribution information shows a large number of nighttime transactions and regular large integer transactions, these transaction behavior features are more likely to point to abnormal patterns such as cash-out risks. This helps the system make judgments or markings that better reflect the actual risk situation, elevating the review logic from static formal review to dynamic behavioral analysis. By introducing objective, difficult-to-forge transaction flow information as a basis, the accuracy, reliability, and risk identification capabilities of the final target category determination can be improved.

[0159] In one optional embodiment, determining the merchant's target category from a first category and a second category based on transaction behavior characteristics includes: generating initial prompt words based on the first category and the second category, wherein the initial prompt words are used to construct judgment and selection logic regarding the first category and the second category; updating the initial prompt words to target prompt words based on transaction behavior characteristics, wherein the transaction behavior characteristics serve as the judgment basis for the judgment and selection logic in the target prompt words; inputting the target prompt words into a large language model, and determining the merchant's target category from the first category and the second category based on the prior knowledge of the large language model during the model training phase.

[0160] Optionally, the initial prompt can refer to a text template including task instructions and candidate options, which can construct the judgment and selection logic regarding the first and second categories. The target prompt can refer to the complete query text formed by filling the initial prompt with specific transaction behavior characteristics as the judgment criteria. Transaction behavior characteristics can serve as objective data input for the judgment and selection logic.

[0161] Optionally, a large language model can refer to a neural network model that has been pre-trained on large-scale text data and possesses powerful language understanding and logical reasoning capabilities. Large language model prompts can refer to a piece of natural language input by the user to the large language model or agent, used to describe the task the user wants the model to complete, the constraints, and the output format. Its components include instructions, context, input, and output formats.

[0162] Optionally, determining the merchant's target category from the first and second categories based on the prior knowledge of the large language model during the model training phase can mean that after receiving the target prompt word, the large language model will mobilize a wide range of knowledge learned during the training phase, including general operating rules and risk patterns of different industries, to perform matching analysis and reasoning on the transaction behavior characteristics described in the target prompt word with the first and second categories, and finally output a more reasonable category selection.

[0163] This application first constructs the judgment and selection logic for the first and second categories by generating initial prompt words, structuring the abstract category selection task into instructions understandable by the large language model. Then, the initial prompt words are updated to target prompt words based on transaction behavior characteristics, allowing objective transaction behavior characteristics to be embedded as specific judgment criteria into the decision instructions, facilitating the natural language fusion of data and tasks. Finally, the target prompt words are input into the large language model, which can utilize the extensive knowledge and logical reasoning capabilities acquired during pre-training. The large language model can transcend fixed rule matching, combining patterns implied by transaction behavior characteristics to evaluate the rationality of the two candidate categories (first and second), and can also identify potential risks where neither category matches. Through interaction with the large language model, the ability to gain insight and analyze complex and hidden risk behaviors can be deepened, thereby improving the intelligence level of target category decision-making and the accuracy of risk identification.

[0164] See Figure 5The document showcases a flowchart of intelligent merchant inspection analysis based on a multimodal big data model. First, based on the text and image materials submitted by merchants, the system can invoke the multimodal big data model for full-scenario rule-based review. This includes judging image clarity and relevance, verifying the consistency of identity documents such as legal representative status and business licenses, verifying transaction information such as payment receipts, verifying the consistency of the store's internal and external environment, and verifying equipment installation status. Simultaneously, for specific transaction analysis scenarios, such as catering, supermarkets, wholesale markets, and ticketing, the system can perform scenario matching. It will export the merchant's transaction transaction information from the bank and input it into the text big data model for processing. The text big data model analyzes the transaction transaction data according to a dynamic rule-based matching strategy, with matching dimensions including industry transaction benchmark range, transaction time distribution, account region, transaction frequency, and daily cumulative transaction amount. Finally, the system integrates the review results of the multimodal big data model on image and text materials, as well as the analysis results of the text big data model on transaction transaction information, to generate a structured analysis report. This analysis report includes a merchant's risk score, detailed audit content, audit suggestions for auditors, and modification suggestions for merchants. Based on the merchant's transaction flow information, it determines the merchant's target category from the first and second categories and provides more comprehensive and accurate decision support for subsequent submissions to the audit system.

[0165] Optionally, to improve the accuracy of inspection report recognition in different scenarios, which may include catering, ticketing, wholesale markets, and ticketing, this application can design different prompt word filters for different scenarios, effectively improving the accuracy of the large language model in dealing with different scenarios. The full-scenario rule strategy can include image blurring, relevance, legal entity identity consistency verification, business license consistency verification, payment code verification, business license verification, in-store / outdoor conditions verification, and equipment installation status verification; dynamic rule matching includes transaction benchmark range, transaction time distribution, account region, transaction frequency, daily cumulative transaction amount, and other transaction information. This application can effectively extract key information from massive amounts of merchant transaction information, and the analysis report can include risk scores, audit content details, audit suggestions (for auditors), and modification suggestions (for merchants), effectively empowering business and improving inspection accuracy.

[0166] Optionally, to address the shortcomings of cross-scene consistency verification and anti-tampering identification in related technologies, the image prediction model can be fine-tuned for scene adaptation by constructing a paired sample library of storefront images (outdoor environment) and interior images (indoor environment) to enhance its spatial correlation and authenticity recognition capabilities. This provides more reliable non-textual features when performing feature extraction and prediction tasks, thereby supporting overall accuracy.

[0167] Optionally, to address the issue of slow development and responsiveness in modifying prompts or business rules in related technologies, this application provides a visual rule builder. This visual rule builder allows business personnel to intuitively define or modify risk rules by combining conditions and action logic blocks. For example, the condition could be that transaction volume fluctuations exceed 30% and document ambiguity exceeds 70%, with the action being to mark it as high-risk. This can be used, for instance, to assist in generating more accurate large language model prompts, or as supplementary evidence for subsequent judgments on whether transaction behavior characteristics are abnormal, thereby indirectly supporting the agility of the entire system in adapting to business changes.

[0168] Optionally, this application combines large language models for pre-inspection review, merchant inspection, initial review by junior examiners, and final review by merchant inspection supervisors. This can reduce the workload at the grassroots level and improve model accuracy, allowing risky merchants to be directly intercepted during the inspection stage, reducing losses for banks. Furthermore, corresponding prompts can be designed for different types of merchants, intelligently identifying different merchant scenarios. Before manual pre-inspection, the large model can identify potential inspection risks, helping to significantly improve inspection efficiency and reduce errors in initial manual review.

[0169] Optionally, the merchant inspection processing system can have functions such as inspection task management, data collection and intelligent analysis, anomaly warning and handling, and report generation and statistics. It can also improve the output accuracy of merchant inspection reports by combining large language models with manual review.

[0170] Figure 6 This diagram illustrates the internal hardware structure of a merchant inspection and processing system. The system includes an input unit, memory, a communication module, an audio processor, a display, and a power supply. The memory comprises a buffer, application / function modules, a data storage area, and drivers, used for temporary storage, processing, and persistent storage of inspection-related data and programs. The communication module includes a transmitter and a receiver, responsible for data interaction with external systems such as bank databases, map interface servers, and large model servers to support functions such as merchant information uploading, location verification, and model invocation. The audio processor connects to a microphone and speaker for processing and outputting in-store voice information. The input unit may include a touchscreen, keyboard, and camera, used to acquire text and image information submitted by merchants. The memory and buffer provide space for the temporary storage and processing of multimodal data. The communication module ensures reliable transmission of inspection data, transaction logs, and model results, while the audio processor is used for feature extraction of voice information. Thus, the merchant inspection and processing system provides the necessary hardware execution environment and resource support for steps such as obtaining merchant inspection information, predicting the merchant category based on the in-store environment images and in-store voice information, and submitting the determined target category of the merchant to the review system.

[0171] See Figure 7 According to another aspect of the embodiments of this application, a processing device for merchant inspection information is also provided, including: an acquisition unit, a first prediction unit, a second prediction unit, a first determination unit, a second determination unit, and an information submission unit.

[0172] The system includes: an acquisition unit for acquiring merchant inspection information, which includes at least images of the merchant's external environment, images of the merchant's internal environment, and voice information from the merchant's internal environment; a first prediction unit for predicting the merchant's category based on feature information in the external environment images, thus obtaining a first category; a second prediction unit for predicting the merchant's category based on the images of the merchant's internal environment and the voice information from the merchant's internal environment, thus obtaining a second category; a first determination unit for determining the merchant's target category as the first category if the predicted first and second categories are the same; a second determination unit for determining the merchant's target category from the first and second categories based on the merchant's transaction flow information if the predicted first and second categories are different; and an information submission unit for submitting the determined merchant's target category to the review system.

[0173] Optionally, the merchant inspection information processing device further includes: a first category determination unit, used to perform first-type image preprocessing on the external environment image to obtain a target external environment image, and then predict the merchant's category based on the feature information extracted from the target external environment image to obtain a first category; and a second category determination unit, used to perform second-type image preprocessing on the internal environment image to obtain a target internal environment image, and then predict the merchant's category based on the feature information extracted from the target internal environment image and internal voice information to obtain a second category, wherein the first-type image preprocessing and the second-type image preprocessing are different processing operations.

[0174] Optionally, the first category determination unit includes a first category image preprocessing subunit, which includes at least one of the following modules: a first operation module, used to expand the dynamic range of the outdoor environment image by merging multiple outdoor environment images with different exposures, wherein the dynamic range is the ratio between the brightest pixel value and the darkest pixel value of the outdoor environment image; a second operation module, used to adjust the contrast of at least one area of ​​the outdoor environment image according to the exposure information of each area in the outdoor environment image; a third operation module, used to eliminate the halo and specular reflection information of the outdoor environment image; and a fourth operation module, used to adjust the color saturation of the outdoor environment image according to the weather conditions when the outdoor environment image was taken.

[0175] Optionally, the second category determination unit includes a second category image preprocessing subunit, which includes at least one of the following modules: a color correction operation module, used to calibrate image colors by adjusting white balance; a background simplification operation module, used to determine the outline of the target object and the background area of ​​the target object by semantic segmentation, and then blur or delete elements in the background area; and a noise reduction operation module, used to reduce image noise in the store environment image when the brightness of the store environment image is detected to be lower than a preset threshold.

[0176] Optionally, the first category determination unit includes: a first feature extraction subunit, used to extract textual and non-textual features from the target store's external environment image; a first probability distribution determination subunit, used to determine, based on the textual features in the target store's external environment image, the first probability distribution of the merchant belonging to N categories using a short text prediction model or a long text prediction model, wherein the input feature length of the short text prediction model is less than the input feature length of the long text prediction model, and N is an integer greater than 1; a second probability distribution determination subunit, used to determine, based on the non-textual features in the target store's external environment image, the second probability distribution of the merchant belonging to N categories using an image prediction model; and a first category determination subunit, used to perform a weighted calculation on the first and second probability distributions, and then determine the category with the highest weighted sum of probabilities as the first category based on the weighted calculation result.

[0177] Optionally, the second category determination unit includes: a second feature extraction subunit, used to extract textual and non-textual features from the target store environment image; a third feature extraction subunit, used to extract semantic features from the store's voice information; a third probability distribution determination subunit, used to determine the third probability distribution to which the merchant belongs to each of the N categories based on the textual features in the target store environment image using a short text prediction model or a long text prediction model; a fourth probability distribution determination subunit, used to determine the fourth probability distribution to which the merchant belongs to each of the N categories based on the non-textual features in the target store environment image using an image prediction model; a fifth probability distribution determination subunit, used to determine the fifth probability distribution to which the merchant belongs to each of the N categories based on the semantic features using a semantic prediction model; and a second category determination subunit, used to perform weighted calculations on the third, fourth, and fifth probability distributions, and then determine the category with the highest weighted sum of probabilities as the second category based on the weighted calculation results.

[0178] Optionally, the merchant inspection information processing device further includes: a first detection unit, used to select a short text prediction model to process the text features when the feature length of the detected text features is less than a preset length; and a second detection unit, used to select a long text prediction model to process the text features when the feature length of the detected text features is greater than or equal to a preset length.

[0179] Optionally, the second determining unit includes: a fourth feature extraction subunit, used to extract transaction behavior features from transaction flow information when the first category and the second category are different, wherein the transaction behavior features include at least: transaction time distribution information, transaction amount information and transaction frequency information; and a target category determining subunit, used to determine the merchant's target category from the first category and the second category based on the transaction behavior features.

[0180] Optionally, the target category determination subunit includes: an initial prompt word generation module, used to generate initial prompt words based on a first category and a second category, wherein the initial prompt words are used to construct the judgment and selection logic regarding the first category and the second category; a target prompt word determination module, used to update the initial prompt words to target prompt words based on transaction behavior features, wherein the transaction behavior features serve as the judgment basis for the judgment and selection logic in the target prompt words; and a model processing module, used to input the target prompt words into a large language model, and determine the merchant's target category from the first category and the second category based on the prior knowledge of the large language model during the model training phase.

[0181] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located performs the above-described method for processing merchant inspection information.

[0182] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described method for processing merchant inspection information.

[0183] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the above-described method for processing merchant inspection information.

[0184] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0185] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0187] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0190] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing merchant inspection information, characterized in that, include: Obtain merchant inspection information, wherein the merchant inspection information includes at least images of the merchant's external environment, images of the merchant's internal environment, and voice information from the merchant's internal environment; Based on the feature information in the external environment image, the category of the merchant is predicted to obtain the first category; Based on the in-store environment images and in-store voice information, the category of the merchant is predicted to obtain a second category; If the predicted first category and second category are the same, the merchant's target category is determined to be the first category; If the predicted first category and second category are different, the merchant's target category is determined from the first category and the second category based on the merchant's transaction flow information; Submit the identified target categories of the merchants to the review system.

2. The method according to claim 1, characterized in that, The method further includes: The store exterior environment image is subjected to a first type of image preprocessing to obtain a target store exterior environment image. Then, the merchant's category is predicted based on the feature information extracted from the target store exterior environment image to obtain the first category. The store environment image is subjected to a second type of image preprocessing to obtain a target store environment image. Then, based on the feature information extracted from the target store environment image and the store voice information, the merchant's category is predicted to obtain a second category. The first type of image preprocessing and the second type of image preprocessing are different processing operations.

3. The method according to claim 2, characterized in that, The first type of image preprocessing includes at least one of the following operations: The first operation is used to expand the dynamic range of the outdoor environment image by merging multiple outdoor environment images with different exposures, wherein the dynamic range is the ratio between the brightest pixel value and the darkest pixel value of the outdoor environment image. The second operation is used to adjust the contrast of at least one area of ​​the outdoor environment image based on the exposure information of each area in the outdoor environment image. The third operation is used to eliminate the halo and specular reflection information in the image of the external environment of the store; The fourth operation is used to adjust the color saturation of the outdoor environment image based on the weather conditions at the time the outdoor environment image was taken.

4. The method according to claim 2, characterized in that, The second type of image preprocessing includes at least one of the following operations: Color correction is used to calibrate image colors by adjusting white balance. Background simplification operation is used to determine the outline of the target object and the background region of the target object by semantic segmentation, and then blur or delete the elements of the background region; The noise reduction operation is used to reduce image noise in the store environment image when the brightness of the image is detected to be lower than a preset threshold.

5. The method according to claim 2, characterized in that, Based on feature information extracted from the external environment image of the target store, the category of the merchant is predicted to obtain a first category, including: Extract textual and non-textual features from the target store's external environment image; Based on the text features in the target store's external environment image, the first probability distribution of the merchant belonging to N categories is determined using either a short text prediction model or a long text prediction model. The input feature length of the short text prediction model is shorter than that of the long text prediction model, and N is an integer greater than 1. Based on the non-textual features in the external environment image of the target store, the second probability distribution of the merchant belonging to N categories is determined by the image prediction model. The first probability distribution and the second probability distribution are weighted and calculated, and then the category with the highest weighted sum of probabilities is determined as the first category based on the weighted calculation result.

6. The method according to claim 2, characterized in that, Based on feature information extracted from the target store environment image and the store's voice information, the merchant's category is predicted, resulting in a second category, including: Extract textual and non-textual features from the target store environment image; extract semantic features from the store's audio information; Based on the text features in the target store environment image, the third probability distribution of the merchant belonging to N categories is determined by using a short text prediction model or a long text prediction model. Based on the non-textual features in the target store environment image, the image prediction model is used to determine the fourth probability distribution of each of the N categories of merchants. Based on the semantic features, the semantic prediction model determines the fifth probability distribution of the N categories to which the merchants belong; The third probability distribution, the fourth probability distribution, and the fifth probability distribution are weighted and calculated. Then, based on the weighted calculation results, the category with the highest weighted sum of probabilities is determined as the second category.

7. The method according to claim 5 or claim 6, characterized in that, The method further includes: If the feature length of the detected text feature is less than the preset length, the short text prediction model is selected to process the text feature; If the feature length of the detected text feature is greater than or equal to the preset length, the long text prediction model is selected to process the text feature.

8. The method according to claim 1, characterized in that, If the first category and the second category are different, the merchant's target category is determined from the first category and the second category based on the merchant's transaction flow information, including: When the first category and the second category are different, transaction behavior features are extracted from the transaction flow information, wherein the transaction behavior features include at least: transaction time distribution information, transaction amount information, and transaction frequency information; The target category of the merchant is determined from the first category and the second category based on the transaction behavior characteristics.

9. The method according to claim 8, characterized in that, Determining the merchant's target category from the first category and the second category based on the transaction behavior characteristics includes: Initial prompt words are generated based on the first category and the second category, wherein the initial prompt words are used to construct judgment and selection logic about the first category and the second category; The initial prompt word is updated to the target prompt word based on the transaction behavior characteristics, wherein the transaction behavior characteristics serve as the basis for the judgment selection logic in the target prompt word; The target prompt word is input into the large language model, and the target category of the merchant is determined from the first category and the second category based on the prior knowledge of the large language model during the model training phase.

10. A device for processing merchant inspection information, characterized in that, include: The acquisition unit is used to acquire merchant inspection information, wherein the merchant inspection information includes at least images of the merchant's external environment, images of the merchant's internal environment, and voice information from inside the store. The first prediction unit is used to predict the category of the merchant based on the feature information in the external environment image of the store, and obtain the first category; The second prediction unit is used to predict the category of the merchant based on the in-store environment image and the in-store voice information, and obtain a second category; The first determining unit is configured to determine the merchant's target category as the first category when the predicted first category and the second category are the same; The second determining unit is used to determine the merchant's target category from the first category and the second category based on the merchant's transaction flow information when the predicted first category and the second category are different. The information submission unit is used to submit the determined target category of the merchant to the review system.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device containing the computer-readable storage medium performs the merchant inspection information processing method according to any one of claims 1 to 9.

12. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the merchant inspection information processing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the merchant inspection information processing method according to any one of claims 1 to 9.

Images