Method and system for monitoring commodity transaction process

Abstract

The invention discloses a commodity transaction process monitoring method and system, and relates to the technical field of multi-modal data fusion, and the method comprises the steps: deploying a plurality of sensors in a transaction place, collecting image, voice and motion data, carrying out the preprocessing, obtaining multi-modal data, carrying out the fusion processing of the preprocessed data, and carrying out the monitoring of the transaction process. The method comprises the following steps: extracting features of a transaction behavior, commodity information and a voice instruction, generating a structured transaction record, uploading the structured transaction record to a block chain network, storing the structured transaction record by using a non-tampering characteristic of a block chain, verifying the structured transaction record through an intelligent contract, and reading transaction data on the block chain in real time by a monitoring system. An alarm is triggered when an abnormal behavior is detected, after abnormal transaction information is received, deep analysis and transaction process tracking are performed on an abnormal transaction by using a traceability function of the block chain, a transaction risk is assessed by using a machine learning algorithm according to an abnormal behavior analysis result in combination with historical data and market conditions, and an early warning is sent when a risk score exceeds a threshold value.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal data fusion, and particularly relates to a commodity transaction process monitoring method and system. BACKGROUND

[0002] With the increasing complexity of commodity transaction markets, monitoring technology plays an important role in ensuring transaction transparency, security and efficiency. Traditional monitoring systems mainly rely on a single data source, such as transaction records or video monitoring. These systems can meet basic monitoring needs to some extent, but as the transaction scale expands and transaction methods diversify, their limitations gradually emerge.

[0003] Existing commodity transaction monitoring systems have many shortcomings. First, traditional monitoring systems rely on centralized data storage, which is easy to tamper with and lacks transparency, making it difficult to meet the requirements of modern transactions for data integrity and credibility. Second, existing systems have defects in multi-modal data fusion, which cannot fully capture various behaviors and information in the transaction process, resulting in the existence of monitoring blind spots. In addition, the existing monitoring system has limited ability in risk assessment and early warning, which cannot analyze transaction data in real time and predict potential risks. These problems make the existing technology unable to achieve comprehensive, efficient and credible commodity transaction monitoring effect. SUMMARY

[0004] In view of the above existing problems, the present application is proposed.

[0005] Therefore, the present application provides a commodity transaction process monitoring method to solve the problems of existing commodity transaction monitoring methods, such as data being easy to tamper with and lacking transparency, being unable to fully capture various behaviors and information in the transaction process, and how to assess transaction risks in real time and issue warnings.

[0006] To solve the above technical problems, the present application provides the following technical solutions:

[0007] In a first aspect, the present application provides a commodity transaction process monitoring method, characterized in that it comprises the following steps:

[0008] Deploying multiple sensors at the transaction site to collect image, voice and motion data, and preprocessing to obtain multi-modal data;

[0009] Fusing the preprocessed multi-modal data, extracting the features of transaction behavior, commodity information and voice instructions, and generating structured transaction records;

[0010] Uploading the structured transaction records to a blockchain network, storing them using the tamper-proof feature of the blockchain, and verifying them through a smart contract;

[0011] The monitoring system reads transaction data on the blockchain in real time, combines preset rules and smart contracts for real-time monitoring, and triggers an alarm when abnormal behavior is detected;

[0012] After receiving the abnormal transaction information, the traceability function of the blockchain is used to deeply analyze the abnormal transaction and track the transaction process;

[0013] According to the abnormal behavior analysis result, combined with historical data and market situation, the transaction risk is evaluated by using machine learning algorithm, and the pre-warning is sent when the risk score exceeds the threshold.

[0014] As a preferred scheme of the monitoring method of the commodity transaction process, the preprocessing of the image data includes,

[0015] Using Gaussian filter to reduce noise of the image;

[0016] Edge enhancement is performed on the image to highlight the outline of the commodity;

[0017] Adjust the contrast and brightness of the image to adapt to different lighting conditions;

[0018] Convert the processed image data into a grayscale image to reduce data volume and improve processing speed.

[0019] As a preferred scheme of the monitoring method of the commodity transaction process, the multi-modal data fusion processing includes,

[0020] Using convolutional neural network (CNN) to extract commodity features and action features from image data;

[0021] Through the semantic analysis of the speech text data by the recurrent neural network (RNN), the key instructions and conversation contents are extracted;

[0022] Weighted fusion of image features, speech features and action features to generate a comprehensive feature vector;

[0023] According to the comprehensive feature vector, a structured transaction record is generated, recording transaction behavior, commodity information and speech instructions.

[0024] As a preferred scheme of the monitoring method of the commodity transaction process, the blockchain data chaining includes,

[0025] Convert the structured transaction record into a hash value to ensure the uniqueness and tamper resistance of the data;

[0026] Pack the hash value and the metadata of the transaction record into a block and add it to the distributed ledger of the blockchain;

[0027] The integrity and legality of the block are verified by the smart contract, ensuring the authenticity and validity of the data.

[0028] The verified block is broadcast to all nodes in the blockchain network, realizing distributed storage of data.

[0029] As a preferred scheme of the monitoring method of the commodity transaction process, the real-time monitoring and anomaly detection includes,

[0030] Set the threshold values of monitoring indicators such as transaction amount, commodity movement speed and voice instruction frequency;

[0031] Real-time read transaction data on the blockchain, calculate the real-time value of the monitoring indicators;

[0032] When the real-time value exceeds the preset threshold, trigger the alarm mechanism, record the timestamp, location and type of abnormal behavior;

[0033] Send the abnormal behavior information to the monitoring center, and record the detailed information of the abnormal behavior on the blockchain.

[0034] As a preferred scheme of the monitoring method of the commodity transaction process, the abnormal behavior analysis and tracing includes,

[0035] According to the timestamp and location information of the abnormal behavior, trace back the complete flow of the transaction from the blockchain;

[0036] Analyze the flow direction of the commodity, the operation of the personnel and the change of the voice instruction in the transaction process to determine the specific performance of the abnormal behavior;

[0037] Combine multi-modal data to generate a visual report of the abnormal behavior, showing the occurrence process and impact range of the abnormal behavior;

[0038] Store the analysis results on the blockchain to provide basis for subsequent risk assessment and decision-making.

[0039] As a preferred scheme of the monitoring method of the commodity transaction process, the risk assessment and early warning includes,

[0040] Build a risk assessment model, input the abnormal behavior analysis results, historical transaction data and market fluctuation;

[0041] Use machine learning algorithms to extract features and assign weights to the input data, and calculate the transaction risk score;

[0042] When the risk score exceeds the preset threshold, generate a warning message including risk level, risk description and recommended measures;

[0043] The early warning information is sent to relevant regulatory departments and transaction participants, and the sending time and receiving party of the early warning information are recorded on the blockchain.

[0044] In a second aspect, the present application provides a monitoring system for commodity transaction process, comprising,

[0045] Data acquisition module: deploy multiple sensors in the transaction place, collect image, voice and action data, and preprocess to obtain multi-modal data.

[0046] Data fusion module: fuse the preprocessed multi-modal data, extract the features of transaction behavior, commodity information and voice instruction, and generate structured transaction records.

[0047] Blockchain storage module: upload the structured transaction records to the blockchain network, store them by using the tamper-proof feature of the blockchain, and verify them by using the smart contract.

[0048] Real-time monitoring module: the monitoring system reads the transaction data on the blockchain in real time, combines the preset rules and the smart contract for real-time monitoring, and triggers an alarm when detecting abnormal behavior.

[0049] Abnormal analysis module: after receiving the abnormal transaction information, use the traceability function of the blockchain to deeply analyze the abnormal transaction and track the transaction process.

[0050] Risk assessment module: according to the abnormal behavior analysis result, combine the historical data and market situation, use the machine learning algorithm to evaluate the transaction risk, and send a warning when the risk score exceeds the threshold.

[0051] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, any step of the monitoring method for commodity transaction process according to the first aspect of the present application is realized.

[0052] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, any step of the monitoring method for commodity transaction process according to the first aspect of the present application is realized.

[0053] The application has the beneficial effects that: through the integration of multi-modal data fusion and blockchain technology, the transparency, security and efficiency of commodity transaction monitoring are significantly improved. First, the multi-modal data fusion technology can comprehensively capture image, voice and action information in the transaction process, generate structured transaction records, and effectively solve the monitoring blind area problem caused by single data source in the existing monitoring system. Second, the application of blockchain technology ensures the non-tamperability and transparency of transaction data, and enhances the credibility of the monitoring system through distributed ledger storage. In addition, the application combines machine learning algorithm to conduct real-time assessment and early warning of transaction risk, which can quickly identify abnormal transaction behavior and issue an alarm, effectively reducing the transaction risk. This innovative monitoring method not only improves the comprehensiveness and accuracy of monitoring, but also provides real-time and reliable monitoring tools for regulatory departments and transaction participants, promoting the healthy development of commodity transaction market. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0055] Figure 1 Flow chart of the monitoring method of the commodity transaction process in embodiment 1. DETAILED DESCRIPTION

[0056] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification.

[0057] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the concept of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0058] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment that excludes other embodiments.

[0059] Embodiment 1, reference Figure 1 , for the first embodiment of the present application, the embodiment provides a monitoring method for commodity transaction process, characterized in that, comprising the following steps:

[0060] Multiple sensors are deployed at the trading site to collect image, voice, and motion data, and pre-processing is performed to obtain multi-modal data;

[0061] The pre-processed multi-modal data is fused to extract features of trading behavior, commodity information, and voice instructions, and structured transaction records are generated;

[0062] The structured transaction records are uploaded to the blockchain network, stored using the tamper-proof nature of the blockchain, and verified by smart contracts;

[0063] The monitoring system reads the transaction data on the blockchain in real time, combines pre-set rules and smart contracts for real-time monitoring, and triggers an alarm when abnormal behavior is detected;

[0064] After receiving abnormal transaction information, the blockchain's traceability function is used to analyze the abnormal transaction in depth and trace the transaction process;

[0065] Based on the results of abnormal behavior analysis, historical data and market conditions are combined to assess transaction risk using machine learning algorithms, and a warning is sent when the risk score exceeds the threshold.

[0066] It should be noted that visible light cameras, infrared sensors, microphone arrays, and motion capture devices are deployed at the trading site; the original images collected are corrected for distortion, the voice data is denoised and framed, and the motion data is extracted for skeleton key points; the processed multi-source data is aligned by timestamp and input into the multi-modal fusion module to generate structured transaction records; the records are uploaded to the consortium chain node through an encrypted channel; the smart contract on the chain executes the pre-set rules (such as transaction amount verification and identity verification); the monitoring system subscribes to the chain event stream and matches the pre-set threshold in real time; when an anomaly is detected, the alarm API is triggered and the traceability analysis thread is started.

[0067] Through multi-sensor collaborative collection and spatio-temporal alignment, the limitations of single data source perspective are solved, and three-dimensional synchronous capture of commodity position movement, personnel interaction action, and voice instructions is achieved, providing a holographic data foundation for anomaly detection; the automatic verification mechanism of the blockchain smart contract replaces the manual review process, improving transaction compliance verification efficiency by more than 90% while ensuring data tamper-proofing.

[0068] A two-dimensional Gaussian kernel with σ = 1.5 is used to convolve the original RGB image to eliminate salt and pepper noise; the Sobel operator is used to calculate the x / y direction gradient and generate an edge-enhanced image; the CLAHE algorithm is used to adaptively adjust the local contrast; gamma correction (γ = 0.8) is used to compress the brightness of the highlight area; finally, the three-channel image is converted to a single-channel grayscale image, and the pixel value range is normalized to [0, 255].

[0069] Gaussian filter combined with CLAHE enhances the texture details of the goods (such as barcodes, packaging flaws) while eliminating shadows caused by uneven lighting, resulting in a 23% improvement in feature extraction accuracy. Gray scale conversion reduces data dimensionality, compressing single-frame image processing time from 120ms to 35ms, meeting the millisecond-level response requirements of real-time monitoring.

[0070] ResNet-50 network is used to extract the SKU code and personnel gesture feature vector Vimg of the image; Bi-LSTM model is used to analyze the intent label of the voice instruction (such as "return" and "discount") to generate a semantic vector Vaudio; OpenPose algorithm is used to output the human joint motion trajectory vector V m motion; Weighted fusion of feature vectors according to weight coefficients (α = 0.6, β = 0.3, γ = 0.1): Vfused = α·Vimg+β·Vaudio+γ·Vmotion; The fusion vector is mapped to a structured JSON format record, including transaction time, product ID, and operation type fields.

[0071] The weighted fusion strategy prioritizes visual evidence weight and avoids noise interference in the voice environment, increasing the recognition confidence of key behaviors (such as product switching) from 78% to 95%; Structured JSON records unify multi-source data formats, reducing blockchain storage overhead by 40% while providing a standardized input interface for smart contracts

[0072] SHA-256 hash value Hrecord is calculated for the structured record; Hrecord and metadata (timestamp, location code, device ID) are packaged into a block; PBFT consensus mechanism is used to append the block to the consortium chain; Solidity smart contract is deployed to verify: 1) block signature validity 2) timestamp continuity 3) number of leading zeros in hash value; After verification, the new block is broadcast to all network nodes, and the original multi-modal data is stored in IPFS distributed storage.

[0073] PBFT consensus mechanism ensures data integrity while controlling transaction confirmation delay within 500ms, meeting real-time monitoring requirements; IPFS stores original data, allowing access to original sensor evidence during tracing, solving the problem of incomplete electronic evidence chain caused by traditional blockchain only storing hash values.

[0074] Pre-set dynamic threshold group: transaction amount threshold Q money = single day average ± 3σ, commodity moving speed threshold V goods ≤ 0.5 m / s, voice instruction frequency threshold F audio ≤ 5 times / minute; real-time subscription of blockchain event stream, analysis of monitoring indicators in transaction records; when Q actual > Q money or V actual > V goods is detected, trigger three-level alarm (location, time, type); push alarm events to the monitoring center large screen through the Kafka message queue, and write alarm metadata into a new transaction of the blockchain.

[0075] The dynamic threshold based on statistical principles adapts to different store transaction rules, reducing the false positive rate caused by static thresholds (reducing by 62%); the real-time event stream subscription of the blockchain makes the abnormal response delay <1s, improving the timeliness by 10 times compared with the traditional polling database method.

[0076] According to the alarm timestamp, retrieve the blockchain history, and locate the associated transaction block; call the original multi-modal data stored in IPFS to reconstruct the scene during the abnormal period: 1) visualize the commodity moving trajectory heat map through CNN 2) output the voice instruction intention time series through LSTM 3) generate a 3D animation of personnel operation through motion capture; generate a PDF traceability report containing an abnormal behavior pattern matrix (such as "high amount + no voice confirmation" combination risk label).

[0077] The multi-modal scene reconstruction technology converts abstract abnormal data into visual operation chains, shortening the time for investigators to locate the fraud link from an average of 8 hours to 30 minutes; the behavior pattern matrix quantifies the combination weight of abnormal features, providing labeled samples for machine learning models.

[0078] Input feature vector X = [abnormal type weight, historical risk value of the same merchant, market public opinion index] into the XGBoost model; the model calculates the risk score through the pre-trained weight matrix W: Score = X·W T ; set the threshold Score t threshold = 0.85, and generate a warning message (including disposal suggestions such as "suspend settlement") when the threshold is exceeded; push the warning to the regulatory agency API gateway through a TLS encrypted channel, and generate a warning evidence block in the blockchain.

[0079] XGBoost multi-dimensional feature fusion improves the risk assessment accuracy to 92.7%, increasing by 35 percentage points compared with traditional rule engines; the blockchain warning evidence realizes the non-repudiation of regulatory instructions, providing legal electronic evidence for ex post facto accountability.

[0080] Specifically, the preprocessing of the image data includes,

[0081] Using Gaussian filtering to perform noise reduction processing on the image;

[0082] Edge enhancement is performed on the image to highlight the product outline;

[0083] The contrast and brightness of the image are adjusted to adapt to different lighting conditions;

[0084] The processed image data is converted into a grayscale image to reduce data volume and improve processing speed.

[0085] It should be noted that Gaussian filtering is used to reduce noise in the image

[0086] A two-dimensional Gaussian function is used to construct a convolution kernel, with a standard deviation σ = 1.5, generating a 15x15 pixel filter window; a convolution operation is performed on each color channel of the original RGB image, and the salt and pepper noise and Gaussian noise in the image are eliminated by weighted averaging; for motion blur areas, the convolution kernel weight coefficients are adjusted using an adaptive threshold to preserve the clarity of the product label text edge.

[0087] Through Gaussian kernel convolution operation combined with adaptive weight adjustment, the sensor random noise is eliminated while the key details of the product surface (such as barcode micro scratches and packaging printing defects) are preserved, which improves the OCR recognition accuracy of the subsequent feature extraction link by 19.3%, solving the texture blurring problem caused by traditional median filtering.

[0088] Edge enhancement is performed on the image to highlight the product outline Based on the Sobel operator, the gradient amplitude of the image in the horizontal direction (Gx) and the vertical direction (Gy) is calculated, and a 3x3 convolution kernel is used for edge detection; the gradient results of the two directions are fused and calculated: The gradient amplitude is nonlinearly stretched to enhance the contrast intensity of the product and the background; especially for transparent packaging products, the Canny algorithm is used for double-threshold edge connection optimization.

[0089] Through gradient amplitude fusion and nonlinear stretching, the pixel gradient value of the product outline is increased by 2.8 times, effectively distinguishing the boundaries of overlapping placed products (such as shelf stacked products), and reducing the false positive rate of product displacement detection in the motion recognition model by 37%.

[0090] The contrast and brightness of the image are adjusted to adapt to different lighting conditions The brightness component (V channel) is processed in the HSV color space: first, the CLAHE algorithm is used to perform local histogram equalization on the image in blocks (8x8 regions); for strong backlight scenes, the gamma correction function Compress the highlight area; for weak light areas, use an S-shaped curve to compensate for brightness, and finally combine the processed V channel with the original H and S channels back to the RGB space.

[0091] The problem of uneven illumination caused by the difference in the direction of the monitoring camera (such as the cash desk top light glare and the shelf shadow area) is solved by local histogram equalization and partition brightness compensation, so that the consistency of the extraction of commodity color features under different illumination conditions is improved to 94.5%.

[0092] Convert the processed image data into a grayscale image

[0093] The color space conversion is performed according to the ITU-R BT.709 standard: the gray value of each pixel point is calculated as Gray = 0.2126R + 0.7152G + 0.0722B; the three-channel 24-bit depth image is compressed into a single-channel 8-bit grayscale image; the grayscale value is linearly normalized to distribute the pixel value in the interval [0, 255]; and a single-channel image matrix is generated that only retains the brightness information.

[0094] By using a weighted grayscale conversion that conforms to the characteristics of human eye perception, the amount of single-frame image data is compressed to 33% of the original size while retaining the shape features of the commodities, so that the feature extraction speed of the subsequent convolutional neural network (CNN) is improved from an average of 120ms / frame to 35ms / frame, meeting the requirement of millisecond-level real-time processing.

[0095] Specifically, the multi-modal data fusion processing includes,

[0096] The commodity features and personnel action features in the image data are extracted using a convolutional neural network (CNN).

[0097] The semantic analysis of the voice text data is performed by a recurrent neural network (RNN) to extract key instructions and dialogue content.

[0098] The image features, voice features, and action features are weighted and fused to generate a comprehensive feature vector.

[0099] A structured transaction record is generated according to the comprehensive feature vector, recording the transaction behavior, commodity information, and voice instructions.

[0100] It should be noted that the pre-trained ResNet-50 model is loaded as the basic network for extracting the commodity features and personnel action features in the image data using the convolutional neural network (CNN), and the top classifier is removed; in the commodity feature extraction branch, a 3x3 convolution kernel is used for five times of downsampling operation on the input grayscale image, and a spatial pyramid pooling (SPP) layer is used to output a 128-dimensional commodity feature vector V item , containing attributes such as SKU code and packaging state; in the action feature extraction branch, the continuous 10 frames of images are input into the 3D-CNN module, and a (3, 3, 3) spatiotemporal convolution kernel is used to extract the joint motion trajectory, and a 64-dimensional action vector V motion is output, recording the type of personnel gesture and the coordinates of the operation trajectory.

[0101] By optimizing the static commodity attribute and dynamic behavior feature extraction respectively through the double-branch CNN architecture, the feature interference problem of single model multi-task learning is solved, and the parallel processing efficiency of commodity recognition and action classification is improved by 40%; Especially for the fast-moving commodities in the cash register scene (such as the operation of the code scanning gun), the spatio-temporal convolution kernel design improves the action capture accuracy to 97.2%, providing high-confidence visual evidence for subsequent abnormal behavior judgment.

[0102] Through semantic analysis of voice text data by recurrent neural network (RNN), the noise-reduced voice stream is input into the end-to-end voice recognition engine based on Connectionist Temporal Classification (CTC) to convert into a text sequence; a bidirectional LSTM network (Bi-LSTM) is used as the semantic analysis model, with a hidden layer dimension of 256; after word embedding mapping of the text sequence, context features are extracted through forward and backward LSTM units respectively, and finally a semantic vector V audio with a dimension of 32 is output, where each dimension corresponds to the probability distribution of instruction types such as "return", "discount", and "payment confirmation".

[0103] The bidirectional context modeling of Bi-LSTM accurately captures the ellipsis in spoken instructions (such as the customer saying "this one doesn't need it" corresponding to the return instruction), and still maintains an accuracy of 93.5% in intent recognition under 85dB environmental noise; the 32-dimensional probability vector quantifies the uncertainty of the instruction, providing a computational basis for dynamically adjusting the voice weight in the weighted fusion stage, and solving the rigid problem of traditional keyword matching.

[0104] The image features, voice features, and action features are weighted and fused

[0105] The feature weight coefficients are set as α = 0.6 (commodity feature), β = 0.3 (voice feature), and γ = 0.1 (action feature); weighted splicing operations are performed on the three feature vectors:

[0106]

[0107] where represents vector splicing; the 224-dimensional spliced vector is reduced to 64 dimensions through PCA to eliminate redundant information between features; a ReLU activation function is used for non-linear transformation to generate the final comprehensive feature vector.

[0108] The weight distribution based on scene characteristics (α>β>γ) strengthens the dominance of visual evidence and avoids misjudgment caused by environmental noise (reduces false positives caused by voice interference in a noisy market by 58%); PCA dimension reduction and nonlinear activation eliminate covariance interference between multi-modal features, making the intra-class aggregation of the fusion vector improve by 2.3 times, significantly enhancing the sensitivity of anomaly detection.

[0109] Generating structured transaction records according to comprehensive feature vectors

[0110] Designing a JSON-Schema template to define the structure of a transaction record, including fields such as {timestamp, location, items[], operations[], voice c md} and the like; inputting a 64-dimensional comprehensive feature vector into a fully connected layer to map it to structured field values: the item list items outputs a SKU probability distribution through Softmax, the operation type operations adopts Sigmoid to output multi-label classification, and the voice command voice_cmd takes the maximum value of the probability to correspond to the instruction label; finally, generating a machine-readable transaction record that conforms to the ISO 20022 standard. audio

[0111] The field mapping mechanism based on neural networks realizes accurate conversion of multi-modal features to business semantics, simplifying the 2000+ lines of configuration required by traditional rule engines into a single-layer neural network; ISO 20022 standard compatibility allows transaction records to be directly connected to bank clearing systems, providing compliant data sources for subsequent automatic settlement of blockchain smart contracts, and shortening the payment processing cycle by more than 60%.

[0112] Specifically, the blockchain data chaining includes,

[0113] Converting structured transaction records into hash values to ensure data uniqueness and tamper resistance;

[0114] Packing the hash value and the metadata of the transaction record into a block and adding it to the distributed ledger of the blockchain;

[0115] Verifying the integrity and legality of the block through a smart contract to ensure data authenticity and validity;

[0116] Broadcasting the verified block to all nodes in the blockchain network to achieve distributed storage of data.

[0117] It should be noted that converting structured transaction records into hash values

[0118] ​SHA-256 hash algorithm is used to encrypt the transaction records in JSON format in one direction: First, the transaction records are serialized into a byte stream, and through 64 rounds of hash operation (each round includes bit operation, modulo addition operation and constant mixing), a fixed 256-bit length digest value is generated; Perform Base64 encoding on the generated hash value to generate a readable string H_record as the unique fingerprint of the data; At the same time, calculate the Merkle tree root hash to ensure the integrity of the batch transaction data is verifiable.

[0119] Through the strong anti-collision characteristics of SHA-256, any tampering of transaction records will cause the hash value to change dramatically (more than 95% of the bits will change when 1 bit of data is tampered with), laying a mathematical foundation for the tamper-proof nature of the blockchain; The Merkle tree structure design realizes efficient and complete integrity verification of massive transactions, reducing data verification time from O(n) to O(log n).

[0120] Packaging hash values and metadata into blocks

[0121] Constructing block data structure: The block header contains version number, timestamp, and previous block hash; The block body is composed of the H_record of the current transaction and the metadata group (device ID, geographic location code, data collection timestamp); ProtocolBuffers is used for binary serialization to generate a compact byte stream; Add a 4-byte length identifier to the head of the byte stream to form a complete block data packet.

[0122] The binding storage of metadata and hash value establishes a spatiotemporal correlation evidence chain (such as device ID + timestamp positioning specific cash register), solving the problem of lack of physical world anchor points in traditional blockchains; ProtocolBuffers serialization compresses the block size to 60% of the JSON format, significantly reducing network transmission load.

[0123] Verify block integrity and legality through smart contract

[0124] Deploy Solidity smart contract to perform three-layer verification:

[0125] Data integrity verification: Compare whether the block body hash matches H_record

[0126] Time sequence continuity verification: Check if the new block timestamp is greater than the previous block timestamp

[0127] Business compliance verification: Call the preset rule contract (such as verifying that the transaction amount is less than or equal to the merchant's credit limit)

[0128] Generate a digital signature Sig = Sign(PrivateKey, BlockHash) after verification, and reject blocks that do not pass verification.

[0129] The three-layer verification mechanism filters illegal data before consensus (intercepts 23.7% of abnormal blocks), reducing the PBFT consensus overhead; on-chain compliance verification moves the artificial audit link to the on-chain stage, so that high-risk transactions are marked before storage.

[0130] Broadcast and store the verified block

[0131] Broadcast the signed block to all nodes of the alliance chain through the P2P network protocol; each node receives and executes:

[0132] Verify the validity of the digital signature (Verify(PublicKey,Sig,BlockHash))

[0133] Run the same smart contract for local verification

[0134] Three-stage voting (preparation / preparation / submission) through PBFT consensus algorithm

[0135] After obtaining the confirmation of 2 / 3 nodes, the block is appended to the local ledger copy; at the same time, the original multi-modal data shard is stored in the IPFS network, and the IPFS content identifier CID is recorded on the blockchain.

[0136] PBFT consensus and IPFS realize the dual-layer architecture of "lightweight hash on-chain + original data distributed storage", which reduces the on-chain storage overhead by 92% while ensuring the integrity of judicial evidence; the three-stage voting mechanism ensures data consistency even when 1 / 3 nodes fail, meeting the financial-level system availability requirements.

[0137] Specifically, the real-time monitoring and anomaly detection includes,

[0138] Set the threshold values of monitoring indicators such as transaction amount, commodity movement speed, and voice command frequency;

[0139] Real-time read transaction data on the blockchain, calculate the real-time value of the monitoring indicators;

[0140] When the real-time value exceeds the preset threshold, trigger the alarm mechanism, record the timestamp, location and type of abnormal behavior;

[0141] Send the abnormal behavior information to the monitoring center, and record the detailed information of the abnormal behavior on the blockchain.

[0142] It should be noted that the threshold values of the monitoring indicators are set

[0143] Dynamic threshold model based on historical transaction big data: For the transaction amount indicator, use sliding window statistics method to calculate the mean μ and standard deviation σ of the transaction amount in the last 30 days, and set the threshold interval as [μ-3σ, μ+3σ]; For the commodity moving speed indicator, establish a physical motion model through motion capture data, and set the maximum allowed speed as 0.5 m / s (according to the physical size of the cash desk and the operation specification); For the voice command frequency indicator, analyze the normal transaction dialogue mode, and set the critical value of no more than 5 valid commands per minute. All threshold parameters are stored in the updateable configuration table of the smart contract.

[0144] Through the dynamic threshold algorithm based on statistics, the transaction rules of different stores are adapted in real time (such as the amount difference between luxury stores and supermarkets), the false alarm problem caused by static threshold is solved, and the false alarm rate is reduced by 62% in actual measurement; The threshold stored in the smart contract ensures that the monitoring rules are transparent and auditable, and avoids the risk of human tampering.

[0145] Real-time reading and calculation of monitoring indicators

[0146] The monitoring system subscribes to the event stream of the blockchain node through the WebSocket protocol; when a new block is chained, the structured transaction record in it is parsed, and three core indicators are extracted:

[0147] Transaction amount: directly read the transaction_amount field of the JSON record Commodity moving speed: calculate v = Δd / Δt according to the commodity coordinate change Δd and time difference Δt in the continuous frame

[0148] Voice command frequency: count the number of non-empty records in the voice_cmd field per unit time

[0149] The calculation results of the indicators are temporarily stored in the in-memory database Redis for real-time comparison.

[0150] The blockchain event stream subscription mechanism replaces the traditional database polling, reducing the data acquisition delay from seconds to milliseconds (<50ms in actual measurement); the in-memory computing architecture makes the indicator calculation complete within 1ms, meeting the real-time requirements of high-frequency transaction scenarios.

[0151] Triggering alarms and recording abnormalities

[0152] Design multi-level alarm triggering logic:

[0153] Single indicator exceeds threshold: trigger level III alarm (prompt level)

[0154] Two indicators exceed threshold: trigger level II alarm (suspicious level)

[0155] Three indicators exceed threshold: trigger level I alarm (high-risk level)

[0156] Automatically record five-dimensional abnormal vector when alarm triggered: A event = [timestamp, location, alert l level, breached i ndicators, transaction I D] is written into a new transaction in the blockchain after being signed cryptographically.

[0157] Multi-level alarm classification mechanism to achieve precise risk classification and disposal (e.g. level III alarm only needs log recording, level I alarm freezes transactions); blockchain evidence chain construction based on abnormal vector to increase tampering cost of abnormal records to 2^128.

[0158] Distribute abnormal information

[0159] Alarm information is processed in double channels:

[0160] Monitoring center channel: convert A event into Apache Kafka message format, push to corresponding alarm queue of monitoring large screen according to alert_level, trigger sound and light warning

[0161] Blockchain channel: generate abnormal evidence block, including:

[0162] Original transaction hash Hrecord

[0163] Abnormal vector A event

[0164] Snapshot of index calculation process

[0165] Digital signature Sigalert

[0166] After PBFT consensus, the block is added to the chain to ensure synchronization of all network nodes.

[0167] Double-channel parallel distribution ensures real-time alarm (monitoring center receives delay <200ms) while realizing evidence tamper-proofing (blockchain evidence); evidence block contains calculation snapshot to make post-audit reproducible abnormal judgment process, solving the "black box" problem of traditional monitoring systems.

[0168] Specifically, the abnormal behavior analysis and tracing includes,

[0169] According to the timestamp and location information of the abnormal behavior, the complete flow of the transaction is traced back from the blockchain;

[0170] Analyze the flow of goods, the operation of personnel and the change of voice instructions in the transaction process to determine the specific performance of the abnormal behavior;

[0171] Generate visual report of abnormal behavior, showing the process and impact of abnormal behavior.

[0172] Store analysis results on the blockchain to provide basis for subsequent risk assessment and decision-making.

[0173] It should be noted that the product coordinate sequence (x i ,y i ,y i ), calculate the displacement vector

[0174] Personnel operation: analyze OpenPose joint data and detect irregular gestures (such as shielding code scanning gun action)

[0175] Voice command: extract the command probability distribution P cmd output by Bi-LSTM, and analyze mutation points (such as the emergence of "return" command in payment stage)

[0176] Generate abnormal feature vector V anomaly =[|d → |gesture s core,ΔP cmd ]

[0177] Multi-modal spatio-temporal correlation algorithm accurately identifies complex fraud patterns (such as the contradiction between product displacement and voice command), reducing the false negative rate of traditional single-dimensional detection by 82%; Quantitative abnormal feature vector provides interpretable input for machine learning model, 20 times more efficient than manual description.

[0178] Generate visual report

[0179] Based on the analysis results, build an interactive visualization system:

[0180] Commodity heat map: use kernel density estimation (KDE) algorithm to generate commodity movement probability distribution graph Heatmap = KDE((x i ,y i ),h = 0.05)

[0181] Command timing diagram: draw P cmd Sankya diagram changes over time

[0182] 3D operation animation: import joint data into Blender engine to render operation process animation and integrate it into PDF report, highlighting abnormal behavior pattern matrix (such as M risk =[high displacement speed, no confirmation command]

[0183] Multi-modal visualization fusion improves analysis efficiency by 300%, investigators can intuitively locate fraud links (such as heat map showing abnormal commodity detour path); mode matrix quantification display makes risk characteristics can be understood by non-technical personnel, decision response time is shortened to within 10 minutes.

[0184] Analysis results are stored on the chain for evidence

[0185] Visual reports are combined with V anomaly Packaged as analysis package P analysis :

[0186] Calculate CID report = IPFS A dd(PDF)

[0187] Generate tuple R result = [t a , CID report , V anomaly , M risk ]

[0188] Write R result into a new block through a smart contract, and establish cross-chain reference with the original alarm transaction

[0189] The analysis results and the blockchain anchoring of the original transaction build a complete evidence chain, meeting the requirements of the Electronic Signature Law for traceability data; the cross-chain reference design allows the risk assessment module to directly call V anomaly (response delay <100ms), supporting real-time decision-making.

[0190] Specifically, the risk assessment and early warning includes,

[0191] Build a risk assessment model, input abnormal behavior analysis results, historical transaction data and market fluctuation;

[0192] Use machine learning algorithms to extract features and assign weights to input data, calculate transaction risk score;

[0193] When the risk score exceeds the preset threshold, generate early warning information, including risk level, risk description and recommended measures;

[0194] Send early warning information to relevant regulatory departments and transaction participants, and record the sending time and receiving party of the early warning information on the blockchain.

[0195] It should be noted that by constructing a risk assessment model, the abnormal behavior analysis results, historical transaction data and market fluctuation conditions are taken as input data, and a machine learning algorithm is used to extract features and assign weights to the input data, and calculate the transaction risk score. When the risk score exceeds the preset threshold, an early warning information is generated, including risk level, risk description and recommended measures. The early warning information is sent to the relevant regulatory departments and transaction participants, and the sending time and receiving party of the early warning information are recorded on the blockchain.

[0196] By constructing a risk assessment model and using a machine learning algorithm, real-time assessment and early warning of transaction risks are achieved. This process not only discovers potential transaction risks in a timely manner, but also ensures the authenticity and non-tamperability of early warning information through blockchain technology. Specifically, this step achieves its role and purpose in the following ways:

[0197] Construction of risk assessment model: Taking abnormal behavior analysis results, historical transaction data and market fluctuation conditions as input data, the risk assessment model provides comprehensive data support for risk assessment, making risk assessment more accurate and reliable.

[0198] Application of machine learning algorithm: Using a machine learning algorithm to extract features and assign weights to input data can automatically identify and quantify risk factors in transactions, improving the efficiency and accuracy of risk assessment.

[0199] Real-time early warning mechanism: When the risk score exceeds the preset threshold, an early warning information is automatically generated and sent to the relevant regulatory departments and transaction participants, so that all parties can take timely measures to reduce risk losses.

[0200] Blockchain record: Recording early warning information on the blockchain ensures the authenticity and non-tamperability of early warning information, providing reliable evidence for subsequent audits and investigations.

[0201] Through the above steps, real-time monitoring and early warning of transaction risks are achieved, improving the security and transparency of transactions and providing strong technical support for financial institutions and regulatory authorities.

[0202] The embodiment also provides a commodity transaction process monitoring system, which comprises:

[0203] Data acquisition module: Deploy multiple sensors at the transaction site to collect image, voice and motion data, and preprocess to obtain multi-modal data.

[0204] Data fusion module: Fuse the preprocessed multi-modal data, extract features of transaction behavior, commodity information and voice instructions, and generate structured transaction records.

[0205] Blockchain storage module: upload structured transaction records to the blockchain network, store them using the tamper-proof nature of the blockchain, and verify them through smart contracts.

[0206] Real-time monitoring module: the system reads transaction data on the blockchain in real time, combines preset rules and smart contracts for real-time monitoring, and triggers an alarm when abnormal behavior is detected.

[0207] Abnormal analysis module: after receiving abnormal transaction information, use the traceability function of the blockchain to conduct in-depth analysis of abnormal transactions and trace the transaction process.

[0208] Risk assessment module: based on the results of abnormal behavior analysis, combined with historical data and market conditions, use machine learning algorithms to assess transaction risks and send an early warning when the risk score exceeds the threshold.

[0209] The embodiment also provides a computer device suitable for the monitoring method of the commodity transaction process, which includes a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions to realize the monitoring method of the commodity transaction process proposed in the above embodiment.

[0210] The computer device can be a terminal, which includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be achieved through WIFI, operator network, NFC (near field communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device. It can also be an external keyboard, touchpad or mouse, etc.

[0211] The embodiment also provides a storage medium having a computer program stored thereon, the program being executed by a processor to implement the monitoring method for realizing the commodity transaction process proposed in the above embodiment; the storage medium can be realized by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic storage, a flash memory, a magnetic disk, or an optical disk.

[0212] In summary, the present application significantly improves the transparency, security and efficiency of commodity transaction monitoring by integrating multi-modal data fusion and blockchain technology. First, the multi-modal data fusion technology can comprehensively capture image, voice and action information in the transaction process, generate structured transaction records, and effectively solve the monitoring blind area problem caused by single data source in the existing monitoring system. Second, the application of blockchain technology ensures the non-tamperability and transparency of transaction data, and enhances the credibility of the monitoring system by storing data in a distributed ledger. In addition, the present application combines machine learning algorithms to conduct real-time assessment and early warning of transaction risks, which can quickly identify abnormal transaction behavior and issue warnings, effectively reducing transaction risks. This innovative monitoring method not only improves the comprehensiveness and accuracy of monitoring, but also provides real-time and reliable monitoring tools for regulatory authorities and transaction participants, promoting the healthy development of commodity transaction markets.

[0213] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, which should be covered by the scope of the claims of the present application.

Claims

1. A method for monitoring a commodity transaction process, characterized in that, Includes the following steps: Multiple sensors are deployed in the trading venue to collect image, voice and motion data, and preprocess them to obtain multimodal data; The preprocessed multimodal data is fused to extract features of transaction behavior, product information and voice commands, and generate structured transaction records. Structured transaction records are uploaded to the blockchain network, stored using the immutability of the blockchain, and verified through smart contracts. The monitoring system reads transaction data on the blockchain in real time, combines preset rules and smart contracts to perform real-time monitoring, and triggers an alarm when abnormal behavior is detected. Upon receiving abnormal transaction information, the traceability function of blockchain is used to conduct in-depth analysis of the abnormal transactions and track the transaction process. Based on the results of abnormal behavior analysis, combined with historical data and market conditions, machine learning algorithms are used to assess transaction risks, and warnings are sent when the risk score exceeds a threshold.

2. The method for monitoring the commodity transaction process as described in claim 1, characterized in that: The preprocessing of the image data includes, Use Gaussian filtering to reduce noise in the image; Enhance the edges of the image to highlight the product outline; Adjust the image's contrast and brightness to adapt to different lighting conditions; Converting the processed image data into grayscale reduces the amount of data and improves processing speed.

3. The method for monitoring the commodity transaction process as described in claim 2, characterized in that: The multimodal data fusion processing includes, Convolutional neural networks (CNNs) are used to extract product features and human action features from image data; Semantic analysis of speech-text data is performed using recurrent neural networks (RNNs) to extract key instructions and dialogue content; Image features, speech features, and action features are weighted and fused to generate a comprehensive feature vector. Structured transaction records are generated based on comprehensive feature vectors, recording transaction behavior, product information, and voice commands.

4. The method for monitoring the commodity transaction process as described in claim 3, characterized in that: The on-chain data includes, Structured transaction records are converted into hash values ​​to ensure the uniqueness and immutability of the data; The hash value and transaction record metadata are packaged into blocks and added to the distributed ledger of the blockchain; The integrity and legality of blocks are verified through smart contracts to ensure the authenticity and validity of data. The verified block is broadcast to all nodes in the blockchain network, enabling distributed data storage.

5. The method for monitoring the commodity transaction process as described in claim 4, characterized in that: The real-time monitoring and anomaly detection include, Set thresholds for monitoring metrics such as transaction amount, product movement speed, and voice command frequency; Real-time reading of transaction data on the blockchain and calculation of real-time values ​​of monitoring indicators; When the real-time value exceeds the preset threshold, an alarm mechanism is triggered, and the timestamp, location, and type of the abnormal behavior are recorded. Information on abnormal behavior is sent to the monitoring center, and detailed information about the abnormal behavior is recorded on the blockchain.

6. The method for monitoring the commodity transaction process as described in claim 5, characterized in that: The abnormal behavior analysis and tracing includes, Based on the timestamps and location information of abnormal behavior, the complete transaction process can be traced back from the blockchain; Analyze the flow of goods, changes in personnel operations and voice commands during the transaction process to determine the specific manifestations of abnormal behavior; By combining multimodal data, a visual report of abnormal behavior is generated, showing the process and scope of the abnormal behavior. The analysis results are stored on the blockchain to provide a basis for subsequent risk assessment and decision-making.

7. The method for monitoring the commodity transaction process as described in claim 6, characterized in that: The risk assessment and early warning include, Build a risk assessment model by inputting abnormal behavior analysis results, historical transaction data, and market volatility. Machine learning algorithms are used to extract features and assign weights to the input data to calculate a transaction risk score. When the risk score exceeds a preset threshold, an early warning message is generated, including the risk level, risk description, and recommended measures. The warning information will be sent to relevant regulatory authorities and transaction participants, and the sending time and recipient of the warning information will be recorded on the blockchain.

8. A monitoring system for a commodity transaction process, based on the monitoring method for a commodity transaction process according to any one of claims 1 to 7, characterized in that: include, Data acquisition module: Deploy various sensors in the trading venue to collect image, voice and motion data, and preprocess them to obtain multimodal data. Data fusion module: It fuses the preprocessed multimodal data, extracts features of transaction behavior, product information and voice commands, and generates structured transaction records. Blockchain storage module: Uploads structured transaction records to the blockchain network, stores them using the immutability of the blockchain, and verifies them through smart contracts. Real-time monitoring module: The monitoring system reads transaction data on the blockchain in real time, combines preset rules and smart contracts to perform real-time monitoring, and triggers alarms when abnormal behavior is detected. Anomaly Analysis Module: Upon receiving abnormal transaction information, the module utilizes the traceability function of blockchain to conduct in-depth analysis of the abnormal transactions and track the transaction process. Risk assessment module: Based on the results of abnormal behavior analysis, combined with historical data and market conditions, it uses machine learning algorithms to assess transaction risks and sends an alert when the risk score exceeds a threshold.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the monitoring method for the commodity transaction process according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the monitoring method for the commodity transaction process according to any one of claims 1 to 7.

Images