Digital space bid evaluation process witness service method and system based on time sequence model

Abstract

The invention provides a time sequence model-based digital space bid evaluation process witness service method and system, relates to the field of digital space bid evaluation, and solves the technical problem that fine-grained, real-time, structured and judicial credible process witness cannot be realized in the whole bid evaluation process on the premise of not storing a complete video in the prior art. The method comprises the following steps: carrying out inter-frame differential processing on a live video stream to obtain a frame to be analyzed; inputting the to-be-analyzed frame into a plurality of identification modules; each identification module outputs abnormal information based on a preset bid evaluation service rule; inputting the abnormal information and the user operation log into a multi-modal large model, and outputting a comprehensive confidence coefficient; determining a target key frame based on the comprehensive confidence; calculating a hash value of the target key frame, combining the hash value, the alarm record and the timestamp into a triple, and uploading the triple to a judicial block chain for uplink evidence storage; and after returning the transaction hash value, the judicial block chain executes a storage operation. The method and device are used in the bid evaluation process.

Description

Technical Field

[0001] This application relates to the field of digital space bidding, and in particular to a method and system for witnessing the digital space bidding process based on a time-series model. Background Technology

[0002] Electronic bidding has completely replaced paper-based bidding evaluation. Current systems generally use a method of "snapshot of key nodes + PDF signature + hash on-chain" to store the bidding evaluation results. However, this model only records discrete static results and cannot capture the operational behavior of the evaluators, interface changes and environmental conditions during the bidding process, resulting in a lack of continuous, fine-grained and traceable time-series evidence chain for post-event auditing.

[0003] Existing electronic bidding systems lack the capability to perceive, structurally fuse, and legally solidify multimodal behaviors throughout the entire bidding process, including visual changes, operation logs, and audio, without relying on complete video storage. This results in the inability to simultaneously capture a complete chain of evidence—from "violation of rules to operational context to system response"—at a second-level granularity, making it difficult to achieve traceable, tamper-proof, and directly usable process witnessing for judicial purposes. Therefore, this application provides a digital space bidding process witnessing service method and system based on a time-series model. Summary of the Invention

[0004] This application provides a digital space bidding process witnessing service method and system based on a time-series model, which solves the technical problem that existing technologies cannot achieve fine-grained, real-time, structured and judicially credible process witnessing of the entire bidding process without saving complete video.

[0005] To achieve the above objectives, this application adopts the following technical solution: Firstly, it provides a method for witnessing the digital spatial bidding process based on a time-series model, including: Real-time collection of live video streams, audio streams, and user operation logs from the bid evaluation terminal at the bid evaluation site; Perform inter-frame differential processing on the live video stream to obtain the frames to be analyzed; The frame to be analyzed is input into several recognition modules; each recognition module outputs abnormal information based on preset bidding business rules; among them, the several recognition modules include an identity recognition module, an action recognition module, an item recognition module, an OCR recognition module, and a speech recognition module, and the audio stream of the frame to be analyzed is input into the speech recognition module; Input the abnormal information and user operation logs into the multimodal large model and output the comprehensive confidence score; if the comprehensive confidence score exceeds the preset threshold, trigger an alarm and lock the corresponding analysis frame and mark it as the target key frame; Before storing the target keyframe, calculate the hash value of the target keyframe, and combine the hash value, alarm record and timestamp into a triple, and upload it to the judicial blockchain for on-chain evidence storage; After receiving the transaction hash value returned by the judicial blockchain, the target keyframe, alarm records and associated user operation logs are stored. Based on the triples corresponding to all target keyframes that have been uploaded to the blockchain and the associated user operation logs, the integrity of the evidence in the bidding process is verified, and a witness report is generated.

[0006] Based on the above technical solution, the digital space bidding process witnessing service method based on a time-series model provided in this application implements real-time, accurate, and judicially credible intelligent supervision of the entire bidding process. By synchronously collecting video, audio, and terminal operation logs from the bidding site, and combining multimodal perception capabilities such as identity recognition, action analysis, prohibited item detection, voice keyword recognition, and OCR numerical extraction, it accurately captures typical violations such as external bid imports, unauthorized communication, and judges leaving their posts without authorization. Utilizing perceptual hashing and inter-frame differential technology, it retains only key frames where the image undergoes substantial changes, significantly reducing invalid data and decreasing the storage resources required for a single bidding process. Once a high-confidence violation is determined, the system immediately calculates the key frame hash and submits the triple consisting of the hash value, alarm record, and timestamp to the judicial blockchain. Local storage is only allowed after receiving on-chain transaction confirmation, achieving a strong consistency guarantee of "evidence not solidified is not stored." The multimodal large-scale model employed is trained through semantic alignment between the bidding rule text and historical evaluation interfaces. During inference, it directly outputs a structured description containing the violation type, object, value, action, and time, eliminating the need for manual interpretation and significantly improving review efficiency while controlling the false positive rate. The system also possesses self-optimization capabilities, automatically distinguishing between positive and negative samples by validating the model output through a rule engine. After each day's tasks, it performs lightweight incremental fine-tuning of the model's classification head, and manages historical negative samples using a time-sliding window, ensuring the model continuously adapts to evolving business rules. This solution constructs an efficient, lightweight, and non-repudiable end-to-end digital witnessing mechanism without recording complete video or relying on underlying system modifications, effectively supporting the fairness and transparency of bidding activities.

[0007] In conjunction with the first aspect above, in one possible implementation, obtaining the frame to be analyzed includes: The perceptual hash values ​​of the current frame and the previous frame in the live video stream are obtained respectively, and the Hamming distance between them is calculated. If the Hamming distance is greater than or equal to a preset distance threshold, the current frame is marked as a frame to be analyzed; otherwise, it is not marked.

[0008] In conjunction with the first aspect above, in one possible implementation, inputting the exception information and user operation logs into the multimodal large model includes: Before the abnormal signals output by each identification module are input into the multimodal large model, a timing consistency check based on operational causal logic is performed: Based on the user operation log and the timestamps output by each identification module, determine whether the order of abnormal information conforms to a reasonable causal chain of human operation; The abnormal information is input into the multimodal large model only when the abnormal signal satisfies a preset causal dependency relationship in the time dimension.

[0009] In conjunction with the first aspect above, in one possible implementation, the training process of the multimodal large model includes: Obtain the scoring rules and corresponding historical evaluation interface screenshots from the publicly tendered documents, construct a pre-training dataset for image-text semantic alignment, and pre-train a basic multimodal artificial intelligence model based on the pre-training dataset; The pre-trained model is fine-tuned based on manually labeled training samples to obtain the multimodal large model; Each training sample includes a frame to be analyzed, a user operation log time-aligned with the frame to be analyzed, and a corresponding structured violation description. The structured violation description contains five fields: violation type, object, value, action, and time. These fields are used to guide the model to fuse the structured abnormal signals output by each identification module during inference and generate a structured violation judgment with comprehensive confidence.

[0010] In conjunction with the first aspect above, in one possible implementation, the fine-tuning of the pre-trained model based on manually labeled training samples includes: When the rule engine determines that the structured violation description output by the multimodal large model is a false alarm, it marks the corresponding input training sample as a negative training sample to obtain a negative training sample set; otherwise, it marks the input training sample as a positive training sample to obtain a positive training sample set. After the evaluation task is completed, incremental fine-tuning is performed on the classification head of the multimodal large model based on the positive training sample set and the negative training sample set; The negative training sample set employs a time-sliding window strategy, retaining historical negative training samples within a preset time window.

[0011] In conjunction with the first aspect above, in one possible implementation, the identity recognition module performs face detection and feature extraction on the frame to be analyzed to obtain a face feature vector; based on a preset judge registration database and a cosine similarity threshold, it determines whether the current person is a registered judge and is on duty; if yes, it outputs the identity matching result and the on-duty status; if no, it does not output any information. The action recognition module is used to determine whether the action appearing in the frame to be analyzed is abnormal; if yes, it outputs the action category and duration; otherwise, it does not output any information. The object recognition module performs target detection on the frame to be analyzed to obtain the category of the object in the image; based on a preset list of prohibited items, it obtains the category of prohibited items. The speech recognition module performs text conversion on the audio stream that is time-aligned with the frame to be analyzed to obtain text segments; it determines whether there are preset illegal keywords in the text segments, and if so, marks the audio stream as an abnormal audio stream; The OCR recognition module performs optical character recognition on the frame to be analyzed to obtain the original text block; based on the preset numerical regular expression, it filters out the numerical text related to the bidding business rules from the original text block and converts it into structured fields.

[0012] In conjunction with the first aspect above, in one possible implementation, determining whether an action appearing in the frame to be analyzed is abnormal includes: The action recognition module performs human key point detection on multiple consecutive frames to be analyzed; it obtains spatiotemporal features by performing temporal modeling on the human key points, the spatiotemporal features including the time sequence of posture parameters, action persistence, rate of change or steady state features; when the spatiotemporal features match a preset evaluation violation pattern, it outputs an action abnormality signal.

[0013] In conjunction with the first aspect above, in one possible implementation, the verification of the integrity of evidence in the bid evaluation process includes: Extract the triples corresponding to all target keyframes in the chain; Calculate the timestamp interval between adjacent triples. If there is a timestamp interval greater than a preset interval threshold and no corresponding operation log record, then mark the interval corresponding to the timestamp interval as an evidence break zone.

[0014] In conjunction with the first aspect above, in one possible implementation, when the target keyframe is stored, its filename is uniquely determined by the transaction hash value returned by the judicial blockchain; the associated alarm records and user operation logs are stored in a structured database with the same transaction hash value as the primary key. The transaction hash value, target keyframe, alarm record, and user operation log together constitute an indivisible evidence unit. If any component is missing or the hash does not match, the corresponding evidence unit will be marked as invalid.

[0015] Secondly, this application provides a digital spatial bidding process witnessing service system based on a time-series model, including: an analysis module and a storage module; wherein, the analysis module is used to collect live video streams, audio streams, and user operation logs of the bidding site in real time; perform inter-frame differential processing on the live video streams to obtain frames to be analyzed; input the frames to be analyzed into several recognition modules; each recognition module outputs abnormal information based on preset bidding business rules; wherein, the several recognition modules include an identity recognition module, an action recognition module, an item recognition module, an OCR recognition module, and a speech recognition module, and the audio stream of the frames to be analyzed is input into the speech recognition module; the abnormal information and user... The operation log is input into the multimodal large model, and the overall confidence score is output. If the overall confidence score exceeds a preset threshold, an alarm is triggered and the corresponding analysis frame is locked and marked as the target key frame. The storage module is used to calculate the hash value of the target key frame before storing it, and combine the hash value, alarm record and timestamp into a triple, and upload it to the judicial blockchain for on-chain evidence storage. After receiving the transaction hash value returned by the judicial blockchain, the target key frame, alarm record and associated user operation log are stored. Based on the triple corresponding to all on-chain target key frames and the associated user operation log, the integrity of the evidence in the bidding process is verified and a witness report is generated.

[0016] This application provides a digital spatial bidding process witnessing service method and system based on a time-series model, enabling real-time, accurate, and traceable supervision of the entire bidding process. By integrating live video streams, audio streams, and user operation logs, a multimodal large model is used to jointly identify identity, actions, items, voice, and OCR information, effectively recognizing high-risk behaviors such as bid rigging, external information importation, and judges leaving their posts. An inter-frame differential mechanism is employed to filter key change frames, avoiding storage and computational redundancy caused by full recording, reducing the amount of evidence data per bidding session by two orders of magnitude. After a violation is triggered, a "first on-chain, then on disk" evidence storage strategy is executed, writing the target key frame hash, alarm record, and timestamp triplet into the judicial blockchain in real time, ensuring that evidence is solidified upon generation and the process is tamper-proof. The multimodal large model is pre-trained based on the semantics of bidding documents and historical bidding data, and outputs structured violation descriptions, including type, object, value, action, and time fields, directly meeting the requirements of judicial evidence format, improving review efficiency by more than ten times. The system incorporates a rule engine feedback mechanism, automatically using false positive samples for incremental model fine-tuning. Combined with a sliding window to manage negative samples, this enables continuous model self-evolution, adapting to dynamic updates in the bidding evaluation rules. The overall solution constructs a lightweight, efficient, and legally credible intelligent witnessing system without saving complete video footage or intruding on the operating system, significantly improving the fairness and compliance of bidding processes.

[0017] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0018] Figure 1 A layered architecture diagram of a digital spatial bidding process witnessing service system based on a time-series model provided in this application embodiment; Figure 2 A flowchart illustrating a digital spatial bidding process witnessing service method based on a time-series model, provided for an embodiment of this application; Figure 3 This is a schematic diagram of the training process of a multimodal large model provided in an embodiment of this application. Detailed Implementation

[0019] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The digital spatial bidding process witnessing service method and system based on a time-series model provided in this application adopts a layered architecture design, such as... Figure 1 As shown, it mainly includes a data acquisition layer, a data access and preprocessing layer, a capability layer, a business layer, and a presentation layer. Each layer works together to achieve real-time monitoring, intelligent analysis, and judicial-grade evidence preservation of the entire bidding process.

[0021] Acquisition Layer (Edge Access): Responsible for acquiring raw data. Multiple high-definition USB cameras are deployed in each evaluation room to capture live video from the judges, outputting a 1080p resolution, 25fps live video stream. Simultaneously, the screen images of the judges' terminals are captured via an HDMI-USB capture stick. Both video streams are pushed to edge nodes deployed locally in the evaluation room using the RTMP protocol. Furthermore, a log acquisition module is embedded in the evaluation client software to monitor and record key user operation events in real time, including button clicks, cell content modifications, copy and paste operations, and window switching. To ensure time consistency of multi-source data, the edge nodes are automatically calibrated daily with a BeiDou high-precision time source, ensuring that the timestamp error between the video frames and the user operation logs is less than 50 milliseconds.

[0022] Data Access and Preprocessing Layer: This layer filters, aligns, and structures the raw data from the acquisition layer. It uses tools like FFmpeg to extract frames from the video stream and calculates the perceptual hash value between adjacent frames. If the difference in perceptual hash between two frames is below a preset threshold, the frame is considered a still image and discarded. This compresses the massive amount of raw frames generated during long-term evaluation (e.g., approximately 80,000 frames over 8 hours) into thousands of keyframes containing valid changes. Simultaneously, this layer uses a 1-second time window to attach all user operation log events captured within the same time period to the corresponding video frame metadata, forming paired data packets of "image + operation," providing structured input for subsequent multimodal large-scale model fusion analysis.

[0023] The Capability Layer is the core intelligent engine of the system, responsible for executing various AI inference tasks. This layer first feeds the pre-processed "image + operation" pairing package into a multimodal large-scale model for comprehensive analysis. This multimodal large-scale model is specially trained to integrate multi-dimensional information such as vision, text, and behavior, outputting structured violation judgment results. In addition, the Capability Layer also runs multiple dedicated recognition modules in parallel, including: a face recognition module (for verifying the identity and on-duty status of judges), an action recognition module (for detecting abnormal behaviors such as looking down at a phone or talking sideways), an object recognition module (for identifying prohibited items such as mobile phones and USB drives), an OCR recognition module (for extracting key values ​​such as price and amount from the image), and a speech recognition module (for capturing sensitive keywords such as "quotation" and "price"). The output of all modules is structured data and can be directly called by the multimodal large-scale model. It is worth mentioning that rules for new violation types can be implemented by updating the prompts in the large-scale model, without retraining the entire model, thus shortening the rule iteration cycle from the "day / week" level to the "hour" level.

[0024] The business layer is responsible for transforming the analysis results from the capability layer into auditable and tamper-proof judicial evidence and providing business management functions. Its core process is a "chain-first, disk-later" evidence storage mechanism: when a multimodal large model determines a violation and triggers an alarm, the system immediately captures the corresponding keyframe, calculates its SHA-256 hash value, and submits the triple consisting of the hash value, alarm record, and timestamp to the judicial blockchain. Only after successfully obtaining the transaction hash returned by the blockchain is the keyframe file stored on disk in a read-only directory. The entire process takes less than 200 milliseconds, ensuring that the evidence is "authentic from generation and tamper-proof in the process." The business layer also integrates modules for expert user management, algorithm management, bidding room management, early warning record management, and terminal device management. After the bidding process, it automatically aggregates all on-chain keyframes and their transaction information, generating a lightweight (less than 2MB) PDF witness report. Each page of the report contains one keyframe, the corresponding violation explanation, and a blockchain QR code that can be scanned for verification.

[0025] Presentation Layer: This layer provides a human-computer interaction interface and real-time visualization capabilities. Through a web page, regulators can view a real-time "alarm screenshot wall." The system pushes newly generated alarm frames to the front end instantly via the WebSocket protocol. Users can click on any alarm screenshot to view the original image details and the corresponding on-chain verification results within 5 seconds, achieving "what you see is what you get."

[0026] Through the aforementioned layered architecture and collaborative working mechanism, this application can achieve judicial-grade, traceable, and highly efficient intelligent witnessing of the entire bidding process without touching the underlying operating system or saving the complete long video, using only the triplet of "live keyframes + user operation logs + blockchain hash".

[0027] To address the technical problem that existing technologies cannot provide fine-grained, real-time, structured, and judicially credible process witnessing for the entire bid evaluation process without saving complete video footage, this application provides a digital spatial bid evaluation process witnessing service method based on a time-series model. This method includes: Real-time collection of live video streams, audio streams, and user operation logs from the bid evaluation terminal at the bid evaluation site; Perform inter-frame differential processing on the live video stream to obtain the frames to be analyzed; The frame to be analyzed is input into several recognition modules; each recognition module outputs abnormal information based on preset bidding business rules; among them, the several recognition modules include an identity recognition module, an action recognition module, an item recognition module, an OCR recognition module, and a speech recognition module, and the audio stream of the frame to be analyzed is input into the speech recognition module; Input the abnormal information and user operation logs into the multimodal large model and output the comprehensive confidence score; if the comprehensive confidence score exceeds the preset threshold, trigger an alarm and lock the corresponding analysis frame and mark it as the target key frame; Before storing the target keyframe, calculate the hash value of the target keyframe, and combine the hash value, alarm record and timestamp into a triple, and upload it to the judicial blockchain for on-chain evidence storage; After receiving the transaction hash value returned by the judicial blockchain, the target keyframe, alarm records and associated user operation logs are stored. Based on the triples corresponding to all target keyframes that have been uploaded to the blockchain and the associated user operation logs, the integrity of the evidence in the bidding process is verified, and a witness report is generated.

[0028] Based on this, real-time and proactive monitoring of the bidding process is achieved. Traditional methods rely on reviewing lengthy recordings after the fact, which is slow and inefficient. This application, however, uses a multimodal AI model to fuse and analyze video, audio, and operation logs, triggering alarms and solidifying evidence within seconds of a violation occurring, completely solving the problems of "late detection and difficulty in obtaining evidence." It significantly reduces storage and computational overhead, improving system efficiency. By performing inter-frame differential analysis (such as perceptual hashing + Hamming distance) on the live stream to filter out frames that truly contain changes for analysis, and combining this with a dual-gating mechanism of "only storing evidence with high confidence," only the minimum set of key frames with judicial value is preserved. This reduces the storage size of a single bidding session from tens of GB to hundreds of MB, saving over 95% of resources, while also preventing invalid data from interfering with the analysis. It ensures the authenticity, integrity, and immutability of evidence. Employing a "first on-chain, then on-disk" mechanism, the hash value of the target keyframe is stored on the judicial blockchain before being written locally, forming a strong association of "screenshot-alarm-hash" triples. Any subsequent tampering can be quickly verified, meeting the strict judicial requirement that electronic evidence be "fixed upon generation." It outputs structured, interpretable, and directly demonstrable descriptions of violations.

[0029] like Figure 2 As shown in the embodiments of this application, the digital spatial bidding process witnessing service method based on a time-series model includes: S201. Real-time collection of live video streams, audio streams, and user operation logs from the bid evaluation site.

[0030] S202. Perform inter-frame differential processing on the live video stream to obtain the frames to be analyzed.

[0031] S203. Input the frame to be analyzed into several identification modules; each identification module outputs abnormal information based on the preset bidding business rules.

[0032] Among them, several recognition modules include an identity recognition module, an action recognition module, an object recognition module, an OCR recognition module, and a speech recognition module, with the audio stream of the frame to be analyzed input to the speech recognition module.

[0033] S204. Input the abnormal information and user operation log into the multimodal large model and output the comprehensive confidence score. If the comprehensive confidence score exceeds the preset threshold, trigger an alarm and lock the corresponding analysis frame, and mark it as the target key frame.

[0034] It should be noted that the abnormal signals output by each identification module undergo a timing consistency check based on operational causal logic before being input into the multimodal large model: Based on the user operation logs and the timestamps output by each identification module, determine whether the order of abnormal information conforms to a reasonable causal chain of human operation; The abnormal information is input into the multimodal large model only when the abnormal signal satisfies the preset causal dependency relationship in the time dimension.

[0035] S205. Before storing the target keyframe, calculate the hash value of the target keyframe, and combine the hash value, alarm record and timestamp into a triple, and upload it to the judicial blockchain for on-chain evidence storage; after receiving the transaction hash value returned by the judicial blockchain, store the target keyframe, alarm record and associated user operation log. S206. Based on the triples corresponding to all target keyframes that have been put on the chain and the associated user operation logs, perform integrity verification on the evidence of the bidding process and generate a witness report.

[0036] The completeness verification of evidence in the bid evaluation process is as follows: Extract the triples corresponding to all target keyframes in the chain; Calculate the timestamp interval between adjacent triples. If there is a timestamp interval greater than a preset interval threshold and no corresponding operation log record, then mark the interval corresponding to the timestamp interval as an evidence break zone.

[0037] Marking areas of evidence gaps in the witness report and indicating that there is no valid evidence of violations during that period does not imply compliance with regulations. It should be noted that the triple hash chain constitutes an immutable time anchor sequence, which is used to reverse verify the integrity of locally stored data.

[0038] When storing the target keyframe, its filename is uniquely determined by the transaction hash value returned by the judicial blockchain; the associated alarm records and user operation logs are stored in the structured database with the same transaction hash value as the primary key. Furthermore, the transaction hash value, target keyframe, alarm record, and user operation log together constitute an indivisible evidence unit. If any component is missing or the hash does not match, the corresponding evidence unit will be marked as invalid.

[0039] Based on the above technical solutions, the digital space bidding process witnessing service method based on a time-series model provided in this application, through real-time acquisition of video streams and user operation logs, adopts a dual-gating mechanism of "edge difference + large model confidence" to trigger keyframe screenshots only when there are significant changes in the image and the AI ​​determines a high risk, and uploads the screenshots to the blockchain for evidence storage within seconds, realizing a closed loop of "log anomaly → AI alarm → frame-level evidence". Its core lies in constructing a multimodal large model aligned with bidding business: using bidding documents and historical screenshots for pre-training with images and text, and then fine-tuning with structured violation descriptions, enabling the model to understand business semantics such as "copying external quotations"; simultaneously, a rule engine feedback mechanism is introduced to automatically use false alarm samples for incremental fine-tuning, achieving model self-evolution. This solution completely abandons the "full recording + post-event review" model, saving only the smallest keyframes with judicial value, reducing storage costs, improving review efficiency, and lowering the false alarm rate, truly achieving real-time, accurate, tamper-proof, and judicially credible intelligent witnessing of the bidding process, effectively ensuring the fairness and impartiality of bidding. This addresses the serious problems in traditional electronic bidding supervision, which relies heavily on manual review of complete video recordings after the fact, leading to delayed response, difficulty in locating evidence, and high rates of missed detection. Furthermore, in cases of bid rigging or collusion, it is often difficult to hold perpetrators accountable due to the inability to obtain evidence in a timely manner or incomplete evidence chains, failing to meet the judicial requirements of "traceable process, verifiable behavior, and credible results."

[0040] In one possible implementation of this application embodiment, the above-mentioned S202 can be specifically described as follows: The perceptual hash values ​​of the current frame and the previous frame in the live video stream are obtained respectively, and the Hamming distance between them is calculated. If the Hamming distance is greater than or equal to a preset distance threshold, the current frame is marked as a frame to be analyzed; otherwise, it is not marked.

[0041] It should be noted that perceptual hashing calculates each frame in the live video stream, including the current frame and the previous frame, by first scaling it to a uniform size (e.g., 32×32 pixels), converting it to grayscale, then extracting low-frequency information through discrete cosine transform (DCT), and finally generating a 64-bit binary hash value. This hash value is sensitive to image content but robust to minor perturbations such as changes in lighting and compression noise.

[0042] Hamming distance calculation: The Hamming distance is calculated by comparing the two 64-bit perceptual hash values ​​of the current frame and the previous frame bit by bit and counting the number of different bits. This distance reflects the degree of difference in visual content between the two frames.

[0043] If the Hamming distance is greater than or equal to the preset distance threshold, it indicates that the image has changed significantly, such as window switching, personnel movement, or the appearance of new values. In this case, the current frame is marked as the frame to be analyzed and enters the subsequent multimodal recognition process. If the Hamming distance is less than the preset distance threshold, it means that the image is basically still or has only slight fluctuations, such as screen refresh or light flicker. In this case, it will not be marked and will be discarded directly to avoid invalid calculations.

[0044] An example scenario: An expert switches from the bidding system interface to an Excel file to view external quotations.

[0045] Previous frame: Displays the evaluation scoring sheet (pHash=a1b2c3d4...); Current frame: Displays an Excel window containing "25.8 million yuan" (pHash=f9e8d7c6...). The Hamming distance is calculated to be 22. If the preset threshold is 12, that is, 22>12 → mark it as a frame to be analyzed. Subsequently, the OCR module extracts "25.8 million yuan", the item module detects the Excel window, and triggers a violation alarm.

[0046] Based on the above technical solutions, in the scenario of witnessing the entire electronic bidding process, it is necessary to perform real-time analysis of high-definition video streams lasting several hours. However, the original video contains a large number of static or redundant frames (such as scenes of experts thinking). Processing the entire stream would lead to a serious waste of computing resources, a surge in storage costs, and may reduce the accuracy of AI model discrimination due to interference from invalid signals. Therefore, there is an urgent need for an efficient mechanism to filter out the key frames that truly contain operational behavior. This application adopts an inter-frame difference technique based on perceptual hashing and Hamming distance, which is designed to solve the above problems. Perceptual hashing extracts low-frequency structural information of the image through DCT to generate a 64-bit fingerprint that is robust to illumination and compression noise, effectively ignoring irrelevant disturbances; Hamming distance quantifies the essential differences in visual content between two frames. This method only retains frames that have undergone significant changes (such as window switching, personnel actions, and numerical input), greatly reducing the amount of data to be processed (the measured compression rate is over 95%), and significantly reducing the computing power and storage pressure on edge devices. Meanwhile, by filtering out high-frequency noise such as screen refresh and light flicker, only high-value samples are fed into the multimodal large model, effectively improving the signal-to-noise ratio and accuracy of subsequent violation identification. Compared with traditional pixel difference or optical flow methods, this solution is computationally lightweight and has strong anti-interference capabilities, perfectly meeting the core requirements of "high efficiency, accuracy, and low overhead" in bidding evaluation scenarios.

[0047] In one possible implementation of this application embodiment, the above-mentioned S203 can be specifically described as follows: The identity recognition module performs face detection and feature extraction on the frame to be analyzed to obtain the face feature vector; based on the preset judge registration library and cosine similarity threshold, it determines whether the current person is a registered judge and is on duty; if yes, it outputs the identity matching result and on-duty status; otherwise, it does not output any information.

[0048] It should be noted that the identity recognition module performs face detection on each frame of the image to be analyzed, such as MTCNN or YOLOv5-face, and extracts a 128-dimensional face feature vector, such as the output of FaceNet. It then calculates the cosine similarity between the vector and the judge registration database (pre-entered expert face templates). If the highest similarity is greater than or equal to a preset threshold (such as 0.85), the registered judge is determined to be on duty, and the following is output: {"identity":"Zhang San","in_seat":true}. Otherwise, no information is output (i.e., silent processing to avoid false alarms of "stranger").

[0049] Example scenario: Evaluation begins, expert "Li Si" takes his seat. Process: The system detects a face, and the feature vector matches "Li Si" in the registry (similarity 0.92 > 0.85). Output: {"identity":"Li Si","in_seat":true} → If he / she leaves the seat for an extended period, a "departure from duty" alarm can be triggered. Counterexample: Cleaning staff briefly enter → no match → no output, does not interfere with the main process.

[0050] The action recognition module is used to determine whether the actions appearing in the frame to be analyzed are abnormal; if yes, it outputs the action category and duration; otherwise, it does not output any information. Specifically, the action recognition module performs human key point detection on multiple consecutive frames to be analyzed; it obtains spatiotemporal features by performing temporal modeling on the human key points, including the time series of posture parameters, action duration, rate of change, or features of steady state; when the spatiotemporal features match the preset evaluation violation pattern, it outputs an action abnormality signal.

[0051] It should be noted that for multiple consecutive frames, such as 5 frames to be analyzed, human key point detection is performed, such as MediaPipePose, to obtain the coordinates of key points such as head, shoulders, hands, and chest. Construct spatiotemporal features: time series of posture parameters, such as head pitch angle of [30°, 40°, 50°, 55°, 60°] per frame; action duration, such as head tilt lasting ≥3 seconds; rate of change, such as hand moving 50 pixels per second; steady state, such as angle fluctuation <5° for 5 consecutive frames.

[0052] If the feature combination matches a preset violation pattern, such as "looking down + hands not on the keyboard", then the output will be {"action_type":"viewing phone","duration_sec":4.2}.

[0053] Example scenario: An expert looks down at his phone under the table.

[0054] Test: Head tilt angle reaches 65°, right hand is away from the mouse area, for 4 seconds.

[0055] Matching rule: "Pitch angle > 45° + hands not in input area + duration > 3 seconds" → judged as "view external device".

[0056] Output: {"action_type":"View phone","duration_sec":4.2} → Trigger alarm.

[0057] Counterexample: Expert tying shoelaces (briefly lowers head but hands are on feet) → does not match the "hand position" condition → no output.

[0058] The object recognition module performs target detection on the frame to be analyzed to obtain the category of the object in the image; based on the preset list of prohibited items, it obtains the category of prohibited items.

[0059] It should be noted that object detection models, such as YOLOv8, are used to identify objects in the image and output the category, confidence score, and bounding box. The test results are compared with a pre-set list of prohibited items, such as: [mobile phones, USB flash drives, hard drives, tablets, walkie-talkies]; If an object category is in the list and the confidence level is greater than the threshold, then output its category.

[0060] Example scenario: A mobile phone appears on the expert's desktop.

[0061] Detection: YOLO outputs {"class":"cellphone","confidence":0.93}. List matching: "cellphone" ∈ prohibited list → outputs {"forbidden_item":"phone"}.

[0062] Counterexample: "Water Cup" detected → Not in list → No output.

[0063] The speech recognition module performs text conversion on the audio stream that is time-aligned with the frame to be analyzed to obtain text segments; it then determines whether there are preset illegal keywords in the text segments. If so, the audio stream is marked as an abnormal audio stream.

[0064] It should be noted that for audio streams aligned with video frame time, such as ±500ms windows, ASR transcription, such as the Whisper model, is performed to obtain text fragments. The system uses a pre-defined database of prohibited keywords for word-by-word matching, such as: ["quotation", "price", "lowest price", "bid rigging", "collusion"]. If any keyword is matched, the audio segment is marked as abnormal, and the time period is recorded.

[0065] Example scenario: An expert says, "This quote is clearly too low."

[0066] Transcription: Text = "This quote is obviously too low." Keyword matching: "quote" ∈ Keyword Library → Mark as an anomaly. Output: {"audio_anomaly":true,"keyword":"quote","timestamp":"14:23:05–14:23:07"}.

[0067] Counterexample: Discussion of "Technical Solution Evaluation Criteria" → No sensitive words → No output.

[0068] The OCR recognition module performs optical character recognition on the frame to be analyzed to obtain the original text block; based on the preset numerical regular expression, it filters out the numerical text related to the bidding business rules from the original text block and converts it into structured fields.

[0069] It should be noted that OCR is performed on the image, such as PaddleOCR, to obtain a list of original text blocks; then, preset numerical regular expressions are used to filter out numerical text related to the evaluation.

[0070] For example, in this scenario: an Excel window pops up on the screen, displaying "Total bid price: 925,000 yuan".

[0071] OCR output: ["Total Bid Price", "925,000 RMB"]

[0072] Regular expression filtering: "925,000 yuan" matches the price pattern → extract.

[0073] Structured output: {"field_type":"price","value":92.5,"unit":"ten thousand yuan"}.

[0074] Counterexample: The screen contains "Page 3" and "Serial Number 5" → does not match the numerical business rules → will be filtered.

[0075] Based on the above technical solution, each module adopts a silent mechanism of "output only when an anomaly is detected." This means that structured results are generated only when a clear violation signal is detected; otherwise, no information is output. This significantly reduces input noise to the subsequent multimodal large-scale model, avoids introducing a large number of normal behaviors into the judgment process, and significantly improves the accuracy and reliability of the overall confidence level. Each module is deeply coupled with the bidding business rules: identity recognition ensures "person-job consistency" to prevent proxy evaluation or absence from duty; action recognition focuses on typical violation postures such as "looking down at a mobile phone" and "talking sideways"; item recognition specifically inspects high-risk prohibited items such as "mobile phones and USB drives"; speech recognition accurately captures sensitive words such as "quotation" and "price"; and OCR specifically extracts key values ​​such as "amount" and "percentage." This domain-customized design enables the system to accurately hit regulatory points and avoid the generalization bias of general AI models. All outputs are structured data (such as fields, categories, and timestamps), which can be directly fused and reasoned by the multimodal large-scale model to ultimately generate a violation description conforming to the format of judicial evidence, without the need for post-processing parsing, achieving end-to-end automation of "perception-judgment-evidence."

[0076] In one possible implementation of the embodiments of this application, such as Figure 3 As shown, the above S203 can be implemented through the following S301, S302 and S303, which are explained in detail below: Constructing a large multimodal model; S301. Obtain the scoring rules and corresponding historical evaluation interface screenshots from the publicly tendered documents, and construct a pre-trained dataset for image-text semantic alignment. S302. Pre-train the basic multimodal artificial intelligence model based on the pre-trained dataset; Each training sample includes a frame to be analyzed, a user operation log time-aligned with the frame to be analyzed, and a corresponding structured violation description. The structured violation description contains five fields: violation type, object, value, action, and time. These fields are used to guide the model to fuse the structured abnormal signals output by each identification module during inference and generate a structured violation judgment with comprehensive confidence.

[0077] It should be noted that, in order to enable the artificial intelligence model to understand the semantics of the bidding business, such as "price score calculation error" and "external information import", this application constructed a semantically aligned graph-text training set: Data source: Text side: Automatically extract scoring rule clauses from publicly available tender documents (e.g., "The maximum technical score is 40 points, which is derived by weighting 5 sub-indicators"). Image side: Collect screenshots of real manual review interfaces during historical electronic bidding processes (such as expert scoring pages, quotation viewing windows, etc.).

[0078] Semantic alignment: Each screenshot is associated with its corresponding scoring rule clause through manual or semi-automatic methods (e.g., a screenshot shows "Price Score = 92.5", corresponding to the rule "Price Score = Full Score × (Lowest Price / Bid Price)").

[0079] Pre-training objective: To enable the model to learn the mapping relationship between "specific interface element + operation context → compliance".

[0080] The model was fine-tuned using high-quality, manually labeled training samples. Each training sample contained triples: Input: A frame (image) to be analyzed + corresponding user operation log (such as window switching, clipboard writing); Tags: Structured violation description, formatted as a JSON object containing five fields.

[0081] S303. Fine-tune the pre-trained model based on manually labeled training samples to obtain a multimodal large model.

[0082] S3031. When the rule engine determines that the structured violation description output by the multimodal large model is a false alarm, the corresponding input training sample is marked as a negative training sample to obtain a negative training sample set; otherwise, the input training sample is marked as a positive training sample to obtain a positive training sample set. S3032. After the evaluation task is completed, based on the positive training sample set and the negative training sample set, perform incremental fine-tuning on the classification head of the artificial intelligence model to obtain a multimodal large model. The negative training sample set employs a time-sliding window strategy, retaining historical negative training samples within a preset time window.

[0083] The classification head is a trainable parameter layer located after the backbone network in a multimodal large model. It is used to map fused features into structured violation judgment results and is usually composed of one or more fully connected neural network layers.

[0084] It should be noted that, in order to continuously improve the accuracy of the model and adapt to the evolution of evaluation behavior, this application introduces a closed-loop negative feedback mechanism: Sample labeling: When the rule engine (based on the logic verification module of the "Electronic Bidding Behavior Code") determines that the structured violation description output by the multimodal large model is a false alarm (such as misjudging "look up formula" as "view external quotation"), the input (frame + log) of this inference is labeled as a negative training sample. If the alarm is determined to be correct, it is marked as a positive training sample.

[0085] Incremental fine-tuning: After the daily evaluation task is completed, the system automatically aggregates all positive and negative training samples of the day; Lightweight fine-tuning is performed only on the classification head (i.e. the last fully connected layer) of the multimodal large model, while the backbone network parameters are frozen to avoid catastrophic forgetting. Negative samples are managed using a time-sliding window strategy (e.g., only retaining data from the most recent N days), which preserves recent false alarm patterns while preventing long-term interference from old noise.

[0086] For example, a false alarm scenario: an expert uses a "calculator" plugin to calculate the discount rate in the bidding evaluation system, and the model misjudges it as "using an external tool".

[0087] Rule engine verification: The current focus window is still the evaluation system, and there is no external process → it is judged as a false alarm; the system automatically adds the sample to the negative training sample set.

[0088] Subsequent fine-tuning: That evening, the system performed incremental training, and the model learned that: "Calculator plugin + windowless switching = compliant operation".

[0089] Result: This type of false alarm no longer occurs in subsequent bid evaluations. The model's F1-score improved daily (in actual testing, it rose from 0.78 to 0.92 within 7 days).

[0090] The sliding window function: If the evaluation system is upgraded in 3 months and a new "AI-assisted scoring" function is added, the old "AI=violation" negative samples will be automatically eliminated, and the model can quickly adapt to the new rules.

[0091] Based on the above technical solutions, in the scenario of intelligent witnessing of electronic bidding, the general multimodal large model, due to its lack of understanding of bidding business rules, is prone to misjudging normal operations (such as consulting internal formulas) as violations (such as importing external quotations), resulting in a high false alarm rate and difficulty in judicial acceptance. Meanwhile, bidding behavior patterns continuously evolve with policy and system updates, and static models cannot adapt to new scenarios, requiring frequent manual retraining, which is costly. Therefore, this application proposes a technical path of "business semantic pre-training + structured fine-tuning + rule feedback self-evolution." A text-image aligned training set is constructed using publicly available bidding documents and historical bidding screenshots, enabling the model to master core business semantics such as "price score calculation" and "independent review" during the pre-training stage. Fine-tuning is performed using structured labels containing five fields, forcing the model to output interpretable and verifiable judgment results. Most importantly, a rule engine is introduced as a "referee," automatically converting false alarm samples into negative samples and correct alarms into positive samples, and incrementally fine-tuning the classification head daily, achieving low-cost self-evolution of the model. Combined with a sliding window mechanism, recent biases are quickly corrected while avoiding the accumulation of old noise. This solution fundamentally addresses the three major bottlenecks of AI models: "lack of business understanding, unreliability, and lack of evolution." Without requiring extensive manual annotation, it continuously improves accuracy and compliance, meeting the stringent requirements of judicial-grade witnessing for reliability, interpretability, and long-term stability.

Claims

1. A witnessing service method for digital spatial bidding process based on a time-series model, characterized in that, include: Real-time collection of live video and audio streams from the bid evaluation site, as well as user operation logs from the bid evaluation terminal; Perform inter-frame differential processing on the live video stream to obtain the frames to be analyzed; The frame to be analyzed is input into several recognition modules; each recognition module outputs abnormal information based on preset bidding business rules; among them, the several recognition modules include an identity recognition module, an action recognition module, an item recognition module, an OCR recognition module, and a speech recognition module, and the audio stream of the frame to be analyzed is input into the speech recognition module; Input the anomaly information and user operation logs into the multimodal large model and output the comprehensive confidence score; if the comprehensive confidence score exceeds the preset threshold, trigger an alarm and lock the corresponding analysis frame, and mark the frame to be analyzed as the target key frame; Calculate the hash value of the target keyframe, and combine the hash value, alarm record and timestamp into a triple, and upload it to the judicial blockchain for on-chain evidence storage; After the judicial blockchain returns the transaction hash value, the target keyframe, alarm records and associated user operation logs will be stored. Based on the triples corresponding to all target keyframes that have been uploaded to the blockchain and the associated user operation logs, the integrity of the evidence in the bidding process is verified, and a witness report is generated.

2. The digital spatial bidding process witnessing service method based on a time-series model according to claim 1, characterized in that, The acquisition of the frame to be analyzed includes: The perceptual hash values ​​of the current frame and the previous frame in the live video stream are obtained respectively, and the Hamming distance between them is calculated. If the Hamming distance is greater than or equal to a preset distance threshold, the current frame is marked as a frame to be analyzed; otherwise, it is not marked.

3. The digital spatial bidding process witnessing service method based on a time-series model according to claim 1, characterized in that, The step of inputting abnormal information and user operation logs into the multimodal large model includes: Before the abnormal signals output by each identification module are input into the multimodal large model, a timing consistency check based on operational causal logic is performed: Based on the user operation log and the timestamps output by each identification module, determine whether the order of abnormal information conforms to a reasonable causal chain of human operation; The abnormal information is input into the multimodal large model only when the abnormal signal satisfies a preset causal dependency relationship in the time dimension.

4. The digital spatial bidding process witnessing service method based on a time-series model according to claim 1, characterized in that, The training process of the multimodal large model includes: Obtain the scoring rules and corresponding historical evaluation interface screenshots from the publicly tendered documents, construct a pre-training dataset for image-text semantic alignment, and pre-train a basic multimodal artificial intelligence model based on the pre-training dataset; The pre-trained model is fine-tuned based on manually labeled training samples to obtain the multimodal large model; Each training sample includes a frame to be analyzed, a user operation log time-aligned with the frame to be analyzed, and a corresponding structured violation description. The structured violation description contains five fields: violation type, object, value, action, and time. These fields are used to guide the model to fuse the structured abnormal signals output by each identification module during inference and generate a structured violation judgment with comprehensive confidence.

5. The digital spatial bidding process witnessing service method based on a time-series model according to claim 4, characterized in that, The fine-tuning of the pre-trained model based on manually labeled training samples includes: When the rule engine determines that the structured violation description output by the multimodal large model is a false alarm, it marks the corresponding input training sample as a negative training sample to obtain a negative training sample set; otherwise, it marks the input training sample as a positive training sample to obtain a positive training sample set. After the evaluation task is completed, incremental fine-tuning is performed on the classification head of the multimodal large model based on the positive training sample set and the negative training sample set; The negative training sample set employs a time-sliding window strategy, retaining historical negative training samples within a preset time window.

6. The digital spatial bidding process witnessing service method based on a time-series model according to claim 1, characterized in that, The identity recognition module performs face detection and feature extraction on the frame to be analyzed to obtain a face feature vector; based on the preset judge registration database and cosine similarity threshold, it determines whether the current person is a registered judge and is on duty. If yes, output the identity matching result and the on-duty status indicator; No, then no information will be output; The action recognition module is used to determine whether the action appearing in the frame to be analyzed is abnormal; if so, it outputs the action category and duration. No, then no information will be output; The object recognition module performs target detection on the frame to be analyzed to obtain the category of the object in the image; Based on a pre-set list of prohibited items, obtain the categories of prohibited items; The speech recognition module performs text conversion on the audio stream that is time-aligned with the frame to be analyzed to obtain text segments; it determines whether there are preset illegal keywords in the text segments, and if so, marks the audio stream as an abnormal audio stream; The OCR recognition module performs optical character recognition on the frame to be analyzed to obtain the original text block; based on the preset numerical regular expression, it filters out the numerical text related to the bidding business rules from the original text block and converts it into structured fields.

7. The digital spatial bidding process witnessing service method based on a time-series model according to claim 6, characterized in that, The determination of whether the action appearing in the frame to be analyzed is abnormal includes: The action recognition module performs human key point detection on multiple consecutive frames to be analyzed; it obtains spatiotemporal features by performing temporal modeling on the human key points, the spatiotemporal features including the time sequence of posture parameters, action persistence, rate of change or steady state features; when the spatiotemporal features match a preset evaluation violation pattern, it outputs an action abnormality signal.

8. The digital spatial bidding process witnessing service method based on a time-series model according to claim 1, characterized in that, The verification of the integrity of evidence in the bid evaluation process includes: Extract the triples corresponding to all target keyframes in the chain; Calculate the timestamp interval between adjacent triples. If there is a timestamp interval greater than a preset interval threshold and no corresponding operation log record, then mark the interval corresponding to the timestamp interval as an evidence break zone.

9. The digital spatial bidding process witnessing service method based on a time-series model according to claim 1, characterized in that, When the target keyframe is stored, its filename is uniquely determined by the transaction hash value returned by the judicial blockchain; the associated alarm records and user operation logs are stored in the structured database with the same transaction hash value as the primary key. The transaction hash value, target keyframe, alarm record, and user operation log together constitute an indivisible evidence unit. If any component is missing or the hash does not match, the corresponding evidence unit will be marked as invalid.

10. A digital spatial bidding process witnessing service system based on a time-series model, operating based on the digital spatial bidding process witnessing service method based on a time-series model as described in any one of claims 1-9, characterized in that, Includes an analysis module and a storage module; The analysis module is used to collect live video streams, audio streams, and user operation logs from the bid evaluation terminal in real time. Perform inter-frame differential processing on the live video stream to obtain the frames to be analyzed; The frame to be analyzed is input into several recognition modules; each recognition module outputs abnormal information based on preset bidding business rules; among them, the several recognition modules include an identity recognition module, an action recognition module, an item recognition module, an OCR recognition module, and a speech recognition module, and the audio stream of the frame to be analyzed is input into the speech recognition module; Input the abnormal information and user operation logs into the multimodal large model and output the comprehensive confidence score; if the comprehensive confidence score exceeds the preset threshold, trigger an alarm and lock the corresponding analysis frame and mark it as the target key frame; The storage module is used to calculate the hash value of the target key frame before storing the target key frame, and combine the hash value, alarm record and timestamp into a triple, and upload it to the judicial blockchain for on-chain evidence storage. After receiving the transaction hash value returned by the judicial blockchain, the target keyframe, alarm records and associated user operation logs are stored. Based on the triples corresponding to all target keyframes that have been uploaded to the blockchain and the associated user operation logs, the integrity of the evidence in the bidding process is verified, and a witness report is generated.

Images