Supply chain collaborative optimization method and device based on multi-modal data fusion and medium

Abstract

The invention discloses a supply chain collaborative optimization method and device based on multi-modal data fusion and a medium, and belongs to the technical field of supply chain intelligent management. The method comprises the following steps: acquiring multi-modal data in a supply chain environment; preprocessing the multi-modal data to generate standardized data; performing space-time alignment on the point cloud and the image data to generate time alignment data; based on uncertainty federated learning, edge node credibility is quantified, conflict evidence is processed, and credibility data is generated; processing the point cloud and the image data by using cross-modal attention fusion to generate fusion features; generating an inventory prediction or path planning decision based on the fused features; and outputting the decision to a supply chain management system to execute automatic operation. According to the method, high-precision fusion and credible collaborative decision-making of multi-modal data are realized, and meanwhile, the technical effects of privacy security and real-time performance are ensured.

Description

Technical Field

[0001] This application relates to the technical field of intelligent supply chain management, and in particular to a supply chain collaborative optimization method, device and medium based on multimodal data fusion. Background Technology

[0002] The supply chain decision-making technology field primarily relies on Enterprise Resource Planning (ERP) and Warehouse Management Systems (WMS) to automate business processes. Existing technologies support basic decision-making through structured data (such as order forms and inventory databases) and employ centralized cloud computing architectures to process standardized information flows. Typical solutions include inventory forecasting models based on historical sales data, logistics route planning algorithms incorporating GPS trajectories, and temperature and humidity threshold alarm mechanisms that monitor the warehouse environment through sensor networks. These technologies constitute the core infrastructure of current supply chain management, providing enterprises with fundamental data analytics capabilities. Current technologies suffer from the following problems: First, multi-source heterogeneous data fusion fails. Traditional systems can only process structured data (such as Excel spreadsheets) and cannot fuse point clouds (3D scanning), images (surveillance videos), text (inspection reports), and sensor data (temperature, humidity / GPS) under a unified spatiotemporal reference, resulting in insufficient utilization of unstructured data. Second, real-time decision-making suffers from severe delays. Response cycles relying on manually compiled data can reach 2-8 hours, failing to meet the agile decision-making needs of emergencies such as warehouse fires. Third, there is a conflict between privacy protection and collaboration. Enterprises refuse to share inventory data due to its confidentiality, leading to "bullwhip effect"-like supply chain fluctuations. Fourth, decision-making in complex environments is unreliable. Single data sources (such as GPS) have high path planning error rates when they fail in scenarios such as tunnels. Fifth, cross-modal semantic fragmentation exists. There is significant temporal misalignment between LiDAR (low-frequency) and camera (high-frequency) data, and there is a lack of mechanisms to resolve cross-modal conflicts such as those between temperature and humidity sensors and visual icing detection, leading to increased misjudgment rates. These deficiencies collectively cause supply chain decision-making delays, resource waste, and weak anti-interference capabilities.

[0003] Therefore, how to achieve high-precision fusion and reliable collaborative decision-making of multimodal data while ensuring privacy, security and real-time performance has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a supply chain collaborative optimization method, device, and medium based on multimodal data fusion to solve the following technical problem: how to achieve high-precision fusion of multimodal data and reliable collaborative decision-making, while ensuring privacy, security, and real-time performance.

[0005] In a first aspect, embodiments of this application provide a supply chain collaborative optimization method based on multimodal data fusion. The method includes: acquiring multimodal data in the supply chain environment; wherein the multimodal data includes point cloud data, image data, text data, and sensor data, the point cloud data representing three-dimensional geometric information of product stacking status, the image data representing visual information from surveillance videos, the text data representing business information from order reports, and the sensor data representing physical information such as environmental temperature, humidity, vibration, and illumination; preprocessing the multimodal data to a standardized data format to generate standardized multimodal data; and processing the point cloud data and image data within the standardized multimodal data. Multimodal spatiotemporal alignment is performed in the time domain to generate time-aligned multimodal data; the credibility of multiple edge nodes in the supply chain environment is measured and conflict evidence is processed based on a preset uncertainty federated learning processing algorithm to generate node credibility data; point cloud data and image data in the time-aligned multimodal data are processed based on a preset cross-modal attention fusion algorithm to generate fused feature data; supply chain decisions are generated based on the fused feature data; wherein, the supply chain decisions include inventory level prediction data or logistics route planning data; the supply chain decisions are output to a preset supply chain management system for execution; wherein, the execution operations include adjusting inventory strategies or optimizing logistics routes.

[0006] In one implementation of this application, acquiring multimodal data in a supply chain environment specifically includes: deploying a multimodal sensor network at the supply chain terminal; wherein the multimodal sensor network includes an RGB-D camera, UWB positioning tags, environmental sensors, and RFID readers; acquiring point cloud data of the stacked state of goods through the RGB-D camera; capturing video streams in real time and extracting keyframe images through a monitoring camera to acquire image data; parsing the order system in the supply chain environment to parse order report documents in the order system to extract business logic description text; acquiring sensor data through the environmental sensors; wherein the process of acquiring sensor data includes periodically reading temperature, humidity, vibration, and light intensity readings; and integrating the point cloud data, image data, text data, and sensor data to obtain the multimodal data.

[0007] In one implementation of this application, preprocessing the multimodal data to standardize the data format to generate standardized multimodal data specifically includes: receiving point cloud data, image data, text data, and sensor data from the multimodal data; cleaning the point cloud data to remove noise points and invalid coordinates to obtain cleaned point cloud data; converting the image data to unify the image resolution and color space to obtain standardized image data; performing natural language processing on the text data to extract keywords and entity information to obtain structured text data; normalizing the sensor data to scale the data values ​​to a uniform range to obtain standardized sensor data; and merging the cleaned point cloud data, standardized image data, structured text data, and standardized sensor data to generate the standardized multimodal data.

[0008] In one implementation of this application, multimodal spatiotemporal alignment of the time domains of point cloud data and image data in the standardized multimodal data is performed to generate time-aligned multimodal data. Specifically, this includes: inputting point cloud data and image data from the standardized multimodal data; wherein the point cloud data corresponds to a low-frequency sampling device, and the image data corresponds to a high-frequency sampling device; synchronizing the time references of the lidar device and the camera device to obtain a synchronization timestamp; performing time series interpolation on the low-frequency point cloud data to match the time points of the high-frequency image data, obtaining interpolated point cloud data; aligning the time domains of the point cloud data and image data based on the synchronization timestamp and the interpolated point cloud data to obtain time-aligned point cloud data and time-aligned image data; and combining the time-aligned point cloud data and time-aligned image data to generate the time-aligned multimodal data.

[0009] In one implementation of this application, the credibility of multiple edge nodes in the supply chain environment is quantified and conflicting evidence is processed based on a preset uncertainty federated learning processing algorithm to generate node credibility data. Specifically, this includes: introducing a preset deep evidence theory model to quantify the predictive uncertainty of multiple edge nodes and obtain node uncertainty values; wherein the quantification process calculates uncertainty based on the degree of evidence conflict; identifying high-conflict evidence based on the node uncertainty values; wherein high-conflict evidence represents contradictions between data from different nodes, and obtaining conflict evidence identifiers; parsing the conflict evidence identifiers based on a preset conflict reprocessing module to correct inconsistent data and obtain a consistent data representation; obtaining a weighted contribution value; and generating the node credibility data based on the consistent data representation and the weighted contribution value.

[0010] In one implementation of this application, point cloud data and image data in the time-aligned multimodal data are processed based on a preset cross-modal attention fusion algorithm to generate fused feature data. Specifically, this includes: defining candidate regions of the point cloud and calculating the center coordinates of each candidate region to obtain a center coordinate vector; extracting features from the image data as key vectors and value vectors to obtain image feature representations; using the center coordinate vectors as query vectors and calculating the fused point cloud data and image data based on an attention mechanism to obtain attention-weighted features; and integrating the attention-weighted features to generate the fused feature data. The fused feature data retains the geometric accuracy of the point cloud and enhances the semantic information of the image.

[0011] In one implementation of this application, generating supply chain decisions based on the fused feature data specifically includes: analyzing the fused feature data to extract supply chain-related features and obtaining inventory features or path features; wherein the inventory features represent commodity demand trends and the path features represent logistics obstacle information; processing the inventory features based on a preset decision model to generate inventory level prediction data, or processing the path features to generate logistics path planning data; and generating the supply chain decision based on the inventory level prediction data or logistics path planning data.

[0012] In one implementation of this application, the supply chain decision is output to a preset supply chain management system for execution, specifically including: transmitting the supply chain decision data to a central supply chain management system; parsing the inventory level forecast data in the central supply chain management system to adjust the inventory strategy, or parsing the logistics route planning data to update the logistics route, and obtaining the operation execution result; recording the operation execution result to optimize the subsequent decision generation process.

[0013] Secondly, embodiments of this application also provide a supply chain collaborative optimization device based on multimodal data fusion. The device includes: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: acquire multimodal data in the supply chain environment; wherein the multimodal data includes point cloud data, image data, text data, and sensor data, where the point cloud data represents three-dimensional geometric information of product stacking status, the image data represents visual information from surveillance videos, the text data represents business information from order reports, and the sensor data represents physical information such as environmental temperature, humidity, vibration, and illumination; and preprocess the multimodal data. Based on a standardized data format, standardized multimodal data is generated; the point cloud data and image data in the standardized multimodal data are spatiotemporally aligned in the time domain to generate time-aligned multimodal data; the credibility of multiple edge nodes in the supply chain environment is measured and conflict evidence is processed based on a preset uncertainty federated learning processing algorithm to generate node credibility data; the point cloud data and image data in the time-aligned multimodal data are processed based on a preset cross-modal attention fusion algorithm to generate fused feature data; supply chain decisions are generated based on the fused feature data; wherein, the supply chain decisions include inventory level prediction data or logistics route planning data; the supply chain decisions are output to a preset supply chain management system for execution; wherein, the execution operations include adjusting inventory strategies or optimizing logistics routes.

[0014] Thirdly, embodiments of this application also provide a non-volatile computer storage medium for supply chain collaborative optimization based on multimodal data fusion, storing computer-executable instructions, characterized in that the computer-executable instructions are configured to: acquire multimodal data in the supply chain environment; wherein the multimodal data includes point cloud data, image data, text data, and sensor data, the point cloud data representing three-dimensional geometric information of the stacked state of goods, the image data representing visual information of surveillance video, the text data representing business information of order reports, and the sensor data representing physical information of environmental temperature, humidity, vibration, and illumination; preprocess the multimodal data to a standardized data format to generate standardized multimodal data; and apply the standardized data to the standard... The process involves: 1) performing multimodal spatiotemporal alignment of point cloud data and image data in the multimodal data domain to generate time-aligned multimodal data; 2) quantifying the credibility of multiple edge nodes in the supply chain environment and processing conflicting evidence based on a preset uncertainty federated learning algorithm to generate node credibility data; 3) processing the point cloud data and image data in the time-aligned multimodal data based on a preset cross-modal attention fusion algorithm to generate fused feature data; 4) generating supply chain decisions based on the fused feature data, wherein the supply chain decisions include inventory level prediction data or logistics route planning data; and 5) outputting the supply chain decisions to a preset supply chain management system for execution, wherein the execution operations include adjusting inventory strategies or optimizing logistics routes.

[0015] The supply chain collaborative optimization method, device, and medium based on multimodal data fusion provided in this application embodiment have at least the following technical effects: By employing a dual spatiotemporal alignment mechanism of hardware clock synchronization and software dynamic interpolation, the temporal misalignment problem of heterogeneous devices such as LiDAR and cameras is effectively solved, improving cross-modal data synchronization accuracy to the 10ms level and significantly enhancing the consistency between point cloud geometric information and image semantic information. A cross-modal attention fusion network based on point cloud center coordinates enhances semantic relevance while preserving 3D spatial accuracy, increasing the utilization rate of unstructured data by over 80% and significantly reducing the decision-making error rate caused by modal fragmentation.

[0016] By introducing a federated learning mechanism based on deep evidence theory, and through quantifying the uncertainty of node predictions and reprocessing conflicting evidence, it effectively identifies and corrects abnormal states caused by environmental interference (such as tunnel signal loss and sensor data conflicts), significantly enhancing the reliability of path planning and inventory forecasting in complex environments. The cloud-edge-device three-tier collaborative architecture compresses the traditional 2-8 hour decision-making cycle to the minute level. Through the collaboration of local modeling at edge nodes and global optimization in the cloud, it achieves real-time response capabilities for inventory strategy adjustments and logistics route updates.

[0017] Based on an uncertainty-weighted federated learning model, knowledge from edge nodes is aggregated without the need for sharing original data, enabling data collaboration among suppliers, logistics providers, and retailers. Through a reverse weighting mechanism based on node contributions, sensitive enterprise data (such as inventory levels and sales records) is protected while effectively mitigating supply chain fluctuations. This fundamentally enhances the credibility and stability of cross-enterprise collaborative decision-making, providing technical assurance for building a secure and reliable supply chain ecosystem. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a supply chain collaborative optimization method based on multimodal data fusion is provided for embodiments of this application. Figure 2 This is a schematic diagram of the internal structure of a supply chain collaborative optimization device based on multimodal data fusion, provided as an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] This application provides a supply chain collaborative optimization method, device, and medium based on multimodal data fusion to solve the following technical problem: how to achieve high-precision fusion of multimodal data and reliable collaborative decision-making, while ensuring privacy, security, and real-time performance.

[0021] The technical solutions proposed in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0022] Figure 1 This application provides a flowchart for supply chain collaborative optimization based on multimodal data fusion, as an embodiment of the present application. Figure 1 As shown in the figure, the supply chain collaborative optimization method based on multimodal data fusion provided in this application embodiment specifically includes the following steps: Step 1: Acquire multimodal data in the supply chain environment; wherein, the multimodal data includes point cloud data, image data, text data and sensor data, the point cloud data represents the three-dimensional geometric information of the stacked state of goods, the image data represents the visual information of the monitoring video, the text data represents the business information of the order report, and the sensor data represents the physical information of the environment such as temperature, humidity, vibration and light.

[0023] Step 1.1: Deploy a multimodal sensor network at the end of the supply chain; wherein the multimodal sensor network includes RGB-D cameras, UWB positioning tags, environmental sensors, and RFID readers.

[0024] Deployment refers to the installation and configuration of sensor networks in a physical environment. The end of the supply chain is the forefront of data processing, such as warehouses or distribution centers. Multimodal sensor networks include components such as RGB-D cameras (devices that simultaneously capture color and depth information), UWB positioning tags (wireless devices based on ultra-wideband technology for precise object location), environmental sensors (devices that monitor physical parameters, such as thermometers and hygrometers), and RFID readers (devices for wirelessly identifying goods). The deployment process involves attaching sensors to key points at the end of the supply chain (such as shelf entrances) and connecting them to the network infrastructure.

[0025] In a specific example, a sensor network is deployed at the end of the supply chain for warehouse A (a hypothetical warehouse location). Specifically, RGB-D cameras are mounted above the shelves to cover the stacking area of ​​goods B (e.g., cardboard boxes); UWB positioning tags are attached to forklifts to track their movement; environmental sensors are distributed in the corners of the warehouse to periodically collect temperature and humidity data; and RFID readers are integrated into the entrance and exit gates to automatically read the product tags. After deployment, the sensor network is automatically synchronized to the central processing unit to ensure real-time data collection covers the entire warehouse.

[0026] Step 1.2: Collect point cloud data of the stacked state of the goods using the RGB-D camera.

[0027] An RGB-D camera is an imaging device that captures color images while providing depth information, generating point cloud data (a set of three-dimensional coordinate points representing the surface position of an object). Data acquisition refers to continuously scanning the environment with the camera to obtain point cloud data, which is used to analyze the stacking status of goods (such as height and shape).

[0028] In a specific example, in warehouse A, RGB-D cameras are configured in high-resolution mode to scan the stacked area of ​​goods B (such as clothing packages). As goods B are stacked on the shelves, the cameras collect point cloud data once per second, recording the three-dimensional position of each package (e.g., package vertex coordinates). Through software processing, stack height and stability information are extracted, avoiding the delays associated with traditional manual inspections.

[0029] Step 1.3: Capture video streams in real time and extract keyframe images through surveillance cameras to collect image data.

[0030] Surveillance cameras are standard vision devices used to continuously capture video; a video stream refers to a continuous sequence of video; keyframe images are representative images extracted from the video stream (e.g., one frame every 5 seconds) to reduce redundant data. Acquiring image data involves real-time processing of the video stream to detect critical events (such as the movement of goods).

[0031] In a specific example, in the unloading area of ​​warehouse A, surveillance cameras capture real-time video streams of forklift activity. When a forklift moves an item C (such as a fragile item), an algorithm identifies motion events and extracts keyframe images (e.g., consecutive images of the grabbing process). This generates image data for analyzing the location and status of the goods without storing the entire video sequence.

[0032] Step 1.4: Parse the order system in the supply chain environment to parse the order report documents in the order system in order to extract the business logic description text.

[0033] An order system is business software (such as an ERP system) that manages supply chain activities; order report documents are text files generated by the system (such as Excel or JSON format) that contain business logic (such as order quantity and priority). Parsing refers to reading the document content and extracting structured information; the business logic description text consists of keywords and phrases (such as "low inventory warning") used for decision support.

[0034] In a specific example, for the order system of warehouse A, the system periodically generates order report documents (containing inbound and outbound records for product C). The parsing module reads the report, extracts the text data (such as "Product C has a high risk of being out of stock"), removes noise through part-of-speech tagging, and generates a business logic description. This avoids the inefficiency of manual report analysis.

[0035] Step 1.5: Collect sensor data through environmental sensors; wherein, the process of collecting sensor data includes periodically reading temperature, humidity, vibration and light intensity readings.

[0036] Environmental sensors are physical devices (such as thermometers and hygrometers) that take periodic readings at fixed intervals (e.g., once per minute). Temperature and humidity readings measure air parameters, vibration readings capture equipment vibrations, and illumination readings measure light intensity. The data acquisition process involves the sensors automatically reporting data to ensure real-time performance.

[0037] In a specific example, in the storage area of ​​warehouse A, environmental sensors are installed beside the shelves, collecting data every minute. For instance, thermometers read environmental parameters (such as 25°C), vibration sensors monitor forklift vibrations (such as intensity values), and light sensors record changes in brightness. These readings are integrated into sensor data to assess storage conditions.

[0038] Step 1.6: Integrate the point cloud data, image data, text data, and sensor data to obtain the multimodal data.

[0039] Integration refers to merging data from different modalities into a unified format (such as a JSON data structure) to ensure interoperability. The acquired multimodal data is output as a whole for easier subsequent processing.

[0040] In a specific example, in warehouse A, data collected from all sub-steps is fed into a central buffer: point cloud data describes the stacking of goods B, image data comes from keyframes, text data is extracted from order reports, and sensor data includes temperature. The integration module uses timestamp alignment to generate multimodal data packets (such as storage objects containing all modalities), avoiding manual data preparation.

[0041] Step 2: Preprocess the multimodal data to standardize the data format to generate standardized multimodal data.

[0042] Step 2.1: Receive point cloud data, image data, text data, and sensor data from the multimodal data.

[0043] Receiving refers to acquiring input data through a software interface; point cloud data is a three-dimensional point set, image data is a pixel array, text data is a character sequence, and sensor data is a numerical sequence.

[0044] In a specific example, during the preprocessing stage, the module receives data from the output of step 1.6: point cloud data describing the geometric information of product C, image data displaying visual content, text data containing keywords, and sensor data containing temperature and humidity values. The system automatically detects missing data and triggers re-acquisition.

[0045] Step 2.2: Perform data cleaning on the point cloud data to remove noise points and invalid coordinates, and obtain cleaned point cloud data.

[0046] Data cleaning is the process of removing useless information; noise points refer to disordered points caused by data acquisition errors (such as dust interference points), and invalid coordinates are coordinates that are outside the physical range (such as negative height values). After cleaning, the point cloud data retains valid three-dimensional information.

[0047] In a specific example, in the point cloud data of warehouse A, for the stacked scanning of goods B, the cleaning module identifies and removes noise points with abnormal coordinate values ​​(such as isolated points), retains the package outline points, obtains clean point cloud data, and improves the accuracy of stacking analysis.

[0048] Step 2.3: Convert the image data to unify the image resolution and color space, and obtain standardized image data.

[0049] Format conversion transforms image attributes; it standardizes resolution to a fixed size (e.g., 1920x1080 pixels) and color space to RGB or grayscale. Standardizing image data ensures visual consistency.

[0050] In a specific example, in the image data of warehouse A, the original keyframes captured by the surveillance cameras were of different sizes. The conversion module adjusted all the images to a uniform resolution (such as 1024x768) and converted them to RGB color to generate standardized image data, eliminating compatibility issues.

[0051] Step 2.4: Perform natural language processing on the text data to extract keywords and entity information, and obtain structured text data.

[0052] Natural Language Processing (NLP) is a computational tool for processing language; keywords are core terms (e.g., "low inventory"), and entity information is object identifiers (e.g., "product C"). Structured text data is organized tabular data.

[0053] In a specific example, in the text data of warehouse A, the NLP module parses order report phrases (such as "demand for product B is rising"), extracts keywords (such as "demand") and entities (such as "product B"), and outputs structured text data, reducing text redundancy.

[0054] Step 2.5: Normalize the sensor data to scale the data values ​​to a uniform range and obtain standardized sensor data.

[0055] Normalization scales numerical values ​​to a standard range (e.g., between 0 and 1); scaling ensures that data from different units are comparable (e.g., temperature value conversion). Standardizing sensor data eliminates unit differences.

[0056] In a specific example, in the sensor data of warehouse A, the vibration readings range from 0 to 100 (units), and the temperature ranges from 10 to 40°C; the normalization module scales all values ​​to the range of 0 to 1 (e.g., 0.5 for temperature represents 25°C) to obtain consistent data.

[0057] Step 2.6: Merge the cleaned point cloud data, standardized image data, structured text data, and standardized sensor data to generate the standardized multimodal data.

[0058] Merging involves integrating cleaned data into a single object; standardized multimodal data serves as a unified input.

[0059] In a specific example, in warehouse A, the merging module combines all preprocessed data: cleaned point clouds, normalized images, structured text, and scaled sensor data, generating normalized multimodal data (such as multidimensional tensors) for subsequent processing.

[0060] Step 3: Perform multimodal spatiotemporal alignment on the point cloud data and image data in the standardized multimodal data to generate time-aligned multimodal data.

[0061] Step 3.1: Input the point cloud data and image data from the standardized multimodal data; wherein the point cloud data corresponds to a low-frequency sampling device and the image data corresponds to a high-frequency sampling device.

[0062] The input is standard data; low-frequency sampling devices (such as lidar) collect point clouds less frequently (once per second), while high-frequency sampling devices (such as cameras) collect images faster (30 frames per second).

[0063] In a specific example, in warehouse A, the input module acquires point cloud data (from an RGB-D camera, 5 times per second) and image data (from a surveillance camera, 30 frames per second) to identify device differences and initiate alignment.

[0064] Step 3.2: Synchronize the time base of the lidar device and the camera device to obtain the synchronization timestamp.

[0065] LiDAR devices are point cloud sources, and camera devices are image sources; the time reference is synchronized through a hardware clock; the synchronization timestamp is a unified time marker.

[0066] In a specific example, in warehouse A, the time of the LiDAR and camera is set to the GPS time source to generate synchronized timestamps (with millisecond-level accuracy), eliminating device latency.

[0067] Step 3.3: Perform time series interpolation on the low-frequency point cloud data to match the time points of the high-frequency image data, and obtain the interpolated point cloud data.

[0068] Interpolation refers to filling time gaps through calculation (such as linear interpolation); matching ensures that point cloud time points are aligned with image time points. After interpolation, the point cloud data has continuous sampling points.

[0069] In a specific example, in the low-frequency point cloud data of warehouse A, the original points are spaced 1 second apart; the interpolation module calculates the estimated point cloud at high-frequency time points (e.g., every 0.1 seconds), obtains the interpolated data, and matches it with a 30Hz image frame.

[0070] Step 3.4: Based on the synchronized timestamp and interpolated point cloud data, align the time domains of the point cloud data and image data to obtain time-aligned point cloud data and time-aligned image data.

[0071] Alignment refers to the temporal association between point cloud and image data; time-aligned point cloud data and image data have the same timestamp index.

[0072] In a specific example, in warehouse A, the alignment module uses synchronized timestamps to map point clouds and images (e.g., 10:00:00 corresponds to the same scene) to generate time-aligned data, ensuring that the product locations match.

[0073] Step 3.5: Combine the time-aligned point cloud data and time-aligned image data to generate the time-aligned multimodal data.

[0074] Combining refers to integrating aligned data; time-aligned multimodal data is used as output, while preserving spatial correlations.

[0075] In a specific example, in repository A, the fusion module packages time-aligned point clouds and images to generate time-aligned multimodal data (such as HDF5 format) for subsequent fusion.

[0076] Step 4: Based on the preset uncertainty federated learning processing algorithm, measure the credibility of multiple edge nodes in the supply chain environment and process conflict evidence to generate node credibility data.

[0077] Step 4.1: Introduce a preset deep evidence theory model to quantify the prediction uncertainty of multiple edge nodes and obtain the node uncertainty value; wherein, the quantification process calculates uncertainty based on the degree of evidence conflict.

[0078] Deep evidence theory models are learning frameworks (such as those based on Dempster-Shafer theory) that quantify the uncertainty of node predictions (e.g., probability values); the degree of evidence conflict refers to the inconsistency of data between nodes. The node uncertainty value represents the reliability of node predictions.

[0079] In a specific example, at multiple edge nodes (such as three processing units) in warehouse A, the model uses input data (point cloud and image) to calculate node predictions (such as product counts); based on the degree of conflict (such as the data difference between nodes 1 and 2), it generates uncertainty values ​​(such as 0.3 indicating low confidence).

[0080] Step 4.2: Identify high-conflict evidence based on the node uncertainty value; wherein, high-conflict evidence represents the contradiction between data from different nodes, and obtain the conflict evidence identifier.

[0081] High-conflict evidence refers to evidence with uncertainty exceeding a threshold (e.g., >0.5), and contradictions include inconsistencies in node data; conflict evidence is identified by marking conflict points.

[0082] In a specific example, in warehouse A, the module detects the uncertainty value of a node (e.g., node 1 with a value of 0.7). When the value exceeds the threshold of 0.5, it is marked as high-conflict evidence (e.g., identifying the node as a source of conflict).

[0083] Step 4.3: Based on the preset conflict reprocessing module, parse the conflict evidence identifier to correct inconsistent data and obtain a consistent data representation.

[0084] The conflict reprocessing module corrects the data through rules or models; consistent data represents the unified data after conflict resolution.

[0085] In a specific example, in warehouse A, the conflict module resolves the identifier (such as node conflict), reweights the node data (such as taking the average), and generates a consistent data representation (such as the corrected quantity of goods).

[0086] Step 4.4: Obtain the weighted contribution value.

[0087] The weighted contribution value quantifies the influence of each node on the decision (e.g., calculated based on uncertainty value), with high-confidence nodes having higher weights.

[0088] In a specific example, in warehouse A, the module calculates a weighted contribution for each node (e.g., a weight of 0.8 for nodes with low uncertainty) to prepare for data integration.

[0089] Step 4.5: Generate the node credibility data based on the consistent data representation and weighted contribution value.

[0090] The node credibility data is integrated with the node evaluation results (such as credibility score).

[0091] In a specific example, in warehouse A, credibility data (such as JSON format, containing a score for each node) is generated based on consistent data and weights to support subsequent decision-making.

[0092] Step 5: Process the point cloud data and image data in the time-aligned multimodal data based on the preset cross-modal attention fusion algorithm to generate fused feature data.

[0093] Step 5.1: Define the point cloud candidate regions and calculate the center coordinates of each candidate region to obtain the center coordinate vector.

[0094] Candidate regions of a point cloud are subsets of the point cloud (such as shelf sub-areas), and their center coordinates are the geometric center point (three-dimensional coordinates). The center coordinate vector is a set of coordinates.

[0095] In a specific example, in the time-aligned data of warehouse A, the point cloud is segmented into candidate regions (such as product B region), the center of each region is calculated (such as (x,y,z)), and the vector is obtained for querying.

[0096] Step 5.2: Extract the features of the image data as key vectors and value vectors to obtain the image feature representation.

[0097] Key vectors and value vectors are feature representations (such as vectors extracted by CNNs); image feature representations organize visual semantics.

[0098] In a specific example, in the image data of warehouse A, a CNN model extracts keyframe features (such as bounding boxes) and outputs key vectors and value vectors as image features.

[0099] Step 5.3: Based on the center coordinate vector as the query vector, and based on the attention mechanism, calculate the fused point cloud data and image data to obtain attention-weighted features.

[0100] The query vector is used to retrieve relevant image features; the attention mechanism focuses on important features. The attention-weighted feature is the output after weight adjustment.

[0101] In a specific example, in warehouse A, the coordinates of the point cloud center are used as the query vector. The attention mechanism calculates the correlation with image features (such as high-resolution representation correlation) and generates weighted features (such as geometrically enhanced semantics).

[0102] Step 5.4: Integrate the attention-weighted features to generate the fused feature data; wherein the fused feature data preserves the geometric accuracy of the point cloud and enhances the semantic information of the image.

[0103] Integration is the aggregation of features; the fusion of feature data is the final output.

[0104] In a specific example, in warehouse A, weighted features are integrated to generate fused feature data (such as combined vectors), preserving the accuracy of product location and visual context.

[0105] Step 6: Generate supply chain decisions based on the fused feature data; wherein, the supply chain decisions include inventory level forecast data or logistics route planning data.

[0106] Step 6.1: Analyze the fused feature data to extract supply chain-related features and obtain inventory features or path features; wherein, the inventory features represent commodity demand trends and the path features represent logistics obstacle information.

[0107] Analysis refers to feature extraction; inventory features are demand indicators (such as stockout risk), and path features are path conditions (such as congestion points).

[0108] In a specific example, in warehouse A, the module extracts inventory features (such as demand trends for product B) and path features (such as forklift aisle obstacles) from the fused data for decision-making.

[0109] Step 6.2: Process the inventory characteristics based on a preset decision model to generate inventory level prediction data, or process the path characteristics to generate logistics path planning data.

[0110] The decision-making model is a machine learning model (such as LSTM or optimization algorithms); the inventory level forecast data is a quantitative inventory recommendation, and the logistics route planning data is the optimal route plan.

[0111] In a specific example, in warehouse A, the decision model predicts inventory based on inventory characteristics (such as the replenishment quantity of product C next month), or plans logistics routes based on path characteristics (such as routes to bypass obstacles).

[0112] Step 6.3: Generate the supply chain decision based on the inventory level forecast data or logistics route planning data.

[0113] Supply chain decisions are the final business recommendations.

[0114] In a specific example, in warehouse A, the decision module generates decisions (such as JSON documents) for inventory adjustments or path updates.

[0115] Step 7: Output the supply chain decision to the preset supply chain management system for execution; wherein, the execution of the operation includes adjusting the inventory strategy or optimizing the logistics route.

[0116] Step 7.1: Transmit the supply chain decision data to the central supply chain management system.

[0117] Transmission occurs via the network (e.g., API calls); the central management system is the core control software.

[0118] In a specific example, at warehouse A, decision data (such as inventory forecasts) is sent to a central system database via HTTP.

[0119] Step 7.2: In the central supply chain management system, parse the inventory level forecast data to adjust the inventory strategy, or parse the logistics route planning data to update the logistics route, and obtain the operation execution results.

[0120] Analysis involves processing decision data; the result of an operation is action feedback (such as successful execution).

[0121] In a specific example, in the central system of warehouse A, inventory data is parsed to adjust strategies (such as increasing inventory of product B) or update paths (such as replanning forklift routes), and the resulting logs are obtained.

[0122] Step 7.3: Record the results of the operation to optimize the subsequent decision generation process.

[0123] Recording refers to storing history; optimization refers to feedback learning.

[0124] In a specific example, in warehouse A, the system records results (such as the effect of inventory adjustments) for future decision-making improvements (such as revising the forecasting model).

[0125] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a supply chain collaborative optimization device based on multimodal data fusion, the structure of which is as follows: Figure 2 As shown.

[0126] Figure 2 This is a schematic diagram of the internal structure of a supply chain collaborative optimization device based on multimodal data fusion, provided as an embodiment of this application. Figure 2 As shown, the device includes: At least one processor 201; And a memory 202 that is communicatively connected to at least one processor; The memory 202 stores instructions executable by at least one processor, which are executed by at least one processor 201 to enable at least one processor 201 to: Acquire multimodal data from the supply chain environment; wherein the multimodal data includes point cloud data, image data, text data, and sensor data, wherein the point cloud data represents the three-dimensional geometric information of the stacked state of goods, the image data represents the visual information of the surveillance video, the text data represents the business information of the order report, and the sensor data represents the physical information of the environment, such as temperature, humidity, vibration, and light. Preprocess the multimodal data to a standardized data format to generate standardized multimodal data; perform multimodal spatiotemporal alignment on the point cloud data and image data in the standardized multimodal data to generate time-aligned multimodal data; measure the credibility of multiple edge nodes in the supply chain environment and process conflict evidence based on a preset uncertainty federated learning processing algorithm to generate node credibility data; process the point cloud data and image data in the time-aligned multimodal data based on a preset cross-modal attention fusion algorithm to generate fused feature data; generate supply chain decisions based on the fused feature data; wherein the supply chain decisions include inventory level prediction data or logistics route planning data; output the supply chain decisions to a preset supply chain management system for execution; wherein the execution operations include adjusting inventory strategies or optimizing logistics routes.

[0127] Some embodiments of this application provide corresponding to Figure 1 A non-volatile computer storage medium for supply chain collaborative optimization based on multimodal data fusion, storing computer-executable instructions, wherein the computer-executable instructions are configured as follows: Acquire multimodal data from the supply chain environment; wherein the multimodal data includes point cloud data, image data, text data, and sensor data, wherein the point cloud data represents the three-dimensional geometric information of the stacked state of goods, the image data represents the visual information of the surveillance video, the text data represents the business information of the order report, and the sensor data represents the physical information of the environment, such as temperature, humidity, vibration, and light. Preprocess the multimodal data to a standardized data format to generate standardized multimodal data; perform multimodal spatiotemporal alignment on the point cloud data and image data in the standardized multimodal data to generate time-aligned multimodal data; measure the credibility of multiple edge nodes in the supply chain environment and process conflict evidence based on a preset uncertainty federated learning processing algorithm to generate node credibility data; process the point cloud data and image data in the time-aligned multimodal data based on a preset cross-modal attention fusion algorithm to generate fused feature data; generate supply chain decisions based on the fused feature data; wherein the supply chain decisions include inventory level prediction data or logistics route planning data; output the supply chain decisions to a preset supply chain management system for execution; wherein the execution operations include adjusting inventory strategies or optimizing logistics routes.

[0128] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0129] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0130] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0131] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0132] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0133] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0134] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0135] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0136] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0137] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0138] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A supply chain collaborative optimization method based on multimodal data fusion, characterized in that, The method includes: Acquire multimodal data in the supply chain environment; wherein, the multimodal data includes point cloud data, image data, text data and sensor data, the point cloud data represents the three-dimensional geometric information of the stacked state of goods, the image data represents the visual information of the surveillance video, the text data represents the business information of the order report, and the sensor data represents the physical information of the environment such as temperature, humidity, vibration and light. The multimodal data is preprocessed to a standardized data format to generate standardized multimodal data; Multimodal spatiotemporal alignment is performed on the point cloud data and image data in the standardized multimodal data in the temporal domain to generate time-aligned multimodal data; Based on a preset uncertainty federated learning processing algorithm, the credibility of multiple edge nodes in the supply chain environment is measured and conflict evidence is processed to generate node credibility data. The point cloud data and image data in the time-aligned multimodal data are processed based on a preset cross-modal attention fusion algorithm to generate fused feature data. Supply chain decisions are generated based on the fused feature data; wherein, the supply chain decisions include inventory level forecast data or logistics route planning data; The supply chain decision is output to a preset supply chain management system for execution; wherein, the execution of the operation includes adjusting inventory strategy or optimizing logistics route.

2. The supply chain collaborative optimization method based on multimodal data fusion according to claim 1, characterized in that, Acquiring multimodal data from the supply chain environment, specifically including: Deploying a multimodal sensor network at the end of the supply chain; wherein, the multimodal sensor network includes RGB-D cameras, UWB positioning tags, environmental sensors, and RFID readers; Point cloud data of the stacked goods are collected using the RGB-D camera; Image data is acquired by capturing video streams in real time and extracting keyframe images through surveillance cameras; The order system in the supply chain environment is parsed to extract business logic description text from the order report documents in the order system. Sensor data is collected through environmental sensors; wherein, the process of collecting sensor data includes periodically reading temperature, humidity, vibration, and light intensity readings; The point cloud data, image data, text data, and sensor data are integrated to obtain the multimodal data.

3. The supply chain collaborative optimization method based on multimodal data fusion according to claim 1, characterized in that, Preprocessing the multimodal data to standardize the data format to generate standardized multimodal data specifically includes: Receive point cloud data, image data, text data, and sensor data from the multimodal data; The point cloud data is cleaned to remove noise points and invalid coordinates, and the cleaned point cloud data is obtained. The image data is format-converted to unify the image resolution and color space, thereby obtaining standardized image data; Natural language processing is performed on the text data to extract keywords and entity information, thereby obtaining structured text data; The sensor data is normalized to scale the data values ​​to a uniform range, thereby obtaining standardized sensor data. The cleaned point cloud data, standardized image data, structured text data, and standardized sensor data are merged to generate the standardized multimodal data.

4. The supply chain collaborative optimization method based on multimodal data fusion according to claim 1, characterized in that, Multimodal spatiotemporal alignment is performed on the point cloud data and image data in the standardized multimodal data in the temporal domain to generate time-aligned multimodal data, specifically including: Input the point cloud data and image data from the standardized multimodal data; wherein the point cloud data corresponds to a low-frequency sampling device and the image data corresponds to a high-frequency sampling device; Synchronize the time base of the lidar and camera devices to obtain synchronization timestamps; Perform time series interpolation on low-frequency point cloud data to match the time points of high-frequency image data, and obtain the interpolated point cloud data; Based on the synchronized timestamp and interpolated point cloud data, align the time domains of the point cloud data and image data to obtain time-aligned point cloud data and time-aligned image data. The time-aligned point cloud data and time-aligned image data are combined to generate the time-aligned multimodal data.

5. The supply chain collaborative optimization method based on multimodal data fusion according to claim 1, characterized in that, Based on a pre-defined uncertainty federated learning algorithm, the credibility of multiple edge nodes in the supply chain environment is measured and conflicting evidence is processed to generate node credibility data, specifically including: A pre-defined deep evidence theory model is introduced to quantify the prediction uncertainty of multiple edge nodes and obtain node uncertainty values; wherein, the quantification process calculates uncertainty based on the degree of evidence conflict. High-conflict evidence is identified based on the node uncertainty value; wherein, high-conflict evidence represents contradictions between data from different nodes, and conflict evidence identifiers are obtained; The conflict evidence identifier is parsed based on the preset conflict reprocessing module to correct inconsistent data and obtain a consistent data representation; Obtain the weighted contribution value; The node credibility data is generated based on the consistent data representation and weighted contribution value.

6. The supply chain collaborative optimization method based on multimodal data fusion according to claim 1, characterized in that, The point cloud data and image data in the time-aligned multimodal data are processed based on a preset cross-modal attention fusion algorithm to generate fused feature data, specifically including: Define candidate regions for the point cloud and calculate the center coordinates of each candidate region to obtain the center coordinate vector; Extract features from image data as key vectors and value vectors to obtain image feature representations; Based on the center coordinate vector as the query vector, and using the attention mechanism to calculate and fuse point cloud data and image data, attention-weighted features are obtained; The attention-weighted features are integrated to generate the fused feature data; wherein the fused feature data preserves the geometric accuracy of the point cloud and enhances the semantic information of the image.

7. The supply chain collaborative optimization method based on multimodal data fusion according to claim 1, characterized in that, The generation of supply chain decisions based on the fused feature data specifically includes: The fused feature data is analyzed to extract supply chain-related features, and to obtain inventory features or path features; wherein, the inventory features represent commodity demand trends, and the path features represent logistics obstacle information; The inventory characteristics are processed based on a preset decision model to generate inventory level prediction data, or the path characteristics are processed to generate logistics path planning data. The supply chain decision is generated based on the inventory level forecast data or logistics route planning data.

8. The supply chain collaborative optimization method based on multimodal data fusion according to claim 1, characterized in that, The supply chain decision is output to a preset supply chain management system for execution, specifically including: The supply chain decision data is transmitted to the central supply chain management system. The inventory level forecast data is analyzed in the central supply chain management system to adjust the inventory strategy, or the logistics route planning data is analyzed to update the logistics route and obtain the operation execution results. Record the results of the operation to optimize the subsequent decision generation process.

9. A supply chain collaborative optimization device based on multimodal data fusion, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Acquire multimodal data in the supply chain environment; wherein, the multimodal data includes point cloud data, image data, text data and sensor data, the point cloud data represents the three-dimensional geometric information of the stacked state of goods, the image data represents the visual information of the surveillance video, the text data represents the business information of the order report, and the sensor data represents the physical information of the environment such as temperature, humidity, vibration and light. The multimodal data is preprocessed to a standardized data format to generate standardized multimodal data; Multimodal spatiotemporal alignment is performed on the point cloud data and image data in the standardized multimodal data in the temporal domain to generate time-aligned multimodal data; Based on a preset uncertainty federated learning processing algorithm, the credibility of multiple edge nodes in the supply chain environment is measured and conflict evidence is processed to generate node credibility data. The point cloud data and image data in the time-aligned multimodal data are processed based on a preset cross-modal attention fusion algorithm to generate fused feature data. Supply chain decisions are generated based on the fused feature data; wherein, the supply chain decisions include inventory level forecast data or logistics route planning data; The supply chain decision is output to a preset supply chain management system for execution; wherein, the execution of the operation includes adjusting inventory strategy or optimizing logistics route.

10. A non-volatile computer storage medium for supply chain collaborative optimization based on multimodal data fusion, storing computer-executable instructions, characterized in that, The computer-executable instructions are set as follows: Acquire multimodal data in the supply chain environment; wherein, the multimodal data includes point cloud data, image data, text data and sensor data, the point cloud data represents the three-dimensional geometric information of the stacked state of goods, the image data represents the visual information of the surveillance video, the text data represents the business information of the order report, and the sensor data represents the physical information of the environment such as temperature, humidity, vibration and light. The multimodal data is preprocessed to a standardized data format to generate standardized multimodal data; Multimodal spatiotemporal alignment is performed on the point cloud data and image data in the standardized multimodal data in the temporal domain to generate time-aligned multimodal data; Based on a preset uncertainty federated learning processing algorithm, the credibility of multiple edge nodes in the supply chain environment is measured and conflict evidence is processed to generate node credibility data. The point cloud data and image data in the time-aligned multimodal data are processed based on a preset cross-modal attention fusion algorithm to generate fused feature data. Supply chain decisions are generated based on the fused feature data; wherein, the supply chain decisions include inventory level forecast data or logistics route planning data; The supply chain decision is output to a preset supply chain management system for execution; wherein, the execution of the operation includes adjusting inventory strategy or optimizing logistics route.

Images