Intelligent logistics distribution method based on multi-modal logistics knowledge graph

Abstract

The invention relates to the field of logistics distribution environment perception, in particular to an intelligent logistics distribution method based on a multi-modal logistics knowledge graph. The method comprises the following steps: firstly, acquiring a road image through a visual sensor, and constructing a visual variation observation range; furthermore, a visual characteristic spectrum is selected to extract corresponding visual characteristics in a visual variation observation range, the extracted visual characteristics construct a time domain change curve of the visual characteristics, and a sensitive time domain and a non-sensitive time domain in the driving environment are dynamically captured according to the real-time change condition of the curve. And in a sensitive time domain, image abnormity is determined by comparing the change trend consistency of new energy logistics vehicle vision and radar geometric features. And in a non-sensitive time domain, whether a visual detection target has a corresponding point cloud cluster or not is determined in an interval mode. Therefore, the perception reliability of the new energy logistics vehicle in a complex high-risk period is enhanced, and the system perception reliability, the overall energy efficiency and the resource utilization rate are improved.

Description

Technical Field

[0001] This invention relates to the field of logistics and distribution environment perception, and in particular to a smart logistics and distribution method based on a multimodal logistics knowledge graph. Background Technology

[0002] With the deep integration of IoT, big data, and AI technologies, smart logistics has become a core driver for improving supply chain efficiency and reducing operating costs. Against the backdrop of current "dual-carbon" goals and energy structure transformation, electric delivery vehicles, represented by new energy logistics vehicles, are rapidly gaining popularity. Their range, energy efficiency optimization, and operational reliability place higher demands on the real-time performance and energy efficiency of sensing systems. However, the complex and ever-changing urban distribution environment—such as sudden traffic flow changes, weather impacts, and mixed right-of-way scenarios—poses severe challenges to single-modal environmental sensing methods: while continuous high-load, all-time-domain high-precision sensing can improve safety, it significantly increases the power consumption and computing power burden of new energy vehicles, affecting range and system economy; while simple timed or low-power sensing strategies are insufficient to guarantee sensing reliability in sudden risk scenarios. Therefore, how to achieve dynamic optimization of sensing resources and energy consumption while ensuring the accuracy of environmental sensing has become a key issue that urgently needs to be addressed for the large-scale and intelligent development of new energy logistics fleets.

[0003] Chinese Patent Publication No. CN119225366A discloses a method and device for obstacle detection of unmanned vehicles in open-pit mines. The method includes: receiving pose data detected by inertial navigation systems (INS) installed at four locations on the unmanned vehicle in the open-pit mine, wherein the four locations correspond to the four wheels, and at least one INS is installed at any one location; if the deviation of the pose data detected at any two locations is greater than a preset deviation, determining that at least one of the wheels corresponding to those two locations has run over a protrusion or dent, and identifying wheels with abnormal height based on wheel height data, wherein the deviation of the pose data includes positional deviation and angular deviation, the height data is obtained from the pose data, and the wheels with abnormal height data are those with abnormal height data; and determining the position and size information of the protrusion or dent based on the pose data of the wheels with abnormal height. This solution solves the problem of limited perception capability in existing obstacle detection methods for unmanned vehicles in open-pit mines.

[0004] However, the following problems still exist in the existing technology. During autonomous delivery by unmanned vehicles, visual perception is susceptible to environmental interference, which can lead to false alarms. Furthermore, the system typically requires continuous high-load fusion verification, which may result in misjudgment of obstacles and generate significant redundant calculations during long-term safe driving. This reduces the reliability of the system's perception as well as its overall energy efficiency and resource utilization. Summary of the Invention

[0005] To address this, the present invention provides a smart logistics delivery method based on a multimodal logistics knowledge graph, which overcomes the problems in the prior art where visual perception is susceptible to environmental interference, resulting in false alarms, and the system usually needs to continuously perform high-load fusion verification, which may not only lead to misjudgment of obstacles, but also generate huge redundant calculations during long-term safe driving, thereby reducing the reliability of system perception and the overall energy efficiency and resource utilization.

[0006] To achieve the above objectives, this invention provides a smart logistics delivery method based on a multimodal logistics knowledge graph, comprising: Step S1: Obtain image data from the visual sensors deployed in the logistics and distribution unit, and label boundary features based on the image data; Step S2: Construct the visual anomaly observation range based on the boundary features, select a visual feature map to extract the corresponding visual features within the visual anomaly observation range, the visual feature map includes the route types of several route segments, and the visual features associated with the route types. Step S3: Construct a visual feature temporal variation curve based on the visual features extracted from continuous image data, and capture the sensitive temporal domain based on the changes in the visual feature temporal variation curve. Step S4: Obtain radar data collected by the lidar deployed in the logistics and distribution unit; analyze the radar data according to sensitive and non-sensitive time domains to simultaneously analyze the judgment results for the image data, including... The radar data is analyzed in real time to obtain the geometric features of the lidar, construct the time-domain variation curve of the geometric features, and compare the variation trend of the time-domain variation curve of the visual features with that of the time-domain variation curve of the geometric features to determine whether the image data is abnormal. The radar data is analyzed at intervals to verify whether the target features in the image data have corresponding point cloud clusters, in order to determine whether the image data is abnormal.

[0007] Furthermore, the process of annotating boundary features based on image data includes, Identify the boundaries of passable roads as boundary features and label them.

[0008] Furthermore, the process of constructing the visual anomaly observation range based on the boundary features includes, Inside the boundary features of the passable road, a strip-shaped area of ​​a predetermined width is defined along the extension direction of the passable road; The strip-shaped region was defined as the observation range for visual anomalies.

[0009] Furthermore, the visual feature map includes route types for several route segments, wherein the route types are divided into high-dynamic road segments and low-dynamic road segments based on real-time traffic flow. If the real-time traffic flow is greater than or equal to the preset traffic flow threshold, it is classified as a high-dynamic road segment. If the real-time traffic flow is less than the preset traffic flow threshold, it is classified as a low-dynamic road segment.

[0010] Furthermore, the visual features associated with the route type include, If a road segment is classified as a high-dynamic road segment, the associated visual feature is the percentage of pixels in the non-road surface chromaticity range within the visual variation observation range. If a road segment is classified as low-dynamic, the associated visual feature is the average chromaticity within the visual variation observation range.

[0011] Furthermore, the process of capturing the sensitive time domain based on the changes in the visual feature time-domain variation curve includes, If the percentage of pixels in the non-road surface chromaticity range is greater than or equal to a preset pixel percentage threshold, or if the average chromaticity is greater than or equal to a preset chromaticity threshold, then the corresponding time will be extended bidirectionally for a predetermined duration, and the extended time domain segment will be captured as a sensitive time domain. The time periods in the time domain that were not captured as sensitive time periods in the corresponding time domain of the visual feature time domain change curve are determined as non-sensitive time domains.

[0012] Furthermore, the synchronous analysis's determination result for the image data includes, If the capture is in the sensitive time domain, the radar data is analyzed in real time to obtain the geometric features of the lidar, construct the time domain change curve of the geometric features, and compare the change trend of the time domain change curve of the visual features with that of the time domain change curve of the geometric features to determine whether the image data is abnormal. If it is determined to be a non-sensitive time domain, the radar data is parsed at intervals to verify whether the target features in the image data have corresponding point cloud clusters, in order to determine whether the image data is abnormal.

[0013] The process of comparing the temporal variation curves of visual features and the temporal variation curves of geometric features further includes, Determine the rate of volume change of point cloud clusters within the visual variation observation range, and construct a time-domain variation curve of geometric features based on the rate of volume change; Align the temporal variation curves of visual features and geometric features, and compare the trends and slopes of change within each temporal segment.

[0014] Furthermore, the process of comparing the temporal variation curves of visual features and geometric features to determine whether the image data is abnormal includes: If the changing trends are inconsistent and the difference ratio of the changing slopes is less than the predetermined difference ratio threshold, the image data is determined to be abnormal.

[0015] Furthermore, the process of verifying whether target features in the image data correspond to point cloud clusters to determine whether the image data is abnormal includes, If the target feature in the image data does not have a corresponding point cloud cluster, the image data is determined to be abnormal.

[0016] Compared with existing technologies, this invention first acquires road images using visual sensors to construct a visual anomaly observation range. Then, it selects visual feature maps to extract corresponding visual features within this observation range. The extracted visual features are used to construct a temporal variation curve, and based on the real-time changes of this curve, it dynamically captures sensitive and non-sensitive time domains in the driving environment. Furthermore, within the sensitive time domain, image anomalies are determined by comparing the consistency of the changing trends of the visual and radar geometric features of the new energy logistics vehicle. In the non-sensitive time domain, the method switches to intermittently confirming whether the visually detected target has a corresponding point cloud cluster. This enhances the perception reliability of new energy logistics vehicles during complex and high-risk periods, improving the reliability of the system's perception, as well as overall energy efficiency and resource utilization.

[0017] In particular, this invention extracts corresponding visual features within the observation range of visual anomalies by selecting visual feature maps. The visual feature maps include route types for several route segments and the visual features associated with those route types. Because existing technologies often employ fixed feature analysis strategies in environmental perception, they fail to consider the fundamental differences in core risk patterns across road segments under varying traffic densities. Using the same features for monitoring could lead to a mismatch between the perception model and the actual risk. Therefore, based on real-time traffic flow, road segments are divided into high-dynamic and low-dynamic segments. High-dynamic segments have high real-time traffic volume and rapidly changing road environments, with risks primarily stemming from the rapid intrusion of discrete dynamic obstacles such as vehicles and pedestrians. For high-dynamic segments, associating pixel proportion features within non-road surface chromaticity ranges can directly and sensitively quantify the degree of abnormal object encroachment on the road surface, achieving rapid target detection against interference. Low-dynamic segments, on the other hand, have low real-time traffic volume and relatively stable environments, with the main risk shifting to the deterioration of the road surface itself, such as oil stains, water accumulation, or deceptive changes in large-scale light and shadow. Therefore, for low-dynamic segments, associating average chromaticity features exhibits excellent detection characteristics for global changes such as road surface pollution, water accumulation, or large-scale reflections, stably representing the overall color consistency of the observed area. This differentiated visual feature extraction method can improve the system's perception accuracy and reliability across different road segments while reducing computational resource consumption.

[0018] In particular, this invention analyzes radar data based on both sensitive and non-sensitive time domains to simultaneously analyze the judgment results for image data. Based on the aforementioned visual feature-driven dynamic environmental perception, if lidar data is verified at the same frequency as the visual sensor, it would generate significant redundant power consumption. Differential analysis of radar data is also problematic because the environment is complex or uncertain in the sensitive time domain, potentially distorting visual information or subjecting it to strong interference. Therefore, real-time analysis of radar data is used to acquire lidar geometric features and construct a time-domain variation curve for these features. By comparing the trend of this geometric feature curve with that of the visual feature curve, the visual perception results can be verified. If the trends are consistent, the confirmation of abnormal images is enhanced; if the trends are inconsistent, visual misjudgments can be effectively identified, thus ensuring the reliability of the perception system during critical risk periods. In the stable non-sensitive time domain, visual perception reliability is high, and the probability of sudden changes is low. Therefore, a low-frequency, interval-based radar analysis mode is used to perform lightweight existence verification on the limited number of suspicious targets initially identified by vision. By confirming whether the target has a corresponding point cloud cluster, purely visual false alarms, such as textures or lighting effects, can be eliminated. This strategy avoids the unnecessary computational burden caused by continuously processing massive point cloud data, and improves the reliability of system perception, as well as overall energy efficiency and resource utilization through differentiated parsing and verification.

[0019] In particular, this invention analyzes radar data in real time to obtain the geometric features of the lidar, constructs a time-domain variation curve of the geometric features, and compares the changing trends of the time-domain variation curve of the visual features with those of the geometric features to determine whether the image data is abnormal. Since the sensitive time domain is a highly dynamic road section, the risk at this time is not the appearance of a single target, but a dynamic process that is ongoing and of unknown nature. For example, it could be a gradually spreading fog, accumulating water, a slowly approaching vehicle or pedestrian, or visual noise caused by violent shaking of light and shadow. Faced with such continuously changing signals, which are inevitably accompanied by continuous changes in the volume of the radar point cloud clusters, the time-domain variation curve of the geometric features, representing the dynamics of the physical volume, is synchronously compared with the time-domain variation curve of the visual features, representing the dynamics of the visual appearance. If the trends are consistent, for example, if the proportion of visually abnormal pixels increases while the radar volume change rate also increases, it proves that the visual perception is of a real physical dynamic event. If the trends are inconsistent, it proves that the visual signal is a false change caused by non-physical factors such as light and shadow, thus eliminating misjudgment and improving the reliability of the system's perception as well as the overall energy efficiency and resource utilization. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating the steps of a smart logistics delivery method based on a multimodal logistics knowledge graph, as an embodiment of the invention. Figure 2A logical block diagram of the visual features associated with the route type in the embodiments of the invention; Figure 3 This is a logic decision diagram for capturing the sensitive time domain in an embodiment of the invention; Figure 4 This is a logic block diagram illustrating the analysis of radar data based on sensitive and non-sensitive time domains, as described in an embodiment of the invention. Detailed Implementation

[0021] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0022] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0023] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the term "connection" should be interpreted broadly. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0024] Please see Figure 1 The diagram illustrates the steps of a smart logistics delivery method based on a multimodal logistics knowledge graph, as described in an embodiment of the invention. The smart logistics delivery method based on a multimodal logistics knowledge graph, as described in this embodiment, includes: Step S1: Obtain image data from the visual sensors deployed in the logistics and distribution unit, and label boundary features based on the image data; Step S2: Construct the visual anomaly observation range based on the boundary features, select a visual feature map to extract the corresponding visual features within the visual anomaly observation range, the visual feature map includes the route types of several route segments, and the visual features associated with the route types. Step S3: Construct a visual feature temporal variation curve based on the visual features extracted from continuous image data, and capture the sensitive temporal domain based on the changes in the visual feature temporal variation curve. Step S4: Obtain radar data collected by the lidar deployed in the logistics and distribution unit; analyze the radar data according to sensitive and non-sensitive time domains to simultaneously analyze the judgment results for the image data, including... The radar data is analyzed in real time to obtain the geometric features of the lidar, construct the time-domain variation curve of the geometric features, and compare the variation trend of the time-domain variation curve of the visual features with that of the time-domain variation curve of the geometric features to determine whether the image data is abnormal. The radar data is analyzed at intervals to verify whether the target features in the image data have corresponding point cloud clusters, in order to determine whether the image data is abnormal.

[0025] In practice, there are no restrictions on the method of acquiring image data from the visual sensors deployed in the logistics and distribution unit. Data can be collected by one or more monocular cameras, binocular stereo vision systems, or panoramic surround-view cameras installed at the front, side, or top of the logistics and distribution unit. As long as the data can continuously acquire digital image sequences containing the passable road areas in front and to the sides, this will not be elaborated further.

[0026] Specifically, the logistics delivery unit can be a small logistics vehicle equipped with an autonomous driving system, which will not be elaborated further.

[0027] Specifically, the process of annotating boundary features based on image data includes, Identify the boundaries of passable roads as boundary features and label them.

[0028] In practice, there are no restrictions on how the boundaries of passable roads are identified as boundary features and annotated. Computer vision technology, such as edge detection algorithms, can be used to automatically identify and annotate road boundaries in images, which will not be elaborated further.

[0029] Specifically, the process of constructing the visual anomaly observation range based on the boundary features includes, Inside the boundary features of the passable road, a strip-shaped area of ​​a predetermined width is defined along the extension direction of the passable road; The strip-shaped region was defined as the observation range for visual anomalies.

[0030] In practice, the preset width is a predetermined multiple of the width of the logistics delivery unit. Typically, the predetermined multiple is selected within the range of [1.25, 1.5], and is preferably 1.35 in practice.

[0031] Specifically, the visual feature map includes route types for several route segments, wherein the route types are divided into high-dynamic road segments and low-dynamic road segments based on real-time traffic flow. If the real-time traffic flow is greater than or equal to the preset traffic flow threshold, it is classified as a high-dynamic road segment. If the real-time traffic flow is less than the preset traffic flow threshold, it is classified as a low-dynamic road segment.

[0032] In implementation, the purpose of the preset traffic flow threshold is to characterize the traffic flow status of a road segment, thereby distinguishing the dynamic characteristics of different road segments. The preset traffic flow threshold is predetermined; those skilled in the art can determine the average real-time traffic flow by statistically analyzing a large amount of actual traffic flow data. This represents the stability of the road segment's traffic flow under normal conditions. To represent the dynamic changes in the road segment's traffic flow, the preset traffic flow threshold is set as the product of the average value and a traffic flow accuracy coefficient. Typically, the accuracy coefficient is selected within the range [0.75, 1.2], and is preferably 0.85 in implementation.

[0033] In practice, the visual feature map does not simply store static route type labels, but pre-sets feature selection mapping rules based on real-time traffic flow: if the traffic flow is greater than or equal to a preset traffic flow threshold, it is classified as a high-dynamic road segment; if the traffic flow is less than the preset traffic flow threshold, it is classified as a low-dynamic road segment.

[0034] Please see Figure 2 The diagram shown is a logical block diagram of the visual features associated with a route type according to an embodiment of the invention. Specifically, the visual features associated with the route type include: If a road segment is classified as a high-dynamic road segment, the associated visual feature is the percentage of pixels in the non-road surface chromaticity range within the visual variation observation range. If a road segment is classified as low-dynamic, the associated visual feature is the average chromaticity within the visual variation observation range.

[0035] This invention extracts corresponding visual features within the observation range of visual anomalies by selecting visual feature maps. The visual feature maps include route types for several route segments and the visual features associated with those route types. Because existing technologies often employ fixed feature analysis strategies in environmental perception, they fail to consider the fundamental differences in core risk patterns across road segments under varying traffic densities. Using the same features for monitoring may lead to a mismatch between the perception model and the actual risk. Therefore, based on real-time traffic flow, road segments are divided into high-dynamic and low-dynamic segments. High-dynamic segments have high real-time traffic volume and rapidly changing road environments, with risks primarily stemming from the rapid intrusion of discrete dynamic obstacles such as vehicles and pedestrians. For high-dynamic segments, associating pixel proportion features within non-road surface chromaticity ranges can directly and sensitively quantify the degree of abnormal object encroachment on the road surface, achieving rapid target detection against interference. Low-dynamic segments, on the other hand, have low real-time traffic volume and relatively stable environments, with the main risk shifting to the deterioration of the road surface itself, such as oil stains, water accumulation, or deceptive changes in large-scale light and shadow. Therefore, for low-dynamic segments, associating average chromaticity features exhibits excellent detection characteristics for global changes such as road surface pollution, water accumulation, or large-scale reflections, stably representing the overall color consistency of the observed area. This differentiated visual feature extraction method can improve the system's perception accuracy and reliability across different road segments while reducing computational resource consumption.

[0036] Please see Figure 3 As shown, it is a logic decision diagram for capturing the sensitive time domain according to an embodiment of the invention. Specifically, the process of capturing the sensitive time domain based on the changes in the visual feature time domain change curve includes, If the percentage of pixels in the non-road surface chromaticity range is greater than or equal to a preset pixel percentage threshold, or if the average chromaticity is greater than or equal to a preset chromaticity threshold, then the corresponding time will be extended bidirectionally for a predetermined duration, and the extended time domain segment will be captured as a sensitive time domain. The time periods in the time domain that were not captured as sensitive time periods in the corresponding time domain of the visual feature time domain change curve are determined as non-sensitive time domains.

[0037] In implementation, the purpose of the pixel proportion threshold is to characterize significant changes in visual features to distinguish between normal and abnormal states. The pixel proportion threshold is predetermined; those skilled in the art can determine the mean pixel proportion by performing statistical analysis on a large amount of actual image data to represent the stability of visual features under normal conditions. To represent significant changes in visual features, the pixel proportion threshold is set to a predetermined multiple of the mean. Typically, the predetermined multiple is selected within the range of [1.05, 1.35], and is preferably 1.25 in implementation.

[0038] In implementation, the purpose of the chromaticity threshold is to characterize the allowable deviation limit of the average color stability of the road surface within the observation range of the visual anomaly. The chromaticity threshold is a predetermined value. Those skilled in the art can calculate the statistical average of the average chromaticity within the observation range of the visual anomaly based on collected historical or experimental image data. This statistical average represents the baseline chromaticity of the visual characteristic of the road under normal and clean conditions, serving as a criterion for identifying overall road surface anomalies. To avoid misjudgments caused by gradual changes in illumination or minor color differences, the chromaticity threshold can be set as the product of the statistical average and the chromaticity error coefficient. Typically, the chromaticity error coefficient is selected within the range of [1.05, 1.4], and is preferably 1.3 in implementation.

[0039] In practice, the predetermined duration is usually selected within the range of [0.03s, 0.08s], and is preferably 0.05s.

[0040] Please see Figure 4 The diagram shown is a logical block diagram illustrating the analysis of radar data based on sensitive and non-sensitive time domains according to an embodiment of the invention. Specifically, the synchronous analysis, based on the determination results of the image data, includes: If the capture is in the sensitive time domain, the radar data is analyzed in real time to obtain the geometric features of the lidar, construct the time domain change curve of the geometric features, and compare the change trend of the time domain change curve of the visual features with that of the time domain change curve of the geometric features to determine whether the image data is abnormal. If it is determined to be a non-sensitive time domain, the radar data is parsed at intervals to verify whether the target features in the image data have corresponding point cloud clusters, in order to determine whether the image data is abnormal.

[0041] This invention analyzes radar data based on both sensitive and non-sensitive time domains to simultaneously analyze judgment results for image data. Based on the aforementioned visual feature-driven dynamic environment perception, if lidar data is verified at the same frequency as the visual sensor, it would generate significant redundant power consumption. Differential analysis of radar data is also problematic because the environment is complex or uncertain in the sensitive time domain, potentially distorting visual information or subjecting it to strong interference. Therefore, real-time analysis of radar data is used to acquire lidar geometric features and construct a time-domain variation curve for these features. By comparing the trend of this geometric feature curve with that of the visual feature curve, the visual perception results can be verified. If the trends are consistent, the confirmation of abnormal images is enhanced; if the trends are inconsistent, visual misjudgments can be effectively identified, ensuring the reliability of the perception system during critical risk periods. In the stable non-sensitive time domain, visual perception reliability is high, and the probability of sudden changes is low. Therefore, a low-frequency, interval-based radar analysis mode is adopted to perform lightweight existence verification on a limited number of suspicious targets initially identified by vision. By confirming the existence of corresponding point cloud clusters for the target, purely visual false alarms, such as those caused by textures or lighting effects, can be eliminated. This strategy avoids the unnecessary computational burden caused by continuously processing massive point cloud data, and improves the reliability of system perception, as well as overall energy efficiency and resource utilization through differentiated parsing and verification.

[0042] Specifically, the process of comparing the changing trends of the visual feature temporal variation curve and the geometric feature temporal variation curve includes, Determine the rate of volume change of point cloud clusters within the visual variation observation range, and construct a time-domain variation curve of geometric features based on the rate of volume change; Align the temporal variation curves of visual features and geometric features, and compare the trends and slopes of change within each temporal segment.

[0043] In practice, the changing trends of the visual feature time-domain change curve and the geometric feature time-domain change curve within the same time-domain segment are compared. The changing trends are divided into rising and falling, and the slope of the change is the average slope of the corresponding curve segment within the time-domain segment.

[0044] In practice, there is no limitation on the method for determining the volume change rate of point cloud clusters within the visual anomaly observation range. All point cloud clusters located within the visual anomaly observation range can be extracted using a clustering algorithm based on spatial distance, and then the volume change rate of the bounding box of the point cloud clusters or the volume of the three-dimensional convex hull of the point cloud clusters can be calculated. As long as it can accurately reflect the volume change of the point cloud clusters, it is acceptable. This will not be elaborated further.

[0045] Specifically, the process of comparing the changing trends of visual feature temporal variation curves with those of geometric feature temporal variation curves to determine whether image data is abnormal includes, If the changing trends are inconsistent and the difference ratio of the changing slopes is less than the predetermined difference ratio threshold, the image data is determined to be abnormal.

[0046] The difference ratio threshold is predetermined. Radar data under normal image data conditions are recorded in advance. Visual feature time-domain change curves and geometric feature time-domain change curves are constructed. The average difference ratio of the corresponding change slopes in several time-domain segments is recorded. The average difference ratio is amplified by a predetermined factor as the difference ratio threshold. The predetermined factor is selected in the interval [1.15, 1.2].

[0047] This invention analyzes radar data in real time to obtain the geometric features of the lidar, constructs a time-domain variation curve of the geometric features, and compares the changing trends of the visual feature time-domain variation curve with those of the geometric feature time-domain variation curve to determine whether the image data is abnormal. Since the sensitive time domain is a highly dynamic road section, the risk is not the appearance of a single target, but an ongoing, undefined dynamic process. For example, it could be a gradually spreading fog, accumulating water, a slowly approaching vehicle or pedestrian, or visual noise caused by violent light and shadow fluctuations. Faced with such continuously changing signals, which are inevitably accompanied by continuous changes in the volume of the radar point cloud clusters, the time-domain variation curve of the geometric features, representing the dynamics of the physical volume, is synchronously compared with the time-domain variation curve of the visual features, representing the dynamics of the visual appearance. If the changing trends are consistent—for example, if the proportion of visually abnormal pixels increases while the radar volume change rate also increases—it proves that the visual perception is of a real physical dynamic event. If the changing trends are inconsistent, it proves that the visual signal is a false change caused by non-physical factors such as illumination or shadows, thus eliminating misjudgments and improving the reliability of the system's perception, as well as overall energy efficiency and resource utilization.

[0048] Specifically, the process of verifying whether target features in the image data correspond to point cloud clusters in order to determine whether the image data is abnormal includes, If the target feature in the image data does not have a corresponding point cloud cluster, the image data is determined to be abnormal.

[0049] In practice, preferably, the overlap between the bounding box of the point cloud cluster within the visual variation observation range and the projection area can be calculated. If the overlap is greater than a preset overlap threshold, it is determined that a corresponding point cloud cluster exists.

[0050] The purpose of the overlap threshold is to characterize the degree of matching between point cloud clusters and image target features. The overlap threshold is predetermined. Typically, the overlap threshold is selected within the range [0.7, 0.9], and preferably 0.8 in practice.

[0051] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A smart logistics delivery method based on a multimodal logistics knowledge graph, characterized in that, include, Acquire image data from the vision sensors deployed in the logistics and distribution unit, and label boundary features based on the image data; Based on the boundary features, a visual anomaly observation range is constructed. A visual feature map is selected to extract the corresponding visual features within the visual anomaly observation range. The visual feature map includes the route types of several route segments and the visual features associated with the route types. Based on the visual features extracted from continuous image data, a temporal variation curve of visual features is constructed, and the sensitive temporal domain is captured based on the changes in the temporal variation curve of visual features. The system acquires radar data collected by lidar deployed in logistics and distribution units, analyzes the radar data based on sensitive and non-sensitive time domains, and simultaneously analyzes the judgment results for image data. include, The radar data is analyzed in real time to obtain the geometric features of the lidar, construct the time-domain variation curve of the geometric features, and compare the variation trend of the time-domain variation curve of the visual features with that of the time-domain variation curve of the geometric features to determine whether the image data is abnormal. The radar data is analyzed at intervals to verify whether the target features in the image data have corresponding point cloud clusters, in order to determine whether the image data is abnormal.

2. The intelligent logistics distribution method based on multimodal logistics knowledge graph according to claim 1, characterized in that, The process of annotating boundary features based on image data includes: Identify the boundaries of passable roads as boundary features and label them.

3. The intelligent logistics distribution method based on multimodal logistics knowledge graph as described in claim 1, characterized in that, The process of constructing the visual anomaly observation range based on the boundary features includes: Inside the boundary features of the passable road, a strip-shaped area of ​​a predetermined width is defined along the extension direction of the passable road; The strip-shaped region is defined as the observation range for visual anomalies.

4. The intelligent logistics distribution method based on multimodal logistics knowledge graph according to claim 1, characterized in that, The visual feature map includes route types for several route segments, wherein the route types are divided into high-dynamic road segments and low-dynamic road segments based on real-time traffic flow. If the real-time traffic flow is greater than or equal to the preset traffic flow threshold, it is classified as a high-dynamic road segment. If the real-time traffic flow is less than the preset traffic flow threshold, it is classified as a low-dynamic road segment.

5. The intelligent logistics distribution method based on multimodal logistics knowledge graph according to claim 1, characterized in that, The visual features associated with the route type include, If a road segment is classified as a high-dynamic road segment, the associated visual feature is the percentage of pixels in the non-road surface chromaticity range within the visual variation observation range. If a road segment is classified as low-dynamic, the associated visual feature is the average chromaticity within the visual variation observation range.

6. The intelligent logistics distribution method based on multimodal logistics knowledge graph according to claim 5, characterized in that, The process of capturing the sensitive time domain based on the changes in the time domain variation curve of visual features includes: If the percentage of pixels in the non-road surface chromaticity range is greater than or equal to a preset pixel percentage threshold, or if the average chromaticity is greater than or equal to a preset chromaticity threshold, then the corresponding time will be extended bidirectionally for a predetermined duration, and the extended time domain segment will be captured as a sensitive time domain. The time periods in the time domain that were not captured as sensitive time periods in the corresponding time domain of the visual feature time domain change curve are defined as non-sensitive time domains.

7. The intelligent logistics distribution method based on multimodal logistics knowledge graph according to claim 1, characterized in that, The synchronous analysis is based on the judgment results of the image data. include, If the capture is in the sensitive time domain, the radar data is analyzed in real time to obtain the geometric features of the lidar, construct the time domain change curve of the geometric features, and compare the change trend of the time domain change curve of the visual features with that of the time domain change curve of the geometric features to determine whether the image data is abnormal. If it is determined to be a non-sensitive time domain, the radar data is parsed at intervals to verify whether the target features in the image data have corresponding point cloud clusters, in order to determine whether the image data is abnormal.

8. The intelligent logistics distribution method based on multimodal logistics knowledge graph according to claim 7, characterized in that, The process of comparing the temporal variation curves of visual features with those of geometric features includes... Determine the rate of volume change of point cloud clusters within the visual variation observation range, and construct a time-domain variation curve of geometric features based on the rate of volume change; Align the temporal variation curves of visual features and geometric features, and compare the trends and slopes of change within each temporal segment.

9. The intelligent logistics distribution method based on multimodal logistics knowledge graph according to claim 8, characterized in that, The process of comparing the temporal variation curves of visual features with those of geometric features to determine whether the image data is abnormal includes the following steps: If the changing trends are inconsistent and the difference ratio of the changing slopes is less than the predetermined difference ratio threshold, the image data is determined to be abnormal.

10. The intelligent logistics distribution method based on multimodal logistics knowledge graph according to claim 7, characterized in that, The process of verifying whether target features in the image data correspond to point cloud clusters to determine whether the image data is abnormal includes, If the target feature in the image data does not have a corresponding point cloud cluster, the image data is determined to be abnormal.

Images