Omnidirectional deep vision data fusion method and system based on supervised learning

Abstract

The application discloses an omnidirectional deep visual data fusion method and system based on supervised learning, relates to the technical field of image recognition, and comprises the following steps: acquiring multiple original depth frames and sensor state identifiers, establishing an ontology coordinate system and generating a direction index, forming a candidate view angle set according to spatial sectors, extracting depth features and view angle quality states, generating sector fusion features through dynamic view angle gating and grouping coding, outputting omnidirectional distance vectors and direction reliabilities by using a supervised fusion model, generating a marked compensation distance when candidate view angles are insufficient or the direction reliability does not meet the condition, and packaging standardized perception data packets; in the training stage, a simulation ray hitting distance or a high-precision offline reconstruction result is used as a teacher label, and the scheme can provide reviewable omnidirectional distance inputs for obstacle avoidance control, navigation planning and motion strategy networks.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and more specifically to a method and system for omnidirectional deep visual data fusion based on supervised learning. Background Technology

[0002] Omnidirectional depth vision data fusion refers to organizing data collected by multiple depth vision sensors deployed at different locations on the device body into an environmental distance representation corresponding to the device body's coordinate system. This data is then used by the robot for obstacle avoidance, navigation planning, posture adjustment, or motion strategy networks. Existing solutions typically use point cloud registration, depth map stitching, or 2.5-dimensional height maps as input, which can provide basic environmental information when the field of view coverage is narrow or the scene is simple.

[0003] In complex terrain and dynamic obstacle environments, directly inputting the original depth map will increase the processing burden of the downstream network. Two-dimensional half-height maps are difficult to represent obstacles above, to the side and behind. Multi-sensor calibration and registration based on explicit geometric models are easily affected by sensor edge distortion, short-term frame loss, lens damage and equipment vibration. If abnormal viewpoints are still entered into the fusion result with fixed weights, it is easy to cause local jumps or gaps in the omnidirectional distance output. Summary of the Invention

[0004] The purpose of this invention is to provide a supervised learning-based omnidirectional depth vision data fusion method and system, which is used to form a standardized perception data package with orientation index, orientation confidence and abnormal orientation marker when there are distortions, occlusions, missing or short-term instability in multi-channel depth vision inputs. This solves the problems in the background technology of multi-view fusion relying on explicit registration, inconsistent output format and difficulty in isolating the influence of abnormal viewpoints.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a supervised learning-based omnidirectional depth vision data fusion method, applied to an omnidirectional depth vision fusion device, the method comprising:

[0006] Acquire multiple raw depth frames collected by multiple depth vision sensors arranged in different spatial positions on the device body, and write them into the sensor status identifier;

[0007] Establish a body coordinate system based on the device's sensing center, generate multiple directional indices, and divide the spatial sectors;

[0008] Determine the candidate viewpoint set for each spatial sector based on the field of view coverage relationship;

[0009] Extract the depth features and view quality state of each candidate view in the candidate view set, and dynamically gating the candidate view based on the view quality state to generate sector fusion features;

[0010] The supervised fusion model outputs an omnidirectional distance vector and directional confidence based on sector fusion features.

[0011] When there are insufficient candidate viewpoints or the orientation confidence does not meet the conditions, a compensated distance with compensation markers is generated, and a perception data packet containing omnidirectional distance vector, orientation confidence and abnormal orientation markers is output.

[0012] Further optimization involves determining the candidate viewpoint set for each spatial sector based on the field-of-view coverage relationship, including:

[0013] Obtain the installation orientation, field of view range, effective distance range, and historical effective field of view relationship of each depth vision sensor;

[0014] Based on the correspondence between multiple directional indices and the field of view range of each depth vision sensor, the candidate sensor number corresponding to each spatial sector is determined.

[0015] Candidate sensors that are in a state of frame loss, abnormal exposure, or depth saturation are removed based on the sensor status identifier, and the depth frames corresponding to the retained candidate sensors are used as the candidate viewpoint set.

[0016] Further preferably, based on the correspondence between multiple directional indices and the field of view range of each depth vision sensor, the candidate sensor number corresponding to each spatial sector is determined, including:

[0017] Divide multiple directional indices into spatial sectors according to their front-back, left-right, up-down, or combinations thereof;

[0018] For each spatial sector, statistics are compiled on depth vision sensors that can cover the orientation index within that spatial sector;

[0019] When multiple depth vision sensors can cover the same spatial sector, the candidate order is determined based on the coverage angle, effective distance, and historical effective field of view relationship, and a candidate view grouping record is formed.

[0020] Further optimization involves extracting the depth features and view quality status of each candidate viewpoint from the candidate viewpoint set, including:

[0021] The original depth frames corresponding to the candidate viewpoints are cropped, missing points are filled, and scale is unified to obtain viewpoint-level depth features. Based on the missing proportion, depth mutation proportion, edge region proportion, consistency with adjacent candidate viewpoints, and historical stability of the candidate viewpoints in the current sampling window, the corresponding viewpoint quality status is generated.

[0022] Further preferred, dynamic gating of candidate views based on view quality status and generation of sector fusion features, including:

[0023] The participation intensity of candidate viewpoints within the corresponding spatial sector is determined based on the viewpoint quality status.

[0024] When a candidate perspective has an increased missing percentage, a concentrated deep mutation, or is inconsistent with adjacent candidate perspectives, the participation intensity of that candidate perspective should be reduced.

[0025] When a candidate viewpoint aligns with feedback from neighboring sectors and historical distance sequences, increase the participation intensity of that candidate viewpoint.

[0026] After completing the intensity adjustment, multiple candidate perspectives are input into the group encoder to obtain the sector fusion features of the corresponding spatial sectors.

[0027] In a further preferred embodiment, before outputting the omnidirectional distance vector and orientation confidence based on sector fusion features using a supervised fusion model, the method further includes:

[0028] Construct training samples, which include multiple raw depth frames generated in the simulation environment, high-precision offline reconstruction results, or measured depth data in scenes with known distances;

[0029] Distance teacher labels are generated based on the hit results between the directional ray emitted outward from the origin of the body coordinate system and the obstacle or environment boundary, and visibility labels, occlusion labels, and credibility labels are generated for the directional index.

[0030] In a further preferred embodiment, after constructing the training samples, the method further includes adding occlusion samples, contamination samples, field-of-view edge distortion samples, and sensor short-term failure samples to the training samples. Based on the distance results, confidence results, and abnormal direction markers output by the supervised fusion model, the parameters of the supervised fusion model are updated with the differences between them and the corresponding teacher labels, so that the supervised fusion model learns the mapping relationship from sector fusion features to omnidirectional distance vectors and directional confidence.

[0031] Further preferred methods include generating a compensated distance with a compensation marker when there are insufficient candidate viewpoints or the direction confidence does not meet the conditions, including:

[0032] Determine the spatial sector to which the direction index to be compensated belongs and its adjacent spatial sectors;

[0033] Read the distance changes between adjacent spatial sectors, the historical distance sequence of the direction index to be compensated, and the continuity of device movement;

[0034] Compensated distances are generated based on distance changes, historical distance sequences, and device motion continuity, and then output by binding the compensated distances with compensation flags and low confidence flags.

[0035] Further preferably, the output includes a perception data packet containing an omnidirectional distance vector, direction confidence, and abnormal direction markers, which includes: encapsulating the distance value, direction confidence, spatial sector status, compensation marker, and abnormal direction marker corresponding to each direction index into a standardized perception data packet, and sending the standardized perception data packet to the obstacle avoidance control module, navigation planning module, or motion strategy network, so that the downstream module can determine the usage method of the corresponding direction distance based on the direction confidence and abnormal direction markers.

[0036] A supervised learning-based omnidirectional depth vision data fusion system is disclosed. The system includes a multi-view depth acquisition module, a direction index establishment module, a spatial sector grouping module, a view quality generation module, a dynamic view gating module, a group coding module, a supervised fusion module, a missing direction compensation module, and a perception data output module. The multi-view depth acquisition module is used to acquire multiple raw depth frames. The direction index establishment module is used to establish a ontological coordinate system and generate a direction index. The spatial sector grouping module is used to determine the candidate view set. The view quality generation module is used to generate view quality states. The dynamic view gating module and the group coding module are used to generate sector fusion features. The supervised fusion module is used to output omnidirectional distance vectors and direction confidence. The missing direction compensation module is used to generate labeled compensation distances. The perception data output module is used to output standardized perception data packets.

[0037] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0038] Compared to existing solutions that primarily rely on point cloud stitching, fixed geometric registration, or directly inputting the original depth map into the downstream control network, this invention first organizes multiple original depth frames into spatial sectors and candidate view sets. Then, it uses view quality status to drive dynamic view gating, enabling abnormal situations such as short-term sensor frame loss, field-of-view edge distortion, lens contamination, and depth saturation to be recorded and suppressed at the sector level. Normal view and feedback from adjacent sectors can continue to participate in fusion. The source, quality changes, and participation intensity of candidate views can all be retained with the sensing data packets. This reduces the continuous impact of abnormal view on the omnidirectional distance vector and provides data basis for subsequent verification. This invention outputs an omnidirectional distance vector, a direction confidence vector, a spatial sector state table, and an anomaly direction markers during the deployment phase. The supervised fusion model first receives the sector fusion features, then generates the direction distance and confidence. When there are insufficient candidate views or the direction confidence does not meet the conditions, it generates a compensated distance with a compensation marker. Finally, it outputs a standardized perception data packet. Since each direction distance corresponds to a clear direction index, confidence source, candidate view record, and anomaly marker, the downstream obstacle avoidance control, navigation planning, or motion strategy network can select the usage method according to the confidence, which is conducive to decoupling the perception module and the control module, and also facilitates the replacement of the fusion model for different robot platforms. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0040] Figure 1 This is a schematic diagram of the overall system architecture of the present invention.

[0041] Figure 2 This is a flowchart of the supervised learning omnidirectional fusion method of the present invention.

[0042] Figure 3 This is a schematic diagram of the spatial sector of the present invention.

[0043] Figure 4 This is a schematic diagram of the dynamic perspective gating of the present invention.

[0044] Figure 5 This is a schematic diagram of the supervised training framework of the present invention.

[0045] Figure 6 This is a schematic diagram illustrating the missing direction compensation of the present invention.

[0046] Figure 7 This is a schematic diagram showing the arrangement and deployment of the depth sensor according to the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] To facilitate understanding and implementation of the present invention by those skilled in the art, the following description is provided. Figures 1 to 6 The specific implementation of the present invention is described. This part focuses on the data flow after multiple depth frames enter the omnidirectional depth vision fusion device. It focuses on explaining the correspondence between sensor status identifiers, spatial sectors, candidate view sets, view quality status, dynamic view gating, supervised fusion model, missing direction compensation and perception data packets, so that the processing chain can be implemented on different robot platforms.

[0049] The term "device body" as used in this application refers to a drone, mobile robot, quadruped robot, inspection vehicle, warehousing and handling equipment, or other carrier with autonomous movement requirements. The term "depth vision sensor" refers to a binocular depth camera, structured light depth camera, time-of-flight depth camera, or a solid-state ranging vision device capable of outputting depth frames. The term "omnidirectional depth vision fusion device" refers to a computing unit on the device body, and also to an edge processing unit that is communicatively connected to the device body. The processing results are read by an obstacle avoidance control module, a navigation planning module, or a motion strategy network.

[0050] The direction index referred to in this application refers to the spatial direction number emitted outward from the device's sensing center. The direction index does not directly represent the image pixel position, but rather the environmental direction that the downstream control module can read. The direction index is obtained by spherical uniform sampling, latitude and longitude grid sampling, equal area polyhedron sampling, or task-focused area densified sampling. The spatial sector is used to organize adjacent direction indices into processing units, and the candidate view set is used to record the sensor sources that can provide depth information for the processing unit.

[0051] To avoid insufficient disclosure, the following is an implementable recording method for several status fields in this application. This recording method does not limit the system to using the same field names. As long as it can generate the view quality status based on multiple original depth frames and sensor status identifiers, and can be used for dynamic view gating and missing direction compensation, it falls within the scope of implementation disclosed in this application.

[0052] To facilitate the explanation of the data transmission relationship after multiple raw depth frames enter the omnidirectional depth vision fusion device, and to provide a feasible recording basis for sensor status identifiers, viewpoint quality status, spatial sector status, and abnormal direction markers, a status field recording table 1 is presented below. This table describes the field content, acquisition method, and purpose of various status fields. In actual systems, fields may be added, removed, or merged based on the number of sensors, the number of direction indices, and the requirements of downstream control interfaces.

[0053] Table 1: Status Field Record Table.

[0054] Record object Field Examples How to obtain use Sensor status indicator Sensor number, sampling time, exposure status, frame drop status, depth saturation status Generated by acquisition driver, sensor control interface and sampling buffer Used to determine whether a candidate viewpoint enters a spatial sector. Viewpoint quality status Missing percentage, mutation percentage, marginal region proportion, consistency of adjacent viewpoints, historical stability The depth is obtained by comparing the effective depth points within the current sampling window with the depth changes and the neighboring viewpoints. Used to generate candidate viewpoint participation intensity Spatial sector status Sector number, direction index range, number of candidate viewpoints, number of low-quality viewpoints, number of compensated directions Recorded by the spatial sector grouping module and the output module Source of downstream control review distance Abnormal direction marker Short-term missing data, persistent anomalies, self-occlusion, external occlusion, low-reliability compensation Based on confidence level, compensation logic and sensor status are jointly generated. Used to distinguish between normal estimated distance and compensated distance.

[0055] Using the field recording method shown in Table 1, the system can transmit the acquisition end status, candidate view quality, spatial sector status, and abnormal direction markers in the same processing chain, making the dynamic view gating, missing direction compensation, and standardized sensing data packet output have traceable data sources. During formal deployment, downstream modules only read the omnidirectional distance vector and direction confidence vector. During debugging and maintenance, the status field is further read to determine whether the distance in a certain direction is a normal estimate, a low-confidence estimate, or a compensation output.

[0056] The above fields are output along with the standardized perception data packet and are also stored in the debug cache or runtime log. During formal deployment, the downstream control module only reads the omnidirectional distance vector and the direction confidence vector. During maintenance and verification, engineers further read the spatial sector status table and abnormal direction markers to determine whether the distance in a certain direction comes from normal estimation, single-view estimation, or missing compensation.

[0057] Example 1

[0058] See Figure 1 and Figure 2 In one embodiment of a hexahedral robot platform, a depth vision sensor is respectively set on the front, rear, left, right, top, and bottom sides of the device body. The six depth vision sensors output raw depth frames in a synchronous or near-synchronous sampling manner. Each raw depth frame carries the sensor number, sampling time, exposure status, frame loss status, depth saturation status, and device posture reference information. The sampling time is written by the main clock of the device body and also by the acquisition driver when receiving frames.

[0059] After receiving multiple raw depth frames, the omnidirectional depth vision fusion device does not first stitch the depth frames into a complete point cloud map. Instead, it first writes each depth frame into a multi-view sampling buffer. The sampling buffer is indexed according to the sampling time and sensor number. If a depth frame is late in the same sampling window, the system waits for the depth frame within a preset fault tolerance range. If it is not received after the fault tolerance range is exceeded, the sensor is written into a short-term missing state. The short-term missing state will participate in the subsequent candidate view set screening.

[0060] The device sensing center is set at the device's geometric center, the center of the inertial measurement unit, or the preset sensor coordinate center. The system uses the device sensing center as the origin of the body coordinates and generates direction indexes according to the spherical uniform sampling method. For example, 128 direction indexes are generated, or the number can be set to 64, 96, 160, or other numbers depending on the downstream control frequency and the device's computing power. The more direction indexes there are, the finer the output distance vector becomes, and the computational task increases accordingly.

[0061] After the direction index is generated, the spatial sector grouping module divides the multiple direction indices into 8 spatial sectors according to the relationships of front-up-right, front-up-left, front-down-right, front-down-left, back-up-right, back-up-left, back-down-right, and back-down-left. Each spatial sector records the sector number, direction index range, candidate sensor number, neighboring sector number, and historical effective field of view relationship. The historical effective field of view relationship is preset by the installation file and is also updated according to the fixed distance plate sample during deployment verification.

[0062] Taking the upper right sector as an example, the candidate view set for this sector includes the forward depth sensor, the right depth sensor, and the upward depth sensor. When both the forward and right depth sensors can observe the channel edge, the system compares the consistency of their distance changes in the overlapping area and writes the consistency result into the view quality record of this sector. If the upward depth sensor can only provide edge field-of-view data, this view can still be retained as an auxiliary candidate view. Figure 7 As shown.

[0063] When the robot passes through a narrow passage, the forward view may provide a longer distance at the depth of the passage, while the lateral view may provide a shorter distance at the edge of the passage. The dynamic view gating module does not simply select one of the depth frames, but allocates the participation intensity according to the view quality state, neighboring sector feedback and historical stability, so that the sector fusion feature retains the forward passage distance and lateral gap changes at the same time, avoiding mistaking the lateral wall as a close obstacle in front.

[0064] If the right-side depth sensor experiences depth saturation due to proximity to the wall, the view quality generation module generates a low-quality state based on the proportion of the saturated area and the consistency with adjacent viewpoints. The dynamic view gating module reduces the participation intensity of the right-side viewpoint in the upper front right sector and the lower front right sector, and synchronously transmits the low-quality state to the missing direction compensation module. The system can still use the forward viewpoint, the upward viewpoint, and the historical distance sequence to generate the corresponding output results.

[0065] After receiving sector fusion features from multiple spatial sectors, the global fusion network outputs an omnidirectional distance vector and direction confidence. One dimension of the omnidirectional distance vector corresponds to a direction index. The direction confidence is used to indicate whether the distance in that direction is estimated by the normal view or by the compensation logic. The system also outputs a spatial sector status table, which records the number of candidate views, the number of low-quality views, the number of single-view estimated directions, and the number of compensation directions for each sector.

[0066] In terms of sampling synchronization, the system divides multiple depth frames within the same sampling window into main frames and trailing frames. The main frame is generated by a sensor with relatively complete device attitude reference information, while the trailing frame is generated by other sensors. If the deviation between the sampling time of the trailing frame and the main frame is within the allowable range, the trailing frame enters the candidate view set. If the deviation exceeds the allowable range, the trailing frame only enters the historical record and does not directly participate in the current sector fusion.

[0067] When the device's attitude changes rapidly, the orientation index establishment module reads the attitude reference information given by the inertial measurement unit and retains the orientation index in the body coordinate system instead of directly converting it to the world coordinate system. As a result, the processing device can keep the omnidirectional distance vector consistent with the device's body direction when the device turns, rolls, or pitches, which makes it easier for downstream motion control to read the distance in front, behind, side, and above.

[0068] Example 2

[0069] See Figure 3 and Figure 4 In one implementation of a dynamic update of spatial sectors and candidate viewpoints, the sensor installation position and device shape are not required to be regular cubes. If the device is elongated, flat, or has an extended protective shell, the system reads the sensor installation file during the initialization phase. The installation file records the sensor installation surface, optical axis direction, horizontal field of view, vertical field of view, effective distance range, shell occlusion boundary, and availability status.

[0070] The spatial sector grouping module determines whether each directional index falls within the effective field of view of a certain sensor based on the installation file. When a directional index falls within the effective field of view of multiple sensors, the system includes the corresponding sensor in the candidate view set and determines the candidate order based on the angle between the directional index and the optical axis of each sensor, the effective distance range of the sensor, and the relationship with the historical effective field of view. The candidate order is used to determine the view that participates first when the view quality is similar.

[0071] The candidate viewpoint set is not fixed after one configuration. If the sensor status indicator shows that a certain depth frame has consecutive frame drops, overexposure, underexposure, or depth holes, the spatial sector grouping module temporarily downgrades the viewpoint from the candidate viewpoint set. If the sensor recovers stability in the subsequent sampling window, the system restores it to the normal candidate viewpoint. The restoration process adopts a step-by-step restoration of participation intensity.

[0072] The extraction of view-level depth features first involves cropping the effective depth range of the original depth frame, then locally filling the depth holes, and unifying the depth scales of different sensors to the same distance unit. The system does not require accurate extrinsic parameter registration at this stage, nor does it require the generation of a unified 3D point cloud. Candidate viewpoints only need to retain the local distance structure and quality state that can be learned by the supervised fusion model.

[0073] The view quality status includes at least the missing proportion, depth mutation proportion, edge region proportion, consistency with adjacent candidate viewpoints, and historical stability. The missing proportion is used to describe depth holes or frame drops, the depth mutation proportion is used to describe local spikes and flying points, the edge region proportion is used to describe whether the distance in that direction mainly comes from the edge of the sensor's field of view, and the consistency is used to determine whether candidate viewpoints support each other.

[0074] In an objective judgment method, the processing device classifies the missing proportion into three categories: normal, attention, and abnormal; the deep mutation proportion into two categories: stable and mutation; and the consistency with adjacent candidate viewpoints into three categories: consistent, pending review, and inconsistent. The above classifications are obtained from the number of effective pixels in the sampling window, the number of distance jumps, and the distance difference of overlapping areas, without the need for subjective human scoring.

[0075] After receiving the view quality status, the dynamic view gating module generates candidate view participation intensity according to the current spatial sector. The participation intensity is used to adjust the influence of the view depth feature entering the group encoder. If a view is at the edge of the field of view but is consistent with the adjacent view and historical distance sequence, the system retains the effective structural information of the view. If a view is located in the center of the image but has short-term dirt or a large number of holes, the system reduces the influence of the view.

[0076] The group encoder consists of a convolutional coding network, a lightweight visual transformation network, or a multilayer perceptual network. Each spatial sector corresponds to a group encoding process. Different sectors share some parameters and use different parameters for the front, side, top, and rear. The specific selection depends on the computing power of the device platform and the focus of the scene. The sector fusion feature output by the group encoder corresponds to the direction index within that sector.

[0077] After the sector fusion features are generated, the system retains the candidate view set and participation intensity record, which makes it easier to explain the source of distance in a certain direction in subsequent outputs. If the downstream control module finds that a certain direction has a persistently low confidence, engineers can check the candidate view status corresponding to that direction through the sector status table without having to re-parse the original depth map.

[0078] During the generation of view quality status, the processing device establishes a window record for each candidate view. The window record includes the effective pixel ratio of the current window, the number of consecutive holes, the edge region hit ratio, the number of times adjacent view is consistent, and the number of times historical low quality is recorded. If a candidate view only has holes in a single frame, the system can first reduce the participation intensity and wait for the next sampling window to be checked. If multiple consecutive windows are in a low quality state, they are marked as a continuous abnormal view.

[0079] When the candidate view set is updated, the processing device also retains the view recovery conditions. The view recovery conditions are used to determine whether the downgraded candidate view has re-entered the normal fusion link. For example, the frame loss state disappears within multiple consecutive sampling windows, the depth hole ratio decreases, and the consistency with adjacent candidate views is restored to a stable range. After these conditions are met, the dynamic view gating module gradually increases the participation intensity of the view.

[0080] To ensure that dynamic view gating corresponds to insufficient candidate views, insufficient candidate views are understood as the number of normal candidate views in a certain spatial sector being lower than the preset minimum number for that sector. It is also understood as the existence of candidate views but all being in a low-quality state. In this case, the system does not regard the output of that sector as a normal fusion result, but instead proceeds to a low-confidence review or missing direction compensation process.

[0081] The table below provides an example of a spatial sector record. This table illustrates the transitive relationship between spatial sectors, candidate view sets, view quality status, and output markers. Actual products will have their fields adjusted based on the number of sensors and orientation indexes.

[0082] To illustrate the correspondence between spatial sectors, candidate view sets, view quality states, and output markers, an example table 2 of spatial sector records is provided below. This table lists the corresponding candidate views, current quality states, gating methods, and output marker methods for different spatial sectors. It is used to explain that dynamic view gating is not a fixed average fusion, but rather adjusts the participation intensity based on the operating status of candidate views and the consistency of adjacent views.

[0083] Table 2: Example Table of Spatial Sector Records

[0084] spatial sector Example of candidate view set Quality status example Gating Processing Example Output tag example Front, top, right Forward, Right, Up Rightward depth saturation, forward and upward are consistent. Reduce rightward participation intensity, retain forward and upward participation. Normal estimate, right-hand low-quality record Back down left Backward, leftward, downward Short-term frame drop on the left, stable on the back Following the backward and downward directions, the review proceeds to the left. Single-view assistance, short-term loss Front, bottom, left Forward, Left, Down All candidate perspectives exist, but the proportion of peripheral regions is high. Reduce overall credibility and wait for the next window Low-confidence estimate, pending verification Back to top right Backward, Rightward, Upward Upward obscured by its own structure Eliminate self-occlusion perspective and use back and right views. Self-masking mark

[0085] Using the recording method shown in Table 2, the candidate view set is dynamically updated for the same spatial sector based on changes in sensor status. When a candidate view exhibits depth saturation, short-term frame drops, excessive edge region proportion, or self-structure occlusion, the system reduces the participation intensity of that candidate view or transfers the corresponding direction to a low-confidence verification and compensation process, thereby preventing abnormal views from entering the omnidirectional distance vector in the form of ordinary fusion results.

[0086] Example 3

[0087] See Figure 5 In one supervised training implementation, the training samples come from the simulation environment and also from teacher data obtained by high-precision reconstruction after offline collection of real scenes. The simulation environment imports device models, sensor models, obstacle models, ground models and lighting conditions. The sensor models retain field-of-view edge noise, random frame drops, depth holes, distance saturation and motion blur, so that the training input and the deployment input have similar data defects.

[0088] The teacher label is generated by the distance between the intersection point of the directional ray and the scene geometry. The system emits multiple directional rays outward from the origin of the body coordinate system. If the ray intersects with an obstacle or the boundary of the environment, the distance between the intersection point and the origin of the body coordinate system is recorded. If the ray does not hit an object within the effective range of the sensor, the maximum effective distance or the obstacle-free filling value is recorded. There is a one-to-one correspondence between the directional index and the teacher distance.

[0089] In addition to distance teacher labels, the training data also includes visibility labels, occlusion labels, and high confidence labels. Visibility labels indicate whether the direction is directly observed by the candidate viewpoint, occlusion labels indicate whether the direction is occluded, and high confidence labels indicate whether the direction is supported by multiple consistent viewpoints. These labels can be used to train the direction confidence output and anomaly direction marking.

[0090] During the training phase, occlusion samples, dirt samples, edge distortion samples, and sensor short-term failure samples are set. Occlusion samples are used to enable the model to learn how to utilize neighborhood sector information when a local viewpoint is unavailable. Dirt samples are used to simulate the effects of raindrops, dust, or mud on the lens. Short-term failure samples are used to train the system to output compensation markers when a certain depth frame is missing, instead of outputting the missing direction as the normal estimated direction.

[0091] The training objectives of the supervised fusion model include a distance error term, a direction confidence error term, a neighborhood consistency constraint term, and an anomaly labeling constraint term. The distance error term makes the predicted distance close to the teacher distance, the direction confidence error term makes the model distinguish between the normal estimated direction and the direction that needs to be compensated, the neighborhood consistency constraint term is used to reduce the jump of adjacent directions when there is no change in the real boundary, and the anomaly labeling constraint term is used to enhance the ability to identify missing viewpoints.

[0092] During training, the model updater updates the parameters of the group encoder, dynamic view gating module, and global fusion network based on the difference between the output of the supervised fusion model and the teacher labels. If the proportion of abnormal samples in a training batch is too high, the system reduces the update step size of that batch. If the normal and abnormal samples in a training batch are balanced, the update is performed according to the conventional training strategy.

[0093] After training, the deployment end no longer needs teacher labels and high-precision offline reconstruction results. The deployment end only receives real-time multi-channel raw depth frames and sensor status identifiers. After spatial sector grouping, dynamic view gating and supervised fusion model inference, the perception data packet is output. Therefore, the training process and the deployment process maintain the consistency between input fields and output fields.

[0094] When collecting data offline in real-world scenarios, multiple depth frames are collected in indoor calibration channels, warehouse rack channels, woodland sample plots, and low-altitude flight test sites. After collection, high-precision lidar or multi-frame offline reconstruction results are used to generate teacher distance. Teacher distance is only used during the training phase, and high-precision reconstruction equipment is not required during the deployment phase.

[0095] Training samples are balanced according to direction index. If the number of samples in the front direction is significantly more than that in the back or above direction in the training set, the model may be more biased towards the common direction. The processing device adds samples of objects approaching from behind, falling objects from above, obstacles extending from the side, and low obstacles to the training data pool to keep the distance estimation of the supervised fusion model stable for different directions.

[0096] After the model training is completed, a deployment check is performed. The deployment check includes reading the sensor installation file, reading the spatial sector grouping table, inputting fixed distance plate samples, checking the output direction confidence and abnormal direction markers. If certain directions continue to output low confidence, it will prompt to re-check the corresponding sensor installation angle or update the candidate view grouping table.

[0097] Example 4

[0098] See Figure 6 In a missing direction compensation and downstream control application, when there are insufficient candidate views for a certain direction index, or when the confidence of the direction output by the supervised fusion model does not meet the preset conditions, the system starts the missing direction compensation module and writes the direction into the abnormal direction list. The abnormal direction list is output together with the perception data packet.

[0099] If the direction reliability does not meet the preset conditions, it means that the direction is provided by a single low-quality view, or the distance difference between the candidate views corresponding to the direction exceeds the verification conditions, or the direction jumps within the continuous sampling window and the neighboring sectors cannot support the jump. The system sets stricter or more lenient verification conditions according to the platform security level.

[0100] The missing direction compensation module first determines the spatial sector to which the direction index to be compensated belongs, then reads the distance changes of adjacent spatial sectors, the historical distance sequence of the direction index to be compensated and the continuity of device movement. If the distance changes of adjacent spatial sectors are gradual and the continuity of device movement indicates that the device has not undergone a sharp turn, the system uses the historical distance trend to generate the compensation distance. If the distance changes of adjacent spatial sectors are large, the system reduces the reliability of the compensation distance.

[0101] The compensated distance is not mixed with the normal estimated distance in the output. The downstream obstacle avoidance control module selects a conservative control strategy based on the low confidence marker, such as reducing speed, increasing the safe distance, or requesting verification in the next frame.

[0102] In the low-altitude obstacle avoidance scenario of the drone, the training environment is set with tree branches, power lines, bird models, walls, ground slopes and falling objects from above. The omnidirectional distance vector of the deployment end can be read by the obstacle avoidance control module. If the distance in the front and lower direction is small and the confidence level is high, the control module reduces the forward speed and adjusts the climb angle. If the distance in the side and rear direction decreases rapidly, the control module restricts the lateral movement in advance.

[0103] In complex terrain traversal scenarios for quadruped robots, a two-dimensional half-height map is insufficient to represent suspended crossbars, side branches, and approaching obstacles from the rear. In this embodiment, the robot motion control strategy receives a low-dimensional omnidirectional distance vector and a confidence vector. When there is a low-distance and high-confidence obstacle in the upper direction, the control strategy restricts head-up or jumping actions. When an approaching object appears in the rear direction, the control strategy avoids rapid backward movement.

[0104] In the scenario of warehouse handling robots, the spatial sector status table is used to distinguish the changes in forward passage distance and lateral clearance. If a certain lateral sensor is blocked by goods and has low confidence, the dynamic view gating module reduces the participation intensity of that view and generates a compensation distance from adjacent sectors and historical distance sequences.

[0105] The perception data packet is sent to the downstream module through shared memory, robot operating system messages, middleware interface or vehicle communication interface. The data packet contains at least a direction index table, omnidirectional distance vector, direction confidence vector, spatial sector status table, compensation flag table and abnormal direction flag table. The downstream module only reads the distance vector, and also reads the confidence and sector status for safety policy purposes.

[0106] The perception data output module generates a window number for each sampling window. If the downstream control module adopts a conservative control strategy in a certain window, the system can check the source of the low confidence direction in that window to determine whether it is a short-term sensor loss, field-of-view edge distortion, lens damage, or dynamic obstacle occlusion.

[0107] If multiple consecutive sampling windows in the direction to be compensated are in a low confidence state, the system changes the state of the spatial sector corresponding to that direction from short-term abnormality to continuous abnormality. In the continuous abnormality state, the compensation distance is still output, but the downstream control module should increase the degree of conservatism, such as reducing the passing speed, pausing complex actions, or requesting the device body to change its attitude to reacquire the viewpoint in that direction.

[0108] When a device moves from a confined space into an open space, the historical distance sequence may still retain a relatively short distance. If multiple adjacent sectors show an increase in distance and the device's movement is continuous and stable, the compensation distance gradually changes to a longer distance and continues to carry the compensation marker until the candidate viewpoint is restored.

[0109] The table below provides an example of a deployment output review record. This table is used to illustrate the relationship between compensation distance, confidence level, and downstream control. The values ​​in the table are only for illustrating the field format and are not intended as data representing the effects that this application must achieve.

[0110] To illustrate the relationship between compensation distance, direction confidence, abnormal direction markers, and downstream control, a deployment output verification record table (Table 3) is presented below. This table records the sampling window and lists the direction index status, main triggering reasons, output processing, and downstream reading methods. It is used to explain the different data output methods of the system under normal estimation, low-confidence estimation, compensation output, and continuous abnormal states.

[0111] Table 3: Deployment Output Review Record Table

[0112] Window number Direction index state Main triggering reason Output processing Downstream reading method W1021 Normal estimate All three candidate perspectives are consistent Output distance and general reliability Used for obstacle avoidance and path planning W1022 Low-confidence estimation Single-view usable with high edge area ratio Output distance with low confidence marker Use conservatively and wait for the next window. W1023 Compensation output Insufficient candidate perspectives and continuous historical sequence Output compensation distance and compensation marker Reduce speed and increase safe distance W1024 Continuous abnormality Low quality of multiple window candidate views Output perception limited state Request resampling or change of pose

[0113] Using the deployment output verification record method shown in Table 3, the compensated distance and the normal estimated distance can be separately marked in the standardized sensing data packet.

[0114] Example 5

[0115] In one system implementation, the multi-view depth acquisition module consists of multiple depth vision sensors and acquisition drivers. The orientation index establishment module and the spatial sector grouping module run in the processor's initialization task. The view quality generation module and the dynamic view gating module run in the real-time inference task. The supervised fusion module calls the neural network acceleration unit to execute.

[0116] The system memory stores sensor installation files, orientation index tables, spatial sector grouping tables, model parameter files, view quality thresholds, compensation strategy tables, and output interface configurations. The sensor installation files record the sensor number, installation orientation, field of view range, effective distance range, and shell occlusion boundary. The spatial sector grouping table records the candidate viewpoints and adjacent sectors for each sector.

[0117] The multi-view depth acquisition module generates an original depth frame buffer in each sampling window. The view quality generation module reads the buffer and generates the view quality state. The dynamic view gating module converts the view quality state into the candidate view participation intensity. The group coding module generates sector fusion features based on the participation intensity. The supervised fusion module outputs distance and confidence. The missing direction compensation module further processes the low confidence direction.

[0118] When the system is deployed on a robot platform with low computing power, reduce the number of orientation indices and the number of spatial sectors, or use a lighter group encoder. When the system is deployed on a high-speed drone or a high-security mobile platform, increase the orientation index density in front, above, and side-rear, and retain a more complete orientation confidence output.

[0119] The number of depth vision sensors is not limited to 6. Depending on the device shape and the sensor field of view, it can be set to 4, 5, 8 or more. If the number of sensors is reduced, the system increases the number of compensation direction verifications. If the number of sensors is increased, the system increases the weight of view consistency judgment in overlapping field of view areas.

[0120] The supervised fusion model in this application is not limited to a fixed network structure. It can adopt convolutional networks, lightweight visual transformation networks, multilayer perceptual networks, or combinations thereof. The key is that the model receives depth features organized by spatial sectors and candidate viewpoints and outputs distance values ​​and confidence values ​​corresponding to orientation indices, rather than directly outputting local image results bound to a certain camera pixel position.

[0121] In the embedded implementation, the real-time tasks are executed sequentially as acquisition tasks, sector grouping tasks, view quality tasks, model inference tasks, and output tasks. The acquisition task is responsible for writing multiple depth frames into the buffer, the sector grouping task is responsible for generating a candidate view set, the view quality task is responsible for generating participation intensity, the model inference task is responsible for outputting distance and confidence, and the output task is responsible for compensation and data encapsulation.

[0122] Example 6

[0123] In one deployment and debugging implementation, after the device is installed for the first time, it enters the perception verification mode. The processing device sequentially reads the sensor installation file, the orientation index table and the spatial sector grouping table, and collects the original depth frames of several windows in the known distance plate, open area and occluded area.

[0124] In the perception verification mode, the processing device does not directly output control-available data to the motion control module. Instead, it writes the omnidirectional distance vector, direction confidence vector, and spatial sector status table into the debugging cache. The debugging cache displays the candidate view source, view participation intensity, compensation mark, and abnormal direction mark for each direction index, which facilitates manual confirmation of whether the sensor installation direction is reversed or deflected.

[0125] If a sector continues to display a low confidence state in a known unobstructed scene, first check the number of candidate views for that sector. If the number of candidate views is insufficient, update the spatial sector grouping table. If the number of candidate views is sufficient but the quality state is low, prompt to clean the lens.

[0126] During the robot's actual operation, the processing device retains low-confidence orientation records from the most recent sampling windows. The low-confidence orientation records do not need to save the complete original depth frame, but save the sampling window number, orientation index, sector number, low-quality viewpoint number, compensation method, and downstream control feedback. This can reduce storage usage and retain the link information required for subsequent maintenance.

[0127] The above deployment and debugging process does not change the main processing chain, but establishes a verifiable operation record between sensor status identification, candidate view set, dynamic view gating, compensation distance and perception data packet, so that this application can maintain feasibility across different robot platforms, number of sensors and scenario tasks.

[0128] In an alternative platform implementation, the device body is equipped with only 4 large field-of-view depth vision sensors, which face forward-upward, forward-downward, backward-upward, and backward-downward directions respectively. The spatial sector grouping module configures a set of candidate viewpoints for each spatial sector according to the field-of-view range of each sensor. If a sector has only 1 normal candidate viewpoint, the system increases the frequency of direction reliability verification for that sector and retains a single viewpoint source mark in the output data packet.

[0129] In another alternative platform implementation, the device body is equipped with eight depth vision sensors to improve the coverage density in the front and side-front directions when the mobile robot moves at high speed.

[0130] For water surface inspection equipment or low-speed indoor inspection equipment, the system reduces the number of top or bottom direction indices while retaining front, side and rear direction indices. The supervised training data is also configured according to the platform's task priorities, and scenarios such as water surface reflection, glass walls, shelf holes and narrow door frames are added to the training samples to make the model learning platform more responsive to common anomalies.

[0131] When a sensor is temporarily blocked, the system first determines whether the blockage is related to the device's own components, such as a robotic arm, protective cover, or load support entering a sensor's field of view. If the blockage comes from the device's own components, the system writes a self-blocking mark in the spatial sector status table. If the blockage comes from an external object, it writes an external blocking mark. The two types of marks are used by downstream control modules to adopt different strategies.

[0132] Regarding the data interface, the standardized sensing data packet adopts a fixed field structure, which includes a packet header, device number, sampling window number, number of direction indices, distance vector, confidence vector, sector status table, compensation flag table, abnormal direction flag table, and verification field. If the downstream control module does not need to read the complete status table, it only subscribes to the distance vector and confidence vector.

[0133] When there are key directions in the task area, the direction index building module performs local encryption on the key directions. For example, drones can encrypt the front and above, front and below, and side and rear. Quadruped robots can encrypt the low area in front, the horizontal bar area above, and the area close behind. The spatial sector status table synchronously records the relationship between the encrypted direction and the normal direction.

[0134] During the maintenance and debugging phase, the system saves the participation intensity of candidate viewpoints in the form of logs. The logs do not participate in real-time control, but they can indicate which viewpoints mainly contribute to the distance in a certain direction within a certain sampling window, and which viewpoints are gated to reduce their participation intensity. This helps engineers locate sensor installation deviations or the source of field of view occlusion.

[0135] The above-disclosed embodiments are used to illustrate the implementation path of this application. Actual products may adjust parameters based on device form, sensor field of view, computing resources, and task priorities. As long as multiple depth inputs are still organized using spatial sectors and candidate view sets, and omnidirectional distance data with confidence and anomaly markers is output through a supervised fusion model, they all fall under the technical approach disclosed in this application.

[0136] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to specific implementations. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A supervised learning-based omnidirectional deep visual data fusion method, characterized in that, The method, applied to an omnidirectional depth vision fusion device, includes: Acquire multiple raw depth frames collected by multiple depth vision sensors arranged in different spatial positions on the device body, and write them into the sensor status identifier; Establish a body coordinate system based on the device's sensing center, generate multiple directional indices, and divide the spatial sectors; Determine the candidate viewpoint set for each spatial sector based on the field of view coverage relationship; Extract the depth features and view quality state of each candidate view in the candidate view set, and dynamically gating the candidate view based on the view quality state to generate sector fusion features; The supervised fusion model outputs an omnidirectional distance vector and directional confidence based on sector fusion features. When there are insufficient candidate viewpoints or the orientation confidence does not meet the conditions, a compensated distance with compensation markers is generated, and a perception data packet containing omnidirectional distance vector, orientation confidence and abnormal orientation markers is output.

2. The omnidirectional deep visual data fusion method based on supervised learning according to claim 1, characterized in that, The determination of the candidate viewpoint set for each spatial sector based on the field of view coverage relationship includes: Obtain the installation orientation, field of view range, effective distance range, and historical effective field of view relationship of each depth vision sensor; Based on the correspondence between the multiple directional indices and the field of view range of each depth vision sensor, the candidate sensor number corresponding to each spatial sector is determined. Candidate sensors that are in a state of frame loss, abnormal exposure, or depth saturation are removed based on the sensor status identifier, and the depth frames corresponding to the retained candidate sensors are used as the candidate viewpoint set.

3. The omnidirectional depth visual data fusion method based on supervised learning according to claim 2, characterized in that, The step of determining the candidate sensor number corresponding to each spatial sector based on the correspondence between the multiple direction indices and the field of view range of each depth vision sensor includes: The multiple directional indices are divided into spatial sectors according to front-back, left-right, up-down or their combinations; For each spatial sector, statistics are compiled on depth vision sensors that can cover the orientation index within that spatial sector; When multiple depth vision sensors can cover the same spatial sector, the candidate order is determined based on the coverage angle, effective distance, and historical effective field of view relationship, and a candidate view grouping record is formed.

4. The omnidirectional deep visual data fusion method based on supervised learning according to claim 3, characterized in that, The extraction of depth features and view quality status for each candidate viewpoint in the candidate viewpoint set includes: The original depth frames corresponding to the candidate viewpoints are cropped, missing points are filled, and scale is unified to obtain viewpoint-level depth features. Based on the missing proportion, depth mutation proportion, edge region proportion, consistency with adjacent candidate viewpoints, and historical stability of the candidate viewpoints in the current sampling window, the corresponding viewpoint quality status is generated.

5. The omnidirectional depth visual data fusion method based on supervised learning according to claim 4, characterized in that, The dynamic gating of candidate views based on view quality status and the generation of sector fusion features include: The participation intensity of candidate viewpoints within the corresponding spatial sector is determined based on the viewpoint quality status. When a candidate perspective has an increased missing percentage, a concentrated deep mutation, or is inconsistent with adjacent candidate perspectives, the participation intensity of that candidate perspective should be reduced. When a candidate viewpoint aligns with feedback from neighboring sectors and historical distance sequences, increase the participation intensity of that candidate viewpoint. After completing the intensity adjustment, multiple candidate perspectives are input into the group encoder to obtain the sector fusion features of the corresponding spatial sectors.

6. The omnidirectional depth visual data fusion method based on supervised learning according to any one of claims 1 to 5, characterized in that, Before the supervised fusion model outputs the omnidirectional distance vector and direction confidence based on the sector fusion features, the method further includes: Construct training samples, which include multiple raw depth frames generated in the simulation environment, high-precision offline reconstruction results, or measured depth data in scenes with known distances; Distance teacher labels are generated based on the hit results between the directional ray emitted outward from the origin of the body coordinate system and the obstacle or environment boundary, and visibility labels, occlusion labels, and credibility labels are generated for the directional index.

7. The omnidirectional depth visual data fusion method based on supervised learning according to claim 6, characterized in that, After constructing the training samples, the method further includes adding occlusion samples, contamination samples, field-of-view edge distortion samples, and sensor short-term failure samples to the training samples. Based on the distance results, confidence results, and abnormal direction markers output by the supervised fusion model, the parameters of the supervised fusion model are updated with the differences between them and the corresponding teacher labels, so that the supervised fusion model learns the mapping relationship from sector fusion features to omnidirectional distance vectors and directional confidence.

8. The omnidirectional deep visual data fusion method based on supervised learning according to any one of claims 1 to 5, characterized in that, The step of generating a compensated distance with a compensation marker when candidate viewpoints are insufficient or direction confidence does not meet the conditions includes: Determine the spatial sector to which the direction index to be compensated belongs and its adjacent spatial sectors; Read the distance changes of the adjacent spatial sectors, the historical distance sequence of the direction index to be compensated, and the continuity of device movement; Compensated distances are generated based on the distance changes, historical distance sequences, and device motion continuity, and then output by binding the compensated distances with compensation flags and low confidence flags.

9. The omnidirectional deep visual data fusion method based on supervised learning according to claim 8, characterized in that, The output includes a perception data packet containing the omnidirectional distance vector, direction confidence, and abnormal direction markers. This includes: encapsulating the distance value, direction confidence, spatial sector status, compensation marker, and abnormal direction marker corresponding to each direction index into a standardized perception data packet; sending the standardized perception data packet to the obstacle avoidance control module, navigation planning module, or motion strategy network; and enabling downstream modules to determine the usage method of the corresponding direction distance based on the direction confidence and abnormal direction markers.

10. A supervised learning-based omnidirectional deep visual data fusion system, characterized in that, To implement the method of claim 1, the system includes a multi-view depth acquisition module, a direction index establishment module, a spatial sector grouping module, a view quality generation module, a dynamic view gating module, a group coding module, a supervised fusion module, a missing direction compensation module, and a perception data output module. The multi-view depth acquisition module acquires multiple raw depth frames. The direction index establishment module establishes a body coordinate system and generates a direction index. The spatial sector grouping module determines a candidate view set. The view quality generation module generates view quality states. The dynamic view gating module and the group coding module generate sector fusion features. The supervised fusion module outputs an omnidirectional distance vector and direction confidence. The missing direction compensation module generates a labeled compensation distance. The perception data output module outputs standardized perception data packets.

Images