30results about How to "Solve the fusion problem" patented technology

The invention discloses a stereo image quality evaluation method of a multi-task coder-decoder based on Transform, and belongs to the technical field of image quality evaluation. The invention provides a multi-task coding-decoding device network architecture based on Transform. The multi-task coding-decoding device network architecture takes image depth estimation as an auxiliary task; a feedback mechanism of a human visual system can be better simulated by adopting a coder-decoder network, the problems existing in left and right view feature extraction are solved, and an information processing process from a global state to a local state is realized; in addition, the global interaction capability of Transform and the local attention capability of CNN are combined to enhance image features, depth estimation is introduced as an auxiliary task, depth prior information with higher quality can be provided, cross-view parallax fusion based on Transform is guided, the problem existing in left and right view feature fusion is solved, and more accurate binocular features are obtained. And feedback compensation is carried out on multi-level binocular feature fusion by using an attention mechanism to obtain a final pixel-level quality map, so that the image quality is accurately evaluated, a research scheme is provided for a deep learning method for non-reference stereo image quality evaluation, and the development of a visual task is effectively promoted.

The invention discloses a single-pixel fluorescent Raman bimodal imaging system and method, and belongs to the technical field of optical imaging. Comprising a light source module, a coding structured light module, a microscopic imaging module, a sample control module and a signal collection module which are sequentially arranged along a light path, the light source module is used for generating laser with required wavelength; the encodable structured light module converts the received laser into structured light; the microscopic imaging module amplifies the structured light, projects the amplified structured light to a sample plane, and collects fluorescence, elastic scattering and inelastic scattering generated after the sample is excited; the signal collection module amplifies, separates and collects the received fluorescence signals and scattering signals, converts the light signals into electric signals, and transmits the electric signals to the control and calculation module for storage and subsequent processing. According to the invention, the problem that the existing imaging system cannot obtain time-space consistent multi-modal images under the same excitation source and cannot realize image fusion is solved.

The invention relates to the technical field of heterogeneous cross-modal fusion, and discloses a pulmonary embolism auxiliary diagnosis method and system based on heterogeneous cross-modal fusion, and the system comprises the following modules: an image module which combines Swin Transform and an improved 3D CNN model to extract features; the polarization self-attention module is used for efficiently processing high-resolution 3D input features and reducing calculation overhead; the multi-scale feature fusion module integrates SE module channel feature extraction, ASPP module multi-scale feature extraction and 3D convolution space feature extraction; the EHR module is used for extracting features from the electronic health records; and the fusion module is composed of a multi-modal fusion module MAFM and is responsible for extracting key features from the image and the EHR module and fusing the key features so as to improve the diagnosis effect. Through a parallel network and a multi-modal fusion mechanism, the diagnosis precision and robustness of pulmonary embolism are remarkably improved.

The invention provides a multi-modal data fusion method and system, a readable storage medium and computer equipment, and the method comprises the steps: obtaining modal data corresponding to each equipment node, the modal data comprising text data, image data, video data and audio data, and each equipment node corresponding to one modal data; performing gradient learning on the initial local model on each equipment node according to the modal data to obtain a final local model on each equipment node; and fusing the final local models on the equipment nodes by adopting a weighted average algorithm to obtain an initial fusion model, and carrying out iterative convergence on the initial fusion model to obtain a final fusion model. According to the method, multi-modal and cross-modal federated learning can be realized, so that correlation information among different modals can be better captured, the performance and generalization ability of the model are improved, and more comprehensive data understanding and application are realized.

The invention discloses a rainfall inversion method based on satellite data multi-branch fusion and geographic information correction, and the method comprises the following steps: S1, obtaining observation data and preprocessing the observation data to obtain multi-modal rainfall inversion data with unified temporal-spatial resolution, the observation data comprising six kinds of image data of infrared, passive microwave, radar reflectivity factor, elevation, longitude and latitude; s2, a double-link precipitation inversion model is established and trained, the double-link precipitation inversion model comprises a regression link and a classification link, and each of the regression link and the classification link comprises a three-branch fusion module and a geographic information correction module; and S3, inputting the multi-modal rainfall inversion data into the trained double-link rainfall inversion model for rainfall inversion, and obtaining a regression result and a classification result as rainfall inversion results. Through multi-branch fusion and geographic information correction, the problems of insufficient satellite data fusion and interaction and lack of consideration of geographic factors in the prior art are solved, and the accuracy of a rainfall inversion result is improved.

The invention provides a robot environment sensing data processing method, system and device and a medium. According to the method, environment sensing data are obtained through all sensors in an environment sensing sensor cluster, then the collected environment sensing data are integrated into an environment sensing data set and sent to a processor, and the processor generates a robot operation data matrix according to the received environment sensing data set; according to the robot operation data matrix, the processor determines an environment sensing result according to the robot operation data matrix, multi-sensor data collection and spatial position correction are achieved, the problem of multi-source heterogeneous data fusion is solved, and after the robot operation data matrix is generated, the environment sensing result is obtained. And environment features can be efficiently extracted, an environment sensing result can be determined, the sensing precision and real-time performance of the robot on a complex environment are remarkably improved, and a reliable data basis is provided for subsequent navigation, obstacle avoidance and other tasks.

The invention discloses a multi-task hybrid expert model fine tuning method based on general LoRA and domain-specific LoRA, which is characterized in that general experts and domain-specific experts are integrated, and residual connection is introduced, so that the processing capability of the model on general tasks and domain-specific tasks is balanced, and the performance and the stability are improved. According to the method, efficient multi-task learning and new field adaptation of parameters are realized.

The invention relates to a controllable data generation method and system based on a potential diffusion model, and a medium, and the method comprises the steps: U1, carrying out the image coding of a noise picture and a target ROI picture through an image encoder of the potential diffusion model, obtaining the embedded data information of the noise picture and the embedded data information of the ROI, inputting the target box information into an MLP model, and carrying out the recognition, target box embedded data information is obtained, the scene cue words are encoded through a natural language editor of the potential diffusion model, and prompt embedded data information is obtained; and U2, on the basis of the prompt embedded data information, the target box embedded data information and the ROI embedded data information, splicing and embedding the data by adopting a cross attention splicing algorithm based on adaptive learning. According to the method, the problem that extreme samples are difficult to collect in a target detection part scene is solved, and the problem that the controllability of diffusion model generation content is not accurate enough is solved by introducing target ROI information and box information.

The present application relates to the field of data processing technology, and discloses a method, system and storage medium for controlling a robotic arm based on multimodal drive. The method comprises: collecting data through a sensor module for preprocessing to obtain a multimodal data set; extracting features from the multimodal data set and performing feature fusion to obtain a fused feature vector; estimating the state data of the fused feature vector through an unscented Kalman filter, and performing state prediction through a recursive neural network to obtain the current state of the robotic arm and the predicted state of the robotic arm; performing robotic arm operation trajectory planning and analysis on the current state of the robotic arm and the predicted state of the robotic arm to obtain the target execution trajectory; generating a control strategy for the target execution trajectory through an event-triggered distributed coordinated control algorithm to obtain a real-time joint control instruction sequence, and controlling the robotic arm to perform operations through the real-time joint control instruction sequence. The present application improves the efficiency and accuracy of robotic arm control based on multimodal drive.

The invention belongs to the technical field of ecological monitoring, and discloses a forest ULS-TLS point cloud automatic registration method based on ground context description, and the method employs a ground point as a registration element, constructs a ground context, can directly achieve the registration of TLS single scanning data to a ULS point cloud, and does not need the pre-registration of different scanning point clouds of a TLS. The extraction process of the ground points is relatively standardized, stable and reliable results can be obtained generally, and the method is not affected by distribution, density and shape complexity of trees in the forest. Compared with extraction of tree attributes, the ground points are used as registration primitives, so that the calculation cost is reduced, and high calculation overhead caused by extraction of the tree attributes is avoided. The invention aims to provide a forest ULS-TLS point cloud automatic registration method based on ground context description so as to quickly, accurately and automatically realize registration of forest ULS and TLS point clouds, comprehensively obtain forest structure information and promote application of a laser radar technology in forest investigation.

The invention belongs to the technical field of intelligent navigation, provides a monocular polarization vision depth estimation method, a monocular polarization vision depth estimation device, monocular polarization vision depth estimation equipment and a medium, and relates to a representation method for obtaining polarization information and direction information fusion of a pixel position by preprocessing a polarization image. And constructing an illegal line guided depth estimation network for outputting the illegal line guided depth of the polarization image. A depth estimation network guided by a scene-level normal is constructed to generate a scene-level normal of a polarization image, a loss function is constructed by using an image segmentation network and an object-level polarization normal estimation physical model to accelerate network convergence, and finally a pixel-by-pixel calculation relation model of a surface normal and depth is established by introducing a normal distance. And converting the generated scene-level normal into the depth guided by the scene-level normal. And performing fusion optimization on the depth guided by the scene-level normal and the depth guided by the illegal line to obtain final depth information of the polarization image. According to the invention, the robustness of monocular depth estimation can be effectively improved.

The invention discloses a multi-unmanned-vehicle cooperative mapping method based on an adaptive particle swarm algorithm, and aims to solve the problem that laser mapping fusion precision is reduced due to existence of dynamic obstacles during cooperative mapping of multiple unmanned vehicles in an indoor environment. The method comprises the following steps: firstly, carrying out feature matching through a self-adaptive local feature point matching region by adopting a joint compatible branch and bound algorithm, and calculating a joint mahalanobis distance optimization matching process of a feature matching hypothesis; secondly, by judging the change between the two frames, removing dynamic obstacles and updating a local map; secondly, optimizing a map fusion problem by adopting a particle swarm optimization algorithm, and searching a conversion matrix with the highest overlapping degree between maps; and finally, in order to prevent the particle swarm optimization from falling into local optimum, map fusion is realized in combination with a simulated annealing algorithm, and the map fusion precision is improved. According to the method, the map fusion problem is converted into the problem of solving the optimal solution by using the adaptive particle swarm algorithm fused with the simulated annealing algorithm, so that the map fusion precision is improved.

The present invention discloses a multi-agent vehicle-road-cloud integrated collaborative decision-making and control architecture system and method based on federated reinforcement learning. A multi-agent federated reinforcement learning decision-making and control framework with embedded vehicle dynamics characteristics is adopted to solve the problem of deep integration of intelligent transportation systems and intelligent vehicles, and realize autonomous driving with deep decision-making and control coordination between vehicles and traffic; a semantic matrix is ​​generated on the road side as the input of vehicle-side reinforcement learning to construct global and local trajectory planning for the vehicle side guided by the road side; a fusion reward function of vehicle-side reinforcement learning is designed based on the driving safety field constructed on the road side to realize comprehensive consideration of vehicle-side safety and comfort; based on road-side federated learning, the vehicle-side neural network parameters are uploaded through V2I communication to solve the problem of vehicle-road information asymmetry caused by privacy awareness; for the distribution of samples in different environments, the local optimal strategy for the current environment is selected through neural network screening, and a shared model that benefits from different environments is synthesized to achieve a balance between sample efficiency and model robustness.

The invention provides a multi-sensor information fusion method based on trust index divergence, which considers the conflict degree between evidences and the uncertainty of the evidences while determining the weight of the evidences, overcomes the problem that the evidence weight determination is mostly based on the conflict degree between the evidences and the characteristics of the evidences are rarely considered in the prior art, and improves the reliability of the evidence weight determination. According to the method, the conflict degree can be reasonably measured and corrected by fully utilizing the existing evidence information, the influence of the conflict evidence can be reduced, and unreasonable results are avoided. Information fusion is carried out on data of a multi-sensor decision-making layer, so that the defects of large calculation amount and subjective factor influence of artificially given weight coefficients in a conventional method can be overcome, the influence of conflict evidences is reduced, and the speed and precision of multi-sensor information fusion are accelerated. The multi-sensor information fusion method provided by the invention is suitable for a target recognition scene of multi-sensor high-conflict information in engineering application, and can effectively solve the fusion problem of the multi-sensor high-conflict information.

The present invention discloses a visible light, infrared and radar fusion target detection method based on deep learning, comprising: automatically analyzing and processing data collected by a visible light camera, a thermal infrared camera and a laser radar, firstly fusing the pre-processing stage of visible light and infrared to realize the conversion of the laser radar point cloud to a bird's-eye view, and then automatically extracting and bifurcating pyramid-type cross-stage features of the fusion image and the bird's-eye view, and then performing bifurcated target detection, and then performing "graph" matching, and finally realizing target detection by fusing the three modes of visible light, infrared and radar, thereby improving the detection accuracy, having good robustness, and being able to adapt to the scene changes and dynamic changes of the load. With the increase of the types of targets in the training set and the increase of the training set, the present invention has a broader detection performance.

The present invention discloses a dual-network playback device and a dual-network integration method, wherein the dual-network playback device includes a master device for accessing a public network, and at least one slave device for accessing a private network connected to the master device via a first interface, wherein the slave device and the master device implement access to the private network, authentication authorization, billing, and data encryption and decryption processing according to data interaction rules of at least one operator. The present invention provides a low-cost, simple-structured, and efficient solution for private network access while being able to provide public network access.

A streaming media distribution method includes: a SIP signaling server controls a camera device to initiate an active connection to a Darwin streaming media server, and sends a video stream to the Darwin streaming media server; the Darwin streaming media server receives the SDP information sent by the SIP signaling server, uses the key information y value and the device ID in the SDP information to establish an association relationship with the SSRC value in the video stream, and uniquely identifies a video stream through the device ID; the Darwin streaming media server adds a QTSSGB28181Module forwarding module to receive the RTSP request of the client and realize the forwarding of the video stream. The present invention adds support for the GB28181 protocol on the basis of the Darwin streaming media server, so that the streaming media server can be better applied to the video monitoring subsystem of the visualization of the power transmission channel.

The invention discloses a laser point cloud and multispectral image fusion method combining Hough transformation and semantic feature points, and the method comprises the steps: inputting a laser point cloud and a multispectral image into a Line-CNN deep learning network, and carrying out the corresponding line feature extraction, and obtaining a point cloud feature straight line graph and a multispectral feature straight line graph; mapping the feature straight-line graph to a Hough space, and obtaining rectangular gravity center and attribute information of point cloud and rectangular gravity center and attribute information of multiple spectrums through peak point detection as semantic feature points; and performing rectangular matching on the semantic feature points, calculating a projection matrix according to a rectangular matching result, assigning the multispectral information to the laser point cloud according to the projection matrix, finally obtaining the laser point cloud with the multispectral information, and realizing accurate fusion of the laser point cloud and the multispectral image. According to the method, the robustness and robustness of matching and fusion of the two-modal data are remarkably improved, and the problems of cross-modal data matching and fusion in a complex ground feature scene are effectively solved.

This invention provides a multi-sensor information fusion method based on trust index divergence. When determining evidence weights, it considers both the degree of conflict between pieces of evidence and the inherent uncertainty of the evidence itself. This overcomes the problem that previous studies mostly relied on the degree of conflict between pieces of evidence when determining evidence weights, rarely considering the characteristics of the evidence itself. It can fully utilize existing evidence information to reasonably measure and correct the degree of conflict, reducing the impact of conflicting evidence and avoiding unreasonable results. Information fusion of multi-sensor decision-making layer data overcomes the shortcomings of previous methods, such as high computational cost and the influence of subjective factors in manually assigned weight coefficients. It reduces the impact of conflicting evidence and accelerates the speed and accuracy of multi-sensor information fusion. The multi-sensor information fusion method proposed in this invention is suitable for target recognition scenarios with high conflict information from multiple sensors in engineering applications, and can effectively solve the problem of fusing high-conflict information from multiple sensors.

The present application relates to a kind of multi-modal visual large model under the dynamic target identification system of heterogeneous sensor anti-interference.The system includes: anti-background interference module, by modality convolution to multi-modal data of heterogeneous sensor, LSTM time series modeling and weighted fusion generation space-time feature, combine time / space attention mechanism and Transformer reinforced feature, use Gaussian probability density model to filter foreground pixel and extract foreground feature;Multi-modal processing module, separate each modality foreground feature and splice after refining, by Transformer extract consistent feature and complementary feature, after fusion and pooling enhancement generation reinforced fusion feature;Target identification module, fusion foreground feature and reinforced fusion feature, after convolution compression and Timesformer long time series modeling, input target detection head, output target classification and boundary box.The present application solves the problem of heterogeneous sensor data interference suppression and dynamic target accurate identification, improves robustness in complex scene.

The present invention discloses a method for fusing laser point cloud and multispectral image by combining Hough transform and semantic feature points, including: inputting laser point cloud and multispectral image into Line‑CNN deep learning network to extract corresponding line features, obtaining point cloud feature straight line graph and multispectral feature straight line graph; mapping feature straight line graph into Hough space, obtaining rectangular centroid and attribute information of point cloud and rectangular centroid and attribute information of multispectral through peak point detection, as semantic feature points; performing rectangular matching on semantic feature points, calculating projection matrix according to rectangular matching result, assigning multispectral information to laser point cloud according to projection matrix, finally obtaining laser point cloud with multispectral information, and realizing accurate fusion of laser point cloud and multispectral image. The present invention significantly improves the robustness and robustness of bimodal data matching and fusion, and effectively solves the problem of cross-modal data matching and fusion in complex ground scene.

The invention discloses a precipitation inversion method based on multi-branch fusion of satellite data and geographic information correction, comprising the following steps: S1, obtaining observation data and preprocessing to obtain multi-modal precipitation inversion data with unified temporal and spatial resolution, wherein the observation data includes six types of image data, namely infrared, passive microwave, radar reflectivity factor, elevation, longitude and latitude; S2, establishing and training a dual-link precipitation inversion model, wherein the dual-link precipitation inversion model includes a regression link and a classification link, wherein both the regression link and the classification link include a three-branch fusion module and a geographic information correction module; S3, inputting the multi-modal precipitation inversion data into the trained dual-link precipitation inversion model for precipitation inversion, and obtaining regression results and classification results as precipitation inversion results. The multi-branch fusion and geographic information correction solve the problems of insufficient satellite data fusion and interaction in the prior art and the problem of lack of consideration of geographic factors, thereby improving the accuracy of precipitation inversion results.

The invention relates to the technical field of abandoned mine recovery evaluation, in particular to an abandoned mine natural recovery evaluation method based on ecological element collaborative characteristics, and the method comprises the steps: obtaining data; performing primary judgment; performing secondary judgment; and finally carrying out collaborative judgment. And adjusting a threshold value and generating an evaluation report. According to the method, the vegetation coverage is rapidly and preliminarily identified; secondly, evaluating the internal recovery potential of the starting area from the dimensions of provenance, microorganism, hydrological connectivity and the like; and finally, for the potential-sufficient region, accurately judging a system which is still fragile in function and a health system which has formed stable self-maintenance capability by quantifying water and soil vegetation process coupling, a biological interaction network and anti-interference capability, and dynamically adjusting an initial threshold value based on the time sequence stability of a classification result of each region. The whole evaluation system has a data-driven adaptive capability, and the problems of low evaluation result accuracy and response speed lag caused by single evaluation dimension are effectively solved.

The invention discloses a system and a method for heterogeneous sensing information fusion and point cloud registration in a man-machine co-fusion environment, and aims to solve the problems that point cloud densities between a depth camera and a laser radar are different, range scales are obviously different, point cloud arrangement rules are completely different, and sensing information fusion faces challenges. Through sliding ball point cloud coarse positioning, coarse registration based on an improved FPFH algorithm of a coincidence ratio, and fine registration based on an improved ICP algorithm generated based on corresponding points, the problems of point cloud range scale difference and density difference are solved, sensing information accurate fusion and point cloud registration between a depth camera and a laser radar in a man-machine co-fusion environment are realized, and the accuracy of point cloud registration is improved. Meanwhile, the method can cope with the indoor environment, especially the situation of non-planar features of the robot, and meets the requirements of man-machine co-fusion safety detection for multi-source sensing information fusion.

The present invention relates to a ship damage assessment method based on state space geometric analysis. The method is based on the principle of system engineering and on the functional relationship components of the ship. The deductive method of the damage state tree is used to analyze the functional association relationship between the ship components. The hesitant fuzzy set theory and Choquet integral are used as a powerful aggregation function for fuzzy elements to analyze the components and the functional relationship, thereby solving the information fusion problem of the component damage state, achieving the development of an information fusion algorithm that is closer to the actual damage state transfer process, and can be used for quantitative analysis and assessment of the damage state of the ship, thereby providing auxiliary reference for decision makers of combat command. The present invention provides a method for evaluating the damage state of the ship for ship designers and combat command decision makers, and guides the damage assessment of the ship.

The invention relates to a moving target identification method, which comprises the following steps of: respectively carrying out feature extraction on acquired multi-modal data to obtain a first modal feature, a second modal feature and a third modal feature; fusing the second modal feature and the third modal feature with frequency characteristics into a fused modal feature through low-rank multi-modal fusion; performing time extraction and position embedding on the first modal feature and the fusion modal feature in sequence to obtain a first processing feature and a second processing feature; cross-modal interaction of the first processing feature and the second processing feature is established by using a cross-modal learning model, a third processing feature and a fourth processing feature are output, and the cross-modal learning model is constructed based on an improved multi-modal Transform model; and performing prediction according to the third processing feature and the fourth processing feature to obtain a target recognition result. According to the invention, the accuracy of moving target identification can be effectively improved.

The present invention discloses a generative adversarial network collaborative gesture synthesis method and device based on dual-path Transformer, and relates to the field of human-computer interaction technology. It includes: obtaining text data, audio data and speaker identity data, inputting them into a constructed Transformer-based cross-modal fusion network model, and obtaining gesture synthesis results; wherein, the Transformer-based cross-modal fusion network model includes: a front-end single-modal encoder module, a cross-modal fusion module and a gesture translation module. The present invention introduces an innovative multimodal fusion mechanism to achieve end-to-end collaborative voice gesture generation. The fusion mechanism is carried out through the synergy of parallel directional cross-modal Transformers and interactive cascaded two-dimensional attention modules, which fills the shortcomings of a single modality and helps the model focus on the context and voice information related to gestures. It realizes the generation of three-dimensional gestures without pre-definition, rather than simply predicting gesture categories, and achieves the best effect in implementation.

The invention provides a VHF and IP voice converged communication method and device for a special ship. The method comprises the following steps: monitoring environment temperature and electromagnetic interference intensity in real time; a pre-stored antenna parameter compensation library is matched based on the environment temperature, and radio frequency front end transmission parameters are dynamically optimized; and performing multi-modal noise suppression on the received VHF audio signal, wherein the multi-modal noise suppression comprises the following steps of: generating a time-frequency mask based on electromagnetic interference intensity by adopting a complex convolution attention network to suppress electromagnetic interference; and converting the processed signal into an IP voice data packet, and reversely converting the IP voice data packet into a VHF signal through a protocol conversion engine to realize a bidirectional communication link. The voice communication quality and reliability in the polar region environment are improved, and the navigation safety is guaranteed.

The invention discloses a 3D micro-emulsion droplet preparation device and method.The 3D micro-emulsion droplet preparation device comprises a preparation cavity formed in a preparation body, transparent windows are arranged on the front side and the rear side of the preparation cavity, and a top exhaust pipeline, a feeding side conveying pipeline, a collecting side conveying pipeline and a generating side conveying pipeline are arranged on the preparation body; the four pipelines communicate with the preparation cavity, and the ends, away from the preparation cavity, of the four pipelines are detachably connected with the plugs correspondingly. During use, the preparation body is vertically placed, plugs above and below the preparation cavity and on the collection side are removed, a continuous phase is introduced into the preparation cavity through the feeding side conveying pipeline, the exhaust pipeline at the top is blocked after the preparation cavity is filled with the continuous phase, the plug installed on the generation side conveying pipeline is removed, and a dispersed phase is introduced through the generation side conveying pipeline. The micro-emulsion droplets with different sizes and frequencies can be prepared by adjusting the flow rates of the dispersed phase and the continuous phase, the whole structure is simple and clear, observation is convenient, operation and use are simple, the preparation cost can be reduced, and the preparation process is simplified.

The application discloses a kind of based on bounded integer sequence and data stream fusion encoding decoding method and device, encoding part includes: the object to be compressed is attempted to multiple compression methods, and the optimal compression scheme is selected according to error size;In optimal compression scheme, the bounded integer sequence in the encoding content is encoded in bounded integer sequence;In optimal compression scheme, the compression method selection mark in the encoding content is encoded;Compression method selection and bounded integer sequence are data stream fusion;Output actual compression data stream;Decoding part includes: compression data stream in identification compression method mark;According to the compression method identified, the remaining part in fixed length compression data is decoded in bounded integer sequence.