Image acquisition and fusion method and device

Abstract

The present application proposes a method and device for image acquisition and fusion, which belongs to the field of robot vision and image data processing technology. The method includes: acquiring video image sequences from multiple fixed cameras; calculating the alignment weight of each fixed camera; using the corresponding fixed camera with the highest alignment weight as the reference camera, and aligning the video image sequences of other fixed cameras with the video image sequence of the reference camera according to the time standard of the reference camera; calculating the fusion weight of each fixed camera based on the loss function of the action in imitation learning; splicing the aligned video image sequences of the multiple fixed cameras based on the fusion weight of each fixed camera to obtain a fused image sequence. The method of the present application reduces the complexity of subsequent image processing and improves the quality of policy input data, while enhancing the robustness and generalization ability of the policy model to diverse inputs.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot vision and image data processing, and in particular relates to a method and device for image acquisition and fusion. Background Art

[0002] In recent years, imitation learning (IL) has been widely used in robotics. The rise of large models, in particular, has significantly advanced IL in robotic control. Imitation learning systems typically record relevant operational data through a visual perception module and convert it into actionable robot actions through policy learning. However, existing visual acquisition methods often suffer from the following issues:

[0003] 1. Limited viewing angle: Traditional cameras are fixed in the experimental environment or on the robot body. They are only effective for a single task and cannot adapt to generalized data collection.

[0004] 2. Missing depth information: Most cameras have certain requirements for data collection. When the camera is too close to the mission target, parts such as the depth camera will fail.

[0005] 3. Difficulty in data labeling: Due to the unreasonable choice of viewing angle, the collected data contains a large amount of invalid information, which increases the complexity of post-processing and increases the cost of data labeling.

[0006] 4. Poor environmental adaptability: Existing solutions lack flexible camera placement and are difficult to adapt to different mission scenarios.

[0007] If multiple camera perspectives are used, the need for image data fusion is inevitable in data usage. At the same time, different task scenarios have different requirements for image information. Maintaining the same fixed camera perspective and image data fusion solution will not be applicable to all task scenarios. Summary of the Invention

[0008] In response to the shortcomings of the existing technology, this application proposes a method and device for image acquisition and fusion, which adopts multiple camera perspectives, can adjust the camera placement at any time, automatically adjust the image fusion scheme according to different tasks, improve the quality of data acquisition and fusion, and thus enhance the effect of imitation learning.

[0009] In a first aspect, the present application proposes a method for image acquisition and fusion, comprising:

[0010] In imitation learning, multiple video image sequences from fixed cameras are acquired;

[0011] Calculate the alignment weight of each fixed camera based on the video image sequence of each fixed camera;

[0012] The fixed camera with the highest alignment weight is used as the reference camera. The video image sequences of other fixed cameras are aligned with the video image sequence of the reference camera according to the time standard of the reference camera.

[0013] Calculate the fusion weight of each fixed camera according to the loss function of the action in imitation learning;

[0014] According to the fusion weight of each fixed camera, the aligned video image sequences of multiple fixed cameras are spliced ​​to obtain a fused image sequence.

[0015] The calculation of alignment weights for each fixed camera based on the video image sequence of each fixed camera includes:

[0016] Calculate the clarity and noise level of the image sequence based on the video image sequence of each fixed camera;

[0017] Calculate a comprehensive score for each fixed camera based on the clarity and noise level of the image sequence;

[0018] The alignment weight of each fixed camera is calculated according to the comprehensive score of each fixed camera.

[0019] The comprehensive score of each fixed camera is calculated as follows: ;

[0020] in, is the comprehensive score of the i-th fixed camera, is the noise level of the image sequence of the i-th fixed camera, is the clarity of the image sequence of the i-th fixed camera, is the minimum value.

[0021] According to the comprehensive score of each fixed camera, the alignment weight of each fixed camera is calculated as follows: ;

[0022] in, is the alignment weight of the i-th fixed camera, exp is the power operation of the natural logarithm e, To control the sensitivity coefficient of weight distribution, is the comprehensive score of the i-th fixed camera, is the comprehensive score of the jth fixed camera, is the bias of the i-th fixed camera, is the bias of the jth fixed camera, and M is the number of fixed cameras.

[0023] The fusion weight of each fixed camera is calculated based on the loss function of the action in imitation learning. The calculation formula is as follows: ; ;

[0024] in, is the loss function for the action of the i-th fixed camera in imitation learning, is the predicted action of the jth imitation learning for the i-th fixed camera, is the action demonstrated by the expert in the jth imitation learning of the i-th fixed camera, N is the batch size in the imitation learning, is the fusion weight of the i-th fixed camera in the t+1 round of imitation learning training, exp is the power operation of the natural logarithm e, To control the sensitivity coefficient of weight distribution, M is the number of fixed cameras, is the loss function of the action of the i-th fixed camera in the t-th round of imitation learning training, is the loss function of the action of the j-th fixed camera in the t-th round of imitation learning training.

[0025] The method of stitching the aligned video image sequences of the multiple fixed cameras according to the fusion weight of each fixed camera to obtain a fused image sequence includes:

[0026] Stitching the aligned video image sequences of the multiple fixed cameras to obtain a stitched image sequence;

[0027] In the stitched image sequence, the aligned fixed camera video image sequence with the smallest fusion weight is smeared, and the stitched image sequence after smearing is used as the fused image sequence.

[0028] In a second aspect, the present application proposes an image acquisition and fusion device, comprising:

[0029] An image acquisition module is used to acquire video image sequences from multiple fixed cameras in imitation learning;

[0030] An alignment weight calculation module is used to calculate the alignment weight of each fixed camera based on the video image sequence of each fixed camera;

[0031] An image alignment module is configured to align the video image sequences of other fixed cameras with the video image sequence of the reference camera according to the time standard of the reference camera, using the corresponding fixed camera with the highest alignment weight as the reference camera;

[0032] The fusion weight calculation module is used to calculate the fusion weight of each fixed camera according to the loss function of the action in imitation learning;

[0033] The image fusion module is used to stitch the video image sequences of multiple aligned fixed cameras according to the fusion weight of each fixed camera to form a fused image sequence.

[0034] The image acquisition module includes multiple fixed brackets, fixed cameras are installed on the fixed brackets, and the height and angle of the fixed cameras are adjusted by the fixed brackets. The multiple fixed cameras are used to obtain video image sequences from the main view, top view, side view and oblique view respectively.

[0035] In a third aspect, the present application proposes an electronic device comprising: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the method of image acquisition and fusion.

[0036] In a fourth aspect, the present application proposes a computer-readable storage medium storing executable instructions, which, when executed, enable a processor to perform the image acquisition and fusion method.

[0037] Beneficial effects:

[0038] This application proposes a method and apparatus for image acquisition and fusion. Through an adjustable fixed camera position design, this approach provides a stable, referenceable temporal alignment benchmark while maintaining image quality. This reduces the complexity of subsequent image processing and improves the quality of policy input data. Furthermore, by incorporating an image smearing enhancement mechanism using random camera perspectives, the policy model's robustness and generalization capabilities to diverse inputs are further enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of a method for image acquisition and fusion according to an embodiment of the present application;

[0040] Figure 2 This is a block diagram of the principle of an image acquisition and fusion device according to an embodiment of the present application;

[0041] Figure 3 This is a schematic diagram of an implementation of an image acquisition module according to an embodiment of the present application;

[0042] Among them, 1-fixed bracket, 2-fixed camera, 3-mobile robot platform. DETAILED DESCRIPTION

[0043] The specific implementation of the present application is further described in detail below with reference to the accompanying drawings and examples.

[0044] Example 1:

[0045] This embodiment proposes a method for image acquisition and fusion, such as Figure 1Shown, including:

[0046] Step S1: In imitation learning, obtain video image sequences of multiple fixed cameras;

[0047] In this embodiment, in order to optimize the camera layout so that it can fully cover the key areas of the expert's operation and ensure data integrity, multiple fixed cameras are used to shoot from the main view, top view, side view, oblique view and other angles to ensure that there is no blind spot coverage of the operation area. Some cameras can be installed on the robotic arm, and through remote control, the task object can be ensured to appear consistently in the camera's field of view while executing the task. Figure 3 As shown in the figure, the primary view camera is mounted on a fixed bracket 1 with adjustable posture. This bracket 1 has multiple degrees of freedom, allowing precise adjustment of the camera's height, tilt, and orientation according to task requirements, ensuring the optimal shooting angle while maintaining stability to reduce the impact of jitter and perspective offset on data acquisition. This embodiment uses multiple RGB-D cameras, such as Intel RealSense and Azure Kinect, and simultaneously adjusts the camera position to ensure the simultaneous acquisition of high-quality RGB and depth information, improving the quality of 3D data.

[0048] Step S2: Calculate the alignment weight of each fixed camera based on the video image sequence of each fixed camera, including:

[0049] Step S2.1: Calculate the clarity and noise level of the image sequence based on the video image sequence of each fixed camera;

[0050] Step S2.2: Calculate the comprehensive score of each fixed camera based on the clarity and noise level of the image sequence, using the following formula: ;

[0051] in, is the comprehensive score of the i-th fixed camera, is the noise level of the image sequence of the i-th fixed camera, is the clarity of the image sequence of the i-th fixed camera, To minimize the denominator.

[0052] Among them, the clarity parameter S i =Var(Laplacian(I)), noise level , where Var is the variance of the image to be calculated, and Laplacian is an image processing filter used to highlight the edges and details in the image. A clear image usually contains rich edge information and has a strong Laplacian response. I is the image matrix, whose elements is the pixel at coordinates The intensity value at . is the kth smooth area in the image, K represents a randomly selected smooth area in the image The number of

[0053] Step S2.3: Calculate the alignment weight of each fixed camera based on the comprehensive score of each fixed camera. The calculation formula is as follows: ;

[0054] in, is the alignment weight of the i-th fixed camera, exp is the power operation of the natural logarithm e, To control the sensitivity of the weight distribution, a larger sensitivity coefficient is used. This results in a greater sensitivity to score differences, favoring the highest-scoring camera (approaching a hard decision). A smaller sensitivity coefficient results in a more even distribution of weights, less biased toward the highest-scoring camera (approaching a smoother convergence). In other words, for example, if the weights are 0.1, 0.1, and 0.8, multiplied by 10, the difference between them becomes greater, while multiplied by 0.1, the difference becomes smaller. is the comprehensive score of the i-th fixed camera, is the comprehensive score of the jth fixed camera, is the bias of the i-th fixed camera, is the bias of the jth fixed camera. Its function is to select the main camera when the main camera’s score parameter is slightly lower than the comprehensive score of the remaining cameras. M is the number of fixed cameras.

[0055] Step S3: The fixed camera corresponding to the one with the highest alignment weight is used as the reference camera, and the video image sequences of the other fixed cameras are aligned with the video image sequence of the reference camera according to the time standard of the reference camera;

[0056] In the prior art, a certain camera is usually designated as the reference camera, and other cameras are aligned with the reference camera. However, in actual applications, this cannot meet the needs of different task scenarios, because different task scenarios have different requirements for image information. Each time the task scene is changed, the selection of the reference camera needs to be adjusted. Therefore, this application proposes a mobile alignment method, so that the method of this application can be applied to different task scenarios.

[0057] In this embodiment, the alignment weight of each fixed camera is the highest for the main view. The high-quality main view camera is used as the benchmark to align the data of other cameras so that the data acquired at the same time point from all view angles are consistent, thereby improving the quality of the image data. The image of each camera is given a comprehensive score based on the clarity parameters and noise level. , according to the comprehensive scores of the three cameras, the weight parameters are dynamically smoothed and normalized (softmax) Use the comparison method to select the perspective and synchronize data.

[0058] Step S4: Calculate the fusion weight of each fixed camera according to the loss function of the action in imitation learning. The calculation formula is as follows: ; ;

[0059] in, is the loss function for the action of the i-th fixed camera in imitation learning, is the predicted action of the jth imitation learning for the i-th fixed camera, is the action demonstrated by the expert in the jth imitation learning of the i-th fixed camera, N is the batch size in the imitation learning, is the fusion weight of the i-th fixed camera in the t+1 round of imitation learning training, exp is the power operation of the natural logarithm e, is the temperature parameter, is the loss function of the action of the i-th fixed camera in the t-th round of imitation learning training, is the loss function of the action of the jth fixed camera in the tth round of imitation learning training, and M is the number of cameras.

[0060] Step S5: splicing the aligned video image sequences of multiple fixed cameras according to the fusion weight of each fixed camera to obtain a fused image sequence, including:

[0061] Stitching the aligned video image sequences of the multiple fixed cameras to obtain a stitched image sequence;

[0062] In the stitched image sequence, the aligned fixed camera video image sequence with the smallest fusion weight is smeared, and the stitched image sequence after smearing is used as the fused image sequence.

[0063] In the prior art, if the fused image does not meet the requirements, people usually think of modifying the fusion algorithm to make the fusion algorithm more suitable for different scene requirements. However, this application adopts the method of adjusting the fusion weight to make the fusion result robust and applicable to different scene requirements.

[0064] In this embodiment, the image data is spliced ​​at the raw data level, the number of cycles is randomly selected, and different weight parameters are used to randomly select different cameras to smear the raw data, thereby adjusting the data set and improving the robustness of the trained model.

[0065] This embodiment proposes a method for image acquisition and fusion, which uses multiple fixed cameras to obtain video image sequences from the main view, top view, side view and oblique view respectively, and uses alignment weights to align the video image sequences of other fixed cameras with the video image sequence of the reference camera. Finally, the fusion weights are used to splice the aligned video image sequences of the multiple fixed cameras to obtain a fused image sequence. The present application obtains high-quality visual data, which helps to improve the model training effect of imitation learning, improves the success rate of robot tasks, and enhances the generalization ability of robot task operations. The method of the present application can be applied to a variety of task scenarios, and the flexible camera position arrangement scheme can be applied to different industrial production, medical assistance, home services and other scenarios. At the same time, it reduces the need for manual adjustment and reduces data processing costs. For sharing image data sets between different devices, the rationality of the position of the main view camera improves the data acquisition quality, and the optimized camera position design improves data integrity and reduces occlusion and information loss problems.

[0066] Example 2:

[0067] This embodiment proposes a device for image acquisition and fusion, such as Figure 2 As shown, it includes: an image acquisition module, an alignment weight calculation module, a fusion weight calculation module, and an image fusion module, wherein the image acquisition module is connected to the alignment weight calculation module, the alignment weight calculation module is connected to the fusion weight calculation module, and the fusion weight calculation module is connected to the image fusion module;

[0068] An image acquisition module is used to acquire video image sequences from multiple fixed cameras in imitation learning;

[0069] An alignment weight calculation module is used to calculate the alignment weight of each fixed camera based on the video image sequence of each fixed camera;

[0070] An image alignment module is configured to align the video image sequences of other fixed cameras with the video image sequence of the reference camera according to the time standard of the reference camera, using the corresponding fixed camera with the highest alignment weight as the reference camera;

[0071] The fusion weight calculation module is used to calculate the fusion weight of each fixed camera according to the loss function of the action in imitation learning;

[0072] The image fusion module is used to stitch the video image sequences of multiple aligned fixed cameras according to the fusion weight of each fixed camera to form a fused image sequence.

[0073] The image acquisition module includes multiple fixed brackets 1, fixed cameras 2 are installed on the fixed brackets 1, and the height and angle of the fixed cameras 2 are adjusted by the fixed brackets 1. The multiple fixed cameras 2 are used to obtain video image sequences from the main view, top view, side view and oblique view respectively, such as Figure 3 As shown, a plurality of fixed brackets 1 are installed on the mobile robot platform 3 , and a fixed camera 2 is installed on the fixed brackets 1 .

[0074] Example 3:

[0075] This embodiment proposes an electronic device, comprising: one or more processors, and a memory, wherein the memory is used to store instructions. When the instructions are executed by the one or more processors, the one or more processors execute the method for dynamic prediction and multi-strategy configuration of online car-hailing boarding points.

[0076] The electronic device can be a mobile phone, computer, or tablet computer, and includes a memory and a processor. The memory stores a computer program that, when executed by the processor, implements the method for dynamic prediction and multi-strategy configuration of online ride-hailing pickup points as described in the embodiments. It is understood that the electronic device may also include an input / output (I / O) interface and a communication component.

[0077] The processor is configured to execute all or part of the steps of the method for dynamically predicting and configuring online ride-hailing pickup points as described in the above-described embodiment. The memory is configured to store various types of data, such as instructions for any application or method in the electronic device, as well as data related to the application.

[0078] The processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and is used to execute the dynamic prediction and multi-strategy configuration method for online car-hailing boarding points described in the above embodiments.

[0079] Example 4:

[0080] This embodiment provides a computer-readable storage medium storing executable instructions. When the instructions are executed, if they are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0081] The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the dynamic prediction and multi-strategy configuration method for online car-hailing boarding points described in various embodiments of this application.

[0082] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (for example, SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR abbreviation, memory data register) memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, CD, server, APP (Application, abbreviation of application software) application store and other media that can store program verification codes, on which computer programs are stored. When the computer program is executed by the processor, it can implement the various steps of the above-mentioned online car-hailing boarding point dynamic prediction and multi-strategy configuration method.

[0083] The various embodiments in this application are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0084] The scope of protection of this application is not limited to the above-described embodiments. Obviously, those skilled in the art may make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, the disclosure is intended to include such modifications and variations.

Claims

1. A method for image acquisition and fusion, characterized in that: include: In imitation learning, multiple video image sequences from fixed cameras are acquired; Calculate the alignment weight of each fixed camera based on the video image sequence of each fixed camera; The fixed camera with the highest alignment weight is used as the reference camera. The video image sequences of other fixed cameras are aligned with the video image sequence of the reference camera according to the time standard of the reference camera. Calculate the fusion weight of each fixed camera according to the loss function of the action in imitation learning; According to the fusion weight of each fixed camera, the video image sequences of multiple aligned fixed cameras are spliced ​​to obtain a fused image sequence; Calculating the alignment weight of each fixed camera according to the video image sequence of each fixed camera includes: Calculate the clarity and noise level of the image sequence based on the video image sequence of each fixed camera; Calculate a comprehensive score for each fixed camera based on the clarity and noise level of the image sequence; The alignment weight of each fixed camera is calculated according to the comprehensive score of each fixed camera.

2. The image acquisition and fusion method according to claim 1, characterized in that: The comprehensive score of each fixed camera is calculated as follows: Among them, Q i is the comprehensive score of the i-th fixed camera, N i is the noise level of the image sequence of the i-th fixed camera, S i is the clarity of the image sequence of the i-th fixed camera, and ε is the minimum value to prevent the denominator from being zero.

3. The image acquisition and fusion method according to claim 1, characterized in that: According to the comprehensive score of each fixed camera, the alignment weight of each fixed camera is calculated as follows: Among them, ω i is the alignment weight of the i-th fixed camera, exp is the power operation of the natural logarithm e, λ is the sensitivity coefficient that controls the weight distribution, Q i is the comprehensive score of the i-th fixed camera, Q j is the comprehensive score of the jth fixed camera, b i is the bias of the i-th fixed camera, b j is the bias of the jth fixed camera, and M is the number of fixed cameras.

4. The image acquisition and fusion method according to claim 1, characterized in that: The fusion weight of each fixed camera is calculated based on the loss function of the action in imitation learning. The calculation formula is as follows: Among them, L i is the loss function for the action of the i-th fixed camera in imitation learning, is the predicted action of the i-th fixed camera in the j-th imitation learning, A i,j is the action demonstrated by the expert in the jth imitation learning of the i-th fixed camera, N is the batch size in the imitation learning, is the fusion weight of the i-th fixed camera in the t+1 round of imitation learning training, exp is the power operation of the natural logarithm e, λ is the sensitivity coefficient for controlling the weight distribution, M is the number of fixed cameras, is the loss function of the action of the i-th fixed camera in the t-th round of imitation learning training, is the loss function of the action of the j-th fixed camera in the t-th round of imitation learning training.

5. The image acquisition and fusion method according to claim 1, characterized in that: The method of stitching the aligned video image sequences of the multiple fixed cameras according to the fusion weight of each fixed camera to obtain a fused image sequence includes: Stitching the aligned video image sequences of the multiple fixed cameras to obtain a stitched image sequence; In the stitched image sequence, the aligned fixed camera video image sequence with the smallest fusion weight is smeared, and the stitched image sequence after smearing is used as the fused image sequence.

6. An image acquisition and fusion device, characterized in that: include: An image acquisition module is used to acquire video image sequences from multiple fixed cameras in imitation learning; An alignment weight calculation module is used to calculate the alignment weight of each fixed camera based on the video image sequence of each fixed camera; An image alignment module is configured to align the video image sequences of other fixed cameras with the video image sequence of the reference camera according to the time standard of the reference camera, using the corresponding fixed camera with the highest alignment weight as the reference camera; The fusion weight calculation module is used to calculate the fusion weight of each fixed camera according to the loss function of the action in imitation learning; The image fusion module is used to stitch the video image sequences of multiple aligned fixed cameras according to the fusion weight of each fixed camera to form a fused image sequence.

7. The image acquisition and fusion device according to claim 6, characterized in that: The image acquisition module includes multiple fixed brackets, fixed cameras are installed on the fixed brackets, and the height and angle of the fixed cameras are adjusted by the fixed brackets. The multiple fixed cameras are used to obtain video image sequences from the main view, top view, side view and oblique view respectively.

8. An electronic device, characterized in that: include: One or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the image acquisition and fusion method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that It stores executable instructions, which, when executed, enable a processor to execute the image acquisition and fusion method described in any one of claims 1 to 5.

Images