An image reconstruction method, device, equipment and medium of a capsule robot

Abstract

The application discloses an image reconstruction method, device and equipment of a capsule robot and a medium. The method comprises the following steps: acquiring a magnetic field conversion matrix corresponding to an external magnetic field module in the capsule robot; wherein the external magnetic field module is used for controlling the pose of a binocular camera in the capsule robot; determining a target camera conversion matrix corresponding to the binocular camera based on the magnetic field conversion matrix; and performing three-dimensional reconstruction on point cloud data collected by the binocular camera based on the target camera conversion matrix to obtain a reconstructed image. The embodiment of the application solves the problem that the camera pose is dependent on image features in the existing image reconstruction process, improves the accuracy of the camera pose, and further improves the quality of the image reconstruction.

Description

Technical Field

[0001] This invention relates to the field of image reconstruction technology, and in particular to an image reconstruction method, apparatus, device, and medium for a capsule robot. Background Technology

[0002] With the development of robotics technology, researchers have developed a variety of capsule robots for gastrointestinal examinations, which can change the current methods of diagnosing and treating gastrointestinal diseases, promote the development of medical robots in the field of medical devices, and to a certain extent promote the medical development process in the field of gastrointestinal examination and treatment.

[0003] Currently, relevant documents have disclosed the use of binocular vision technology to control capsule robots for image acquisition, and the use of external magnetic field modules for magnetic control to achieve continuous orientation positioning of capsule robots, thereby improving the working efficiency of capsule robots and enabling flexible adjustment of camera posture.

[0004] However, most mainstream image reconstruction methods rely on the parallax principle when estimating camera pose, assuming that the parallax variation between adjacent pixels in an image is uniform and smooth. But the parallax principle does not hold true in sloping areas or at the edges of objects, resulting in a "parallax gradient" problem. This leads to inaccurate estimated camera pose and poor image reconstruction results. Summary of the Invention

[0005] This invention provides an image reconstruction method, apparatus, device, and medium for capsule robots to solve the problem that camera pose depends on image features in existing image reconstruction processes, thereby improving the accuracy of camera pose and thus improving the quality of image reconstruction.

[0006] An embodiment of the present invention provides an image reconstruction method for a capsule robot, the method comprising:

[0007] Obtain the magnetic field transformation matrix corresponding to the external magnetic field module in the capsule robot; wherein, the external magnetic field module is used to control the pose of the binocular camera in the capsule robot;

[0008] Based on the magnetic field conversion matrix, the target camera conversion matrix corresponding to the binocular camera is determined;

[0009] Based on the target camera transformation matrix, the point cloud data acquired by the binocular camera is reconstructed in three dimensions to obtain a reconstructed image.

[0010] According to another embodiment of the present invention, an image reconstruction apparatus for a capsule robot is provided, the apparatus comprising:

[0011] A magnetic field conversion matrix acquisition module is used to acquire the magnetic field conversion matrix corresponding to the external magnetic field module in the capsule robot; wherein, the external magnetic field module is used to control the pose of the binocular camera in the capsule robot;

[0012] The target camera conversion matrix determination module is used to determine the target camera conversion matrix corresponding to the binocular camera based on the magnetic field conversion matrix.

[0013] The image reconstruction determination module is used to perform three-dimensional reconstruction of the point cloud data acquired by the binocular camera based on the target camera transformation matrix to obtain a reconstructed image.

[0014] According to another embodiment of the present invention, a capsule robot is provided, the capsule robot comprising: an external magnetic field module, an internal magnetic field module, a binocular camera, and a controller;

[0015] The external magnetic field module is used to control the pose of the binocular camera through the internal magnetic field module.

[0016] The internal magnetic field module and the binocular camera are installed inside the shell of the capsule robot, and the binocular camera is used to collect point cloud data;

[0017] The controller includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the image reconstruction method for the capsule robot according to any embodiment of the present invention.

[0018] According to another embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the image reconstruction method of the capsule robot according to any embodiment of the present invention.

[0019] The technical solution of this invention obtains the magnetic field transformation matrix corresponding to the external magnetic field module in the capsule robot. The external magnetic field module is used to control the pose of the binocular camera in the capsule robot. Based on the magnetic field transformation matrix, the target camera transformation matrix corresponding to the binocular camera is determined. Based on the target camera transformation matrix, the point cloud data collected by the binocular camera is reconstructed in three dimensions to obtain the reconstructed image. This solution considers the correlation between the pose of the external magnetic field module and the pose of the binocular camera, solves the problem that the camera pose depends on image features in the existing image reconstruction process, improves the accuracy of the camera pose, and thus improves the quality of image reconstruction.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of the structure of a capsule robot provided in one embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram illustrating the visual acquisition range of a binocular camera according to an embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of the structure of a terminal device provided in one embodiment of the present invention;

[0025] Figure 4 A flowchart illustrating an image reconstruction method for a capsule robot provided in one embodiment of the present invention;

[0026] Figure 5 A flowchart illustrating another image reconstruction method for a capsule robot provided in one embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram illustrating the solution of a pose objective function according to an embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of the structure of an image reconstruction device for a capsule robot provided in one embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," "initial," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] Figure 1 This is a schematic diagram of the structure of a capsule robot provided in one embodiment of the present invention. Figure 1 As shown, the capsule robot 10 includes an external magnetic field module 110, an internal magnetic field module 120, a binocular camera 130, and a controller. Figure 1 (Not shown in the image). The external magnetic field module 110 is used to control the pose of the binocular camera 130 through the internal magnetic field module 120. The internal magnetic field module 120 and the binocular camera 130 are disposed inside the shell 140 of the capsule robot 10. The binocular camera 130 is used to collect point cloud data.

[0032] For example, the external magnetic field module 110 can be a permanent magnet, and the internal magnetic field module 120 can be a Halbach array magnetic field module. The types of magnetic fields for the external magnetic field module 110 and the internal magnetic field module 120 are not limited here. Specifically, the external magnetic field module 110 can generate an environmental magnetic field, thereby controlling changes in the magnetic field of the internal magnetic field module 120. The force and torque generated by the change in the magnetic field can cause the binocular camera in the capsule robot 10 to move.

[0033] Specifically, the binocular cameras 130 are placed on the same horizontal line, and the parallax is inversely proportional to the depth, that is, the parallax of distant objects in the image is small, and the parallax of nearby objects is large.

[0034] When selecting a binocular camera 130, the following can be considered: try to place the binocular camera 130 as close as possible to the side of the housing 140; the closer the distance, the higher the accuracy. Use a camera with a large focal length and high resolution, which is beneficial for high-precision point cloud acquisition. When acquiring point clouds, ideally, the baseline between the binocular cameras 130 should be larger to improve the acquisition accuracy.

[0035] Figure 2This is a schematic diagram illustrating the visual acquisition range of a binocular camera according to an embodiment of the present invention. Specifically, the center distance of the binocular camera 130 is 9mm, the shooting angle is 120°, the initial length of the overlapping field of view is 2.6mm, the major axis of the overlapping field of view is 24.6mm, and the focusing length is 15mm. It should be noted that the field of view acquisition parameters of the binocular camera 130 are only illustrative examples here and are not intended to limit it. Users can customize the field of view acquisition parameters of the binocular camera 130 according to their actual needs.

[0036] In one alternative embodiment, the housing 140 has protruding structures at both ends along the arrangement direction of the binocular cameras 130. For example... Figure 1 As shown, the internal magnetic field module 120 is located at the bottom of the binocular camera 130. This arrangement allows the capsule robot 10 to possess both passive self-stabilization and active magnetic stabilization. The passive self-stabilization structure originates from the roly-poly shell design of the capsule robot 10. This structure positions the center of gravity at the bottom of the capsule robot 10, providing stable support and the ability to recover its original state under stress, while remaining horizontally in a free posture when relaxed. The active magnetic stabilization structure relies on the internal magnetic field module 120 located at the bottom of the capsule robot 10's shell 140. Under the control of the external magnetic field module 110, the internal magnetic field module 120 drives the entire capsule robot 10 to perform the required movements. Both structures maintain a downward vertical force during linear or tumbling movements, ensuring the capsule robot 10's stability against the irregular walls of the gastrointestinal tract. This allows for real-time, instantaneous control, enabling the capsule robot 10 to smoothly traverse gastrointestinal folds, reducing irritation to the gastrointestinal tract and minimizing the impact of manipulation on gastrointestinal mucus.

[0037] In an optional embodiment, the capsule robot 10 further includes an integrated circuit control board 150, which is attached to the internal magnetic field module 120. The integrated circuit control board 150 is electrically connected to the internal magnetic field module 120 and the binocular camera 130, and is communicatively connected to the external magnetic field module 110. Correspondingly, the controller is integrated on the integrated circuit control board 150.

[0038] In another alternative embodiment, the capsule robot 10 also includes a terminal device, which is communicatively connected to the external magnetic field module 110, the internal magnetic field module 120 and the binocular camera 130, respectively. Correspondingly, the controller is integrated on the terminal device.

[0039] Figure 3This is a schematic diagram of a terminal device provided according to one embodiment of the present invention. The terminal device 20 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The terminal device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0040] like Figure 3 As shown, the terminal device 20 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the terminal device 20. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0041] Multiple components in terminal device 20 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows terminal device 20 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0042] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the image reconstruction method for capsule robots provided in the above embodiments.

[0043] In some embodiments, the image reconstruction method for the capsule robot provided in the above embodiments can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the terminal device 20 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the image reconstruction method for the capsule robot described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the image reconstruction method for the capsule robot by any other suitable means (e.g., by means of firmware).

[0044] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0045] Figure 4 This is a flowchart illustrating an image reconstruction method for a capsule robot according to an embodiment of the present invention. This embodiment is applicable to the 3D reconstruction of images acquired by a capsule robot, and is particularly suitable for capsule robots containing an external magnetic field module. The method can be executed by an image reconstruction device of the capsule robot, which can be implemented in hardware and / or software and can be configured within the capsule robot. Figure 4 As shown, the method includes:

[0046] S210. Obtain the magnetic field transformation matrix corresponding to the external magnetic field module in the capsule robot.

[0047] Specifically, the magnetic field transformation matrix is ​​used to characterize the transformation of the external magnetic field module from the previous magnetic field pose to the current magnetic field pose. For example, the previous magnetic field pose is represented by M... i The current magnetic field pose is represented by M. j If expressed as such, then the magnetic field transformation matrix can be used. express.

[0048] S220. Based on the magnetic field transformation matrix, determine the target camera transformation matrix corresponding to the binocular camera.

[0049] In one optional embodiment, determining the target camera transformation matrix corresponding to the binocular camera based on the magnetic field transformation matrix includes: obtaining a first magnetization vector corresponding to the XY plane and a second magnetization vector corresponding to the YZ plane; determining the first camera transformation matrix corresponding to the XY plane based on the first magnetization vector, the rotation angular frequency of the external magnetic field module, and the length of the major axis of rotation; determining the second camera transformation matrix corresponding to the YZ plane based on the second magnetization vector; and determining the target camera transformation matrix corresponding to the binocular camera based on the first and second camera transformation matrices.

[0050] Specifically, the magnetic field transformation matrix Satisfying the formula:

[0051]

[0052] In an alternative embodiment, the binocular camera from pose T i To posture T j First camera transformation matrix Satisfying the formula:

[0053]

[0054] Binocular camera from attitude T i To posture T j Second camera transformation matrix Satisfying the formula:

[0055]

[0056] Among them, M t M represents the first magnetization vector. s Let w represent the second magnetization vector, w represent the rotational angular frequency, L represent the length of the major axis of rotation, and τ represent the transpose.

[0057] In one optional embodiment, determining the target camera transformation matrix corresponding to the stereo camera based on the first camera transformation matrix and the second camera transformation matrix includes: determining the initial camera transformation matrix corresponding to the stereo camera based on the first camera transformation matrix and the second camera transformation matrix, and using the initial camera transformation matrix as the target camera transformation matrix corresponding to the stereo camera.

[0058] S230. Based on the target camera transformation matrix, the point cloud data acquired by the binocular camera is reconstructed in three dimensions to obtain the reconstructed image.

[0059] For example, the 3D reconstruction algorithms include, but are not limited to, global stereo matching algorithms, local stereo matching algorithms, and semi-global stereo matching algorithms, etc. The 3D reconstruction algorithm used is not limited here.

[0060] The technical solution of this embodiment obtains the magnetic field transformation matrix corresponding to the external magnetic field module in the capsule robot. The external magnetic field module is used to control the pose of the binocular camera in the capsule robot. Based on the magnetic field transformation matrix, the target camera transformation matrix corresponding to the binocular camera is determined. Based on the target camera transformation matrix, the point cloud data collected by the binocular camera is reconstructed in three dimensions to obtain the reconstructed image. The correlation between the pose of the external magnetic field module and the pose of the binocular camera is considered, which solves the problem that the camera pose depends on image features in the existing image reconstruction process, improves the accuracy of the camera pose, and thus improves the quality of image reconstruction.

[0061] Figure 5 This is a flowchart illustrating another image reconstruction method for a capsule robot according to an embodiment of the present invention. This embodiment further refines the method for determining the target camera transformation matrix in the above embodiments. Figure 5 As shown, the method includes:

[0062] S310. Obtain the magnetic field transformation matrix corresponding to the external magnetic field module in the capsule robot.

[0063] S320. Obtain the first magnetization vector corresponding to the XY plane and the second magnetization vector corresponding to the YZ plane of the magnetic field transformation matrix.

[0064] S330. Based on the first magnetization vector, the rotation angular frequency of the external magnetic field module, and the length of the major axis of rotation, determine the first camera transformation matrix corresponding to the XY plane.

[0065] S340. Based on the second magnetization vector, determine the second camera transformation matrix corresponding to the YZ plane.

[0066] S350. Based on the first camera transformation matrix and the second camera transformation matrix, determine the initial camera transformation matrix corresponding to the stereo camera.

[0067] S360. Using the initial camera transformation matrix as the initial value of the pose objective function, the target camera transformation matrix is ​​obtained by optimizing the pose objective function.

[0068] For example, the definition For a set of potentially overlapping local 3D point cloud data acquired from different viewpoints (i.e., different camera poses), define For a point cloud set, where, p k and q kThese are three-dimensional spatial points in point cloud sets P and Q, respectively.

[0069] In an optional embodiment, the pose objective function satisfies the formula:

[0070]

[0071] Where, N p R represents the number of point clouds in the point cloud set P. j Indicates attitude T j The corresponding attitude matrix, t j Indicates attitude T j The corresponding displacement matrix, φ(p k Q) represents the actual point cloud p in the point cloud set P. k Transform to the corresponding mapping function in the point cloud set Q.

[0072] Figure 6 This is a schematic diagram illustrating the solution of a pose objective function according to an embodiment of the present invention. Specifically, Figure 6 In the diagram, the x-axis represents the camera transformation matrix, the y-axis represents the pose objective function, and the arrows indicate the iteration direction. A1 represents the initial value of the pose objective function determined using existing technology, and A2 represents the target value obtained by optimizing the pose objective function based on the initial value A1. The x-axis of this target value corresponds to the target camera transformation matrix. For example, the initial value A1 can be randomly generated or obtained based on image features. B1 represents the initial camera transformation matrix obtained using the solution method provided in this embodiment of the invention, and this initial camera transformation matrix is ​​used as the initial value of the pose objective function. B2 represents the target value obtained by optimizing the pose objective function based on the initial value B1. The x-axis of this target value corresponds to the target camera transformation matrix.

[0073] pass Figure 6 Compared to the initial values ​​determined by existing technologies, the initial camera transformation matrix provided by the embodiments of the present invention is closer to the true camera transformation matrix. Therefore, the accuracy of the target camera transformation matrix obtained by optimizing the pose objective function is higher.

[0074] S370. Based on the target camera transformation matrix, the point cloud data acquired by the binocular camera is reconstructed in three dimensions to obtain the reconstructed image.

[0075] In one optional embodiment, the point cloud data includes left-side point cloud data and right-side point cloud data. Accordingly, based on the target camera transformation matrix, the point cloud data acquired by the binocular camera is reconstructed in three dimensions to obtain a reconstructed image, including: constructing an energy function based on the target camera transformation matrix using the parallax principle; optimizing and solving the energy function to obtain a disparity map corresponding to the left-side and right-side point cloud data acquired by the binocular camera; and determining the reconstructed image of the capsule robot based on the disparity map and the camera parameters of the binocular camera.

[0076] In an alternative embodiment, the energy function satisfies the formula:

[0077]

[0078] Where D represents the disparity map, C(p, D) p ) represents the data item representing the matching cost, p represents the point cloud in the point cloud set P, P1 and P2 represent two functions with a difference, and S i Indicates the stereo camera at attitude T i The point cloud collection collected below, S j Indicates the stereo camera at attitude T j The collection of point clouds collected below.

[0079] The advantage of this setup is that, utilizing the principle of parallax, there are constraints on stereo matching, and the search range for parallax is finite; it cannot be searched within an infinitely large interval. Since real-world objects are continuous, the parallax in an image should also change uniformly and smoothly. However, large jumps in parallax can occur at the edges of objects. This embodiment of the invention simplifies the energy function by combining the capsule robot's pose T dataset with the SGM algorithm, thereby improving the computational speed of 3D reconstruction.

[0080] The technical solution of this embodiment determines the initial camera transformation matrix corresponding to the stereo camera based on the first camera transformation matrix and the second camera transformation matrix. The initial camera transformation matrix is ​​used as the initial value of the pose objective function, and the target camera transformation matrix is ​​obtained by optimizing the pose objective function. This solves the problem of low accuracy of the obtained camera transformation matrix. Compared with the randomly generated initial value in the prior art, the embodiment of this invention can converge the pose objective function to the global optimum faster and more accurately, and is less likely to get trapped in local optima, thereby improving the accuracy of the camera transformation matrix and further improving the quality of the reconstructed image.

[0081] Figure 7 This is a schematic diagram of the structure of an image reconstruction device for a capsule robot provided in one embodiment of the present invention. Figure 7As shown, the device includes: a magnetic field conversion matrix acquisition module 410, a target camera conversion matrix determination module 420, and a reconstructed image determination module 430.

[0082] The magnetic field conversion matrix acquisition module 410 is used to acquire the magnetic field conversion matrix corresponding to the external magnetic field module in the capsule robot; the external magnetic field module is used to control the pose of the binocular camera in the capsule robot.

[0083] The target camera transformation matrix determination module 420 is used to determine the target camera transformation matrix corresponding to the binocular camera based on the magnetic field transformation matrix.

[0084] The image reconstruction determination module 430 is used to perform three-dimensional reconstruction of the point cloud data acquired by the stereo camera based on the target camera transformation matrix to obtain the reconstructed image.

[0085] The technical solution of this embodiment obtains the magnetic field transformation matrix corresponding to the external magnetic field module in the capsule robot. The external magnetic field module is used to control the pose of the binocular camera in the capsule robot. Based on the magnetic field transformation matrix, the target camera transformation matrix corresponding to the binocular camera is determined. Based on the target camera transformation matrix, the point cloud data collected by the binocular camera is reconstructed in three dimensions to obtain the reconstructed image. The correlation between the pose of the external magnetic field module and the pose of the binocular camera is considered, which solves the problem that the camera pose depends on image features in the existing image reconstruction process, improves the accuracy of the camera pose, and thus improves the quality of image reconstruction.

[0086] Based on the above embodiments, optionally, the target camera transformation matrix determination module 420 includes:

[0087] The magnetization vector acquisition unit is used to acquire the first magnetization vector corresponding to the XY plane and the second magnetization vector corresponding to the YZ plane of the magnetic field transformation matrix;

[0088] The first camera transformation matrix determination unit is used to determine the first camera transformation matrix corresponding to the XY plane based on the first magnetization vector, the rotation angular frequency of the external magnetic field module, and the length of the major axis of rotation.

[0089] The second camera transformation matrix determination unit is used to determine the second camera transformation matrix corresponding to the YZ plane based on the second magnetization vector.

[0090] The target camera transformation matrix determination unit is used to determine the target camera transformation matrix corresponding to the stereo camera based on the first camera transformation matrix and the second camera transformation matrix.

[0091] Based on the above embodiments, optionally, the binocular camera from pose T i To posture T j First camera transformation matrix Satisfying the formula:

[0092]

[0093] Binocular camera from attitude T i To posture T j Second camera transformation matrix Satisfying the formula:

[0094]

[0095] Among them, M t M represents the first magnetization vector. s Let w represent the second magnetization vector, w represent the rotational angular frequency, L represent the length of the major axis of rotation, and τ represent the transpose.

[0096] Based on the above embodiments, optionally, the target camera transformation matrix determination unit is specifically used for:

[0097] Based on the first camera transformation matrix and the second camera transformation matrix, determine the initial camera transformation matrix corresponding to the stereo camera;

[0098] The initial camera transformation matrix is ​​used as the initial value of the pose objective function, and the target camera transformation matrix is ​​obtained by optimizing the pose objective function.

[0099] Based on the above embodiments, optionally, the pose objective function satisfies the formula:

[0100]

[0101] Where, N p R represents the number of point clouds in the point cloud set P. j Indicates attitude T j The corresponding attitude matrix, t j Indicates attitude T j The corresponding displacement matrix, φ(p k Q) represents the actual point cloud p in the point cloud set P. k Transform to the corresponding mapping function in the point cloud set Q.

[0102] Based on the above embodiments, optionally, the reconstructed image determination module 430 is specifically used for:

[0103] An energy function is constructed based on the parallax principle and the target camera transformation matrix.

[0104] The energy function is optimized to obtain the disparity map corresponding to the left and right point cloud data acquired by the binocular camera;

[0105] Based on the disparity map and the camera parameters of the binocular camera, the reconstructed image of the capsule robot is determined.

[0106] Based on the above embodiments, optionally, the energy function satisfies the formula:

[0107]

[0108] Where D represents the disparity map, C(p, D) p ) represents the data item representing the matching cost, p represents the point cloud in the point cloud set P, P1 and P2 represent two functions with a difference, and S i Indicates the stereo camera at attitude T i The point cloud collection collected below, S j Indicates the stereo camera at attitude T j The collection of point clouds collected below.

[0109] The image reconstruction device for capsule robots provided in this embodiment of the invention can execute the image reconstruction method for capsule robots provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0110] One embodiment of the present invention also provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform an image reconstruction method for a capsule robot.

[0111] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0112] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on a terminal device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the terminal device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0114] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0115] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0116] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0117] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An image reconstruction method of a capsule robot, characterized by, include: Obtain the magnetic field transformation matrix corresponding to the external magnetic field module in the capsule robot; wherein, the external magnetic field module is used to control the pose of the binocular camera in the capsule robot; Based on the magnetic field conversion matrix, the target camera conversion matrix corresponding to the binocular camera is determined; Based on the target camera transformation matrix, the point cloud data acquired by the binocular camera is reconstructed in three dimensions to obtain a reconstructed image; The step of determining the target camera transformation matrix corresponding to the binocular camera based on the magnetic field transformation matrix includes: Obtain the first magnetization vector corresponding to the XY plane and the second magnetization vector corresponding to the YZ plane of the magnetic field transformation matrix; Based on the first magnetization vector, the rotation angular frequency of the external magnetic field module, and the length of the major axis of rotation, the first camera transformation matrix corresponding to the XY plane is determined; Based on the second magnetization vector, determine the second camera transformation matrix corresponding to the YZ plane; Based on the first camera transformation matrix and the second camera transformation matrix, the target camera transformation matrix corresponding to the stereo camera is determined.

2. The method according to claim 1, characterized in that, The binocular camera transforms from pose to pose of the first camera transform matrix satisfies the equation: The binocular camera transforms from pose to pose of the second camera satisfies the equation: in, Denotes the first magnetization vector. Indicates the second magnetization vector. Indicates the rotational angular frequency. Indicates the length of the major axis of rotation. This indicates transpose.

3. The method according to claim 1, characterized in that, Determining the target camera transformation matrix corresponding to the stereo camera based on the first camera transformation matrix and the second camera transformation matrix includes: Based on the first camera transformation matrix and the second camera transformation matrix, the initial camera transformation matrix corresponding to the stereo camera is determined; The initial camera transformation matrix is ​​used as the initial value of the pose objective function, and the target camera transformation matrix is ​​obtained by optimizing the pose objective function.

4. The method according to claim 3, characterized in that, The pose objective function satisfies the following formula: in, Represents a set of point clouds The number of midpoint clouds, Indicates posture The corresponding attitude matrix, Indicates posture The corresponding displacement matrix, This represents a real collection of point clouds. Midpoint cloud Convert to point cloud collection The corresponding mapping function in Represents a set of point clouds The sequence number of the midpoint cloud.

5. The method according to any one of claims 1-4, characterized in that, The point cloud data includes left-side point cloud data and right-side point cloud data. Correspondingly, the step of performing 3D reconstruction on the point cloud data acquired by the binocular camera based on the target camera transformation matrix to obtain a reconstructed image includes: Using the parallax principle, an energy function is constructed based on the target camera transformation matrix; The energy function is optimized to obtain the disparity map corresponding to the left and right point cloud data acquired by the binocular camera. Based on the disparity map and the camera parameters of the binocular camera, the reconstructed image of the capsule robot is determined.

6. The method according to claim 5, characterized in that, The energy function satisfies the following formula: in, Represents a disparity map. The data item representing the matching cost, Represents a set of point clouds Point clouds in and Two functions with a difference, Indicates the attitude of the stereo camera The point cloud collection collected below, Indicates the attitude of the stereo camera The point cloud collection collected below, Point The corresponding depth parameters, Represents the feature points in the point cloud set. Indicates from posture to posture The transformation matrix.

7. An image reconstruction device for a capsule robot, characterized in that, include: A magnetic field conversion matrix acquisition module is used to acquire the magnetic field conversion matrix corresponding to the external magnetic field module in the capsule robot; wherein, the external magnetic field module is used to control the pose of the binocular camera in the capsule robot; The target camera conversion matrix determination module is used to determine the target camera conversion matrix corresponding to the binocular camera based on the magnetic field conversion matrix. The image reconstruction determination module is used to perform three-dimensional reconstruction of the point cloud data acquired by the binocular camera based on the target camera transformation matrix to obtain a reconstructed image; The target camera transformation matrix determination module includes: The magnetization vector acquisition unit is used to acquire the first magnetization vector corresponding to the XY plane and the second magnetization vector corresponding to the YZ plane of the magnetic field conversion matrix; The first camera transformation matrix determination unit is used to determine the first camera transformation matrix corresponding to the XY plane based on the first magnetization vector, the rotation angular frequency of the external magnetic field module, and the length of the major axis of rotation. The second camera transformation matrix determination unit is used to determine the second camera transformation matrix corresponding to the YZ plane based on the second magnetization vector. The target camera transformation matrix determination unit is used to determine the target camera transformation matrix corresponding to the stereo camera based on the first camera transformation matrix and the second camera transformation matrix.

8. A capsule robot, characterized in that, The capsule robot includes: an external magnetic field module, an internal magnetic field module, a binocular camera, and a controller; The external magnetic field module is used to control the pose of the binocular camera through the internal magnetic field module. The internal magnetic field module and the binocular camera are installed inside the shell of the capsule robot, and the binocular camera is used to collect point cloud data; The controller includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the image reconstruction method for the capsule robot according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the image reconstruction method for the capsule robot according to any one of claims 1-6.

Images