An image processing method, a calibration system and related devices

Abstract

This application discloses an image processing method applicable to electronic device positioning scenarios. The method includes: acquiring multiple images, each image including a calibration body and an electronic device rigidly connected to the calibration body, the electronic device having a light spot and containing an inertial measurement unit (IMU); determining a first relative positional relationship between the calibration body and the light spot based on the multiple images; acquiring a second relative positional relationship between the calibration body and the IMU based on the multiple images and IMU data; and determining a third relative positional relationship between the light spot and the IMU based on the first and second relative positional relationships, the third relative positional relationship being used to position the electronic device. The third relative positional relationship between the light spot and the IMU is determined using the calibration body rigidly connected to the electronic device as an intermediate medium. Based on this third relative positional relationship, the measurement data output by the IMU can be correctly represented on the coordinate system of the light spot, improving the positioning accuracy of the electronic device.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method, calibration system and related equipment. Background Technology

[0002] With the popularization of augmented reality (AR), virtual reality (VR) technologies, and mixed reality (MR) technologies, AR, VR, and MR devices have been widely used in many scenarios such as work and entertainment. Unlike traditional terminal devices, current AR, VR, and MR devices use independent controllers as the primary interaction solution.

[0003] Currently, the aforementioned controllers commonly employ outside-in tracking technology for positioning. Positioning is achieved by using a helmet-mounted camera to capture images of specially designed indicator lights on the controller, combined with the controller's built-in Inertial Measurement Unit (IMU). However, due to the limited number of indicator lights on the controller (typically only a dozen or so) and their relatively low manufacturing precision, the origin of the controller's indicator light coordinate system does not coincide with the origin of the IMU coordinate system. This results in a certain positional deviation between the IMU and controller coordinate systems. Consequently, the IMU data cannot directly reflect the controller's motion, thus affecting the controller's positioning accuracy.

[0004] Therefore, how to correctly represent the measurement data output by the IMU on the coordinate system of the handle light point is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] This application provides an image processing method for calibrating the external parameters of an electronic device's external light spot and the IMU coordinate system, thereby improving the positioning accuracy of the electronic device.

[0006] The first aspect of this application provides an image processing method applicable to electronic device positioning scenarios, such as virtual reality (VR) / augmented reality (AR) games, learning, and competitions. This method can be executed by an image processing device or by a component of the image processing device (e.g., a processor, chip, or chip system). The method includes: acquiring multiple images, each image including a calibration body and an electronic device with a light spot, the calibration body and the electronic device being rigidly connected, the electronic device containing an inertial measurement unit (IMU), and the electronic device having a different position in at least two of the multiple images; determining a first relative positional relationship between the calibration body and the light spot based on the multiple images; acquiring IMU data collected by the IMU device, the IMU data being correlated with the multiple images; obtaining a second relative positional relationship between the calibration body and the IMU device based on the multiple images and the IMU data; and determining a third relative positional relationship between the light spot and the IMU coordinate system based on the first and second relative positional relationships, the third relative positional relationship being used to position the electronic device.

[0007] In this embodiment, a calibration body rigidly connected to an electronic device (e.g., a controller, aircraft, etc.) is constructed, and this calibration body is used as an intermediate medium or reference to determine the third relative positional relationship between the light spot and the IMU device. Furthermore, based on this third relative positional relationship, the measurement data output by the IMU can be correctly represented on the coordinate system of the controller's light spot, thereby improving the positioning accuracy of the electronic device.

[0008] Optionally, in one possible implementation of the first aspect, the above step of determining the first relative positional relationship between the calibration body and the light spot based on multiple images includes: obtaining the first coordinates of the calibration body in the image coordinate system based on multiple images; obtaining the second coordinates of the light spot in the image coordinate system based on multiple images; and determining the first relative positional relationship between the calibration body and the light spot based on the first coordinates and the second coordinates.

[0009] In this possible implementation, the first relative positional relationship is obtained by associating the relative position between the calibration body and the external light spot of the electronic device by using the image as a reference coordinate system.

[0010] Optionally, in one possible implementation of the first aspect, the above step of obtaining the second relative positional relationship between the calibration body and the IMU device based on multiple images and IMU data includes: determining the error between the first motion trajectory of the calibration body and the second motion trajectory of the IMU device based on multiple images and IMU data; and determining the second relative positional relationship between the calibration body and the IMU device based on the error.

[0011] In this possible implementation, the second relative positional relationship between the calibration body and the IMU device can be obtained by measuring the error between the first motion trajectory of the calibration body and the second motion trajectory of the IMU device.

[0012] Optionally, in one possible implementation of the first aspect, the above steps further include: determining the intrinsic parameters of the IMU device based on the error and IMU data.

[0013] In this possible implementation, the intrinsic parameters of the IMU device can also be determined through errors, or trajectory errors. For example, these IMU intrinsic parameters may include one or more of the following: zero-bias error of the angular velocity and linear acceleration sensors, scale error, and angle error.

[0014] A second aspect of this application provides an image processing apparatus. The image processing apparatus includes: an acquisition unit for acquiring multiple images, each image including a calibration body and an electronic device with a light spot, the calibration body and the electronic device being rigidly connected, the electronic device containing an inertial measurement unit (IMU), and the electronic device having a different position in at least two of the multiple images; a determination unit for determining a first relative positional relationship between the calibration body and the light spot based on the multiple images; the acquisition unit further for acquiring IMU data collected by the IMU device, the IMU data being correlated with the multiple images; the acquisition unit further for acquiring a second relative positional relationship between the calibration body and the IMU device based on the multiple images and the IMU data; and the determination unit further for determining a third relative positional relationship between the light spot and the IMU device based on the first and second relative positional relationships, the third relative positional relationship being used to locate the electronic device.

[0015] Optionally, in one possible implementation of the second aspect, the determining unit is specifically used to obtain the first coordinates of the calibration body in the image coordinate system based on multiple images; the determining unit is specifically used to obtain the second coordinates of the light spot in the image coordinate system based on multiple images; and the determining unit is specifically used to determine the first relative positional relationship between the calibration body and the light spot based on the first coordinates and the second coordinates.

[0016] Optionally, in one possible implementation of the second aspect, the acquisition unit described above is specifically used to determine the error between the first motion trajectory of the calibration body and the second motion trajectory of the IMU device based on multiple images and IMU data; the acquisition unit is specifically used to determine the second relative positional relationship between the calibration body and the IMU device based on the error.

[0017] Optionally, in one possible implementation of the second aspect, the aforementioned determining unit is further configured to determine the intrinsic parameters of the IMU device based on the error and IMU data.

[0018] A third aspect of this application provides a calibration system applicable to electronic device positioning scenarios, such as virtual reality (VR) / augmented reality (AR) games, learning, and competitions. The calibration system includes: a camera, an electronic device with a light spot, a calibration body, and an image processing device; wherein the electronic device is rigidly connected to the calibration body, and the electronic device contains an inertial measurement unit (IMU); the camera is used to acquire multiple images of the calibration body and the electronic device during movement; the calibration body is used to determine a third relative positional relationship between the light spot and the IMU, which is used to position the electronic device; the image processing device is used to acquire multiple images and IMU data collected by the IMU during movement, and to determine the third relative positional relationship based on the multiple images and IMU data.

[0019] Optionally, in one possible implementation of the third aspect, the image processing device described above is specifically used to: determine a first relative positional relationship between the calibration body and the light spot based on multiple images; the image processing device is specifically used to: acquire a second relative positional relationship between the calibration body and the IMU device based on multiple images and IMU data; and the image processing device is specifically used to: determine a third relative positional relationship based on the first and second relative positional relationships.

[0020] Optionally, in one possible implementation of the third aspect, the aforementioned image processing device is specifically used to determine the error between the first motion trajectory of the calibration body and the second motion trajectory of the IMU device based on multiple images and IMU data; the image processing device is specifically used to determine the second relative positional relationship between the calibration body and the IMU device based on the error.

[0021] Alternatively, in one possible implementation of the third aspect, the aforementioned image processing device is further used to determine the intrinsic parameters of the IMU device based on the error and IMU data.

[0022] A fourth aspect of this application provides an image processing apparatus, comprising: a processor coupled to a memory for storing programs or instructions, wherein when the programs or instructions are executed by the processor, the image processing apparatus implements the methods described in the first aspect or any possible implementation thereof.

[0023] The fifth aspect of this application provides a computer-readable medium having a computer program or instructions stored thereon, which, when run on a computer, cause the computer to perform the methods of the first aspect or any possible implementation thereof.

[0024] The sixth aspect of this application provides a computer program product that, when executed on a computer, causes the computer to perform the methods of the first aspect or any possible implementation thereof.

[0025] The technical effects of the second, third, fourth, fifth, and sixth aspects or any of their possible implementations can be found in the first aspect or the technical effects of different possible implementations of the first aspect, and will not be repeated here.

[0026] As can be seen from the above technical solutions, this application has the following advantages: By constructing a calibration body rigidly connected to the electronic device (e.g., a controller, aircraft, etc.), and using this calibration body as an intermediate medium or reference, the third relative positional relationship between the light spot and the IMU device can be determined. Furthermore, based on this third relative positional relationship, the measurement data output by the IMU can be correctly represented on the coordinate system of the controller's light spot, thereby improving the positioning accuracy of the electronic device. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the calibration system provided in the embodiments of this application;

[0028] Figure 2 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0029] Figure 3 A schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0030] Figure 4 This is another schematic diagram of the image processing device provided in an embodiment of this application. Detailed Implementation

[0031] This application provides an image processing method for calibrating the external parameters of an electronic device and an IMU coordinate system, thereby improving the positioning accuracy of the electronic device.

[0032] To facilitate understanding, the relevant terms and concepts mainly involved in the embodiments of this application will be introduced below.

[0033] 1. External parameters and internal parameters of IMU and electronic equipment

[0034] The extrinsic parameters of the IMU and electronic equipment can be understood as the relative positional relationship between the IMU coordinate system and the electronic equipment.

[0035] IMU internal parameters include one or more of the following: zero bias error of the angular velocity and linear acceleration sensors, scale error, and angle error.

[0036] 2. Calibration body

[0037] The calibration body in the embodiments of this application can also be called a three-dimensional calibration body. The calibration body can be a hexahedron, a sphere, a cube, etc., and is not specifically limited here.

[0038] The calibration plate serves as a fixed reference coordinate system. Since the size and pattern of the calibration plate are known, the correspondence between 3D and 2D coordinates can be obtained through the size and pattern. Furthermore, the calibration plate can be used to transform objects in an image containing the calibration plate from 2D coordinates to 3D coordinates.

[0039] For example, taking a hexahedron as the calibration body, each face of the calibration body contains a pattern of a calibration chart. The patterns on the six faces can be the same or different. Of course, in order to ensure the accuracy of determining the relative positional relationships and to distinguish the six faces, the patterns on the six faces can be different to facilitate quick differentiation of different calibration charts. The pattern or type of the calibration chart can be set according to actual needs, and is not limited here.

[0040] 3. Outside-in tracking technology

[0041] "Outside-in" tracking technology refers to positioning the device at a certain distance from the object being tracked, capturing the object's specific position parameters for localization (either by sending a signal from outside, which the object receives, or by the object transmitting a signal, which is then captured by an external device). This is the most traditional positioning technology, offering advantages such as high accuracy, low latency, and low cost. However, it requires the installation of external signal transmitters or receivers, which is inconvenient, and the positioning range is fixed.

[0042] 4. Electronic devices with light spots

[0043] The electronic device is used to locate the object as described in section 1 above. The light spot on the electronic device can be a light-emitting spot or a reflected light spot, etc.

[0044] Currently, the aforementioned electronic devices with illuminated points often employ an "outside-in" tracking positioning method. This involves using a helmet-mounted camera to capture images of specially designed illuminated points on the electronic device, combined with the device's built-in IMU (Integrated Measurement Unit) for positioning. However, due to the limited number of illuminated points on the electronic device (typically only a dozen or so) and their relatively low manufacturing precision, the origins of the illuminated points and the IMU coordinate system do not coincide. This results in a positional deviation between the IMU and electronic device coordinate systems. Consequently, the IMU data cannot directly reflect the movement of the electronic device, thus affecting its positioning accuracy.

[0045] Therefore, how to correctly represent the measurement data output by the IMU on the coordinate system of the electronic device's lamp points is a technical problem that urgently needs to be solved.

[0046] To address the aforementioned technical problems, embodiments of this application provide an image processing method. This method constructs a calibration body rigidly connected to an electronic device (e.g., a controller), and uses this calibration body as an intermediate medium or reference to determine the third relative positional relationship between the light spot and the IMU device. Furthermore, based on this third relative positional relationship, the measurement data output by the IMU can be correctly represented on the coordinate system of the controller's light spot, thereby improving the positioning accuracy of the electronic device.

[0047] Before describing the methods provided in the embodiments of this application, the application scenarios to which the methods provided in the embodiments of this application are applicable will be described first. The application scenarios of the methods provided in the embodiments of this application can be as follows: Figure 1 As shown. The scene includes: camera 101, calibration body 102, electronic device 103 with light spot, and image processing device 104.

[0048] The calibration body 102 is rigidly connected to the electronic device 103 (also known as a rigid connection), and the electronic device contains an IMU device.

[0049] Camera 101 can be a standalone camera device or a camera component located in other devices (e.g., VR / AR headsets, VR / AR glasses, etc.), and the specifics are not limited here. Camera 101 is mainly used to acquire multiple images of the calibration object 102 and the electronic device 103 during their movement.

[0050] In addition, to ensure the accuracy of subsequent electronic device positioning, the position of the camera 101 is fixed when capturing multiple images.

[0051] The calibration body 102 is mainly used to determine the third relative positional relationship between the electronic device 103 and the IMU coordinate system. The IMU coordinate system is a coordinate system with the IMU device as the origin, and the third relative positional relationship is used to locate the electronic device 103. For a further description of the calibration body 102, please refer to the explanation in the aforementioned terminology section; it will not be repeated here.

[0052] The electronic device 103 contains an IMU device internally and a light spot (e.g., a light-emitting spot or a reflected light spot) externally. In some scenarios, the electronic device 103 may also be referred to as a positioning device. During the movement of the electronic device 103, the calibration body 102, which is rigidly connected to the electronic device 103, will also move accordingly.

[0053] Image processing device 104 is used to acquire multiple images and IMU data collected by the IMU device during the movement of the electronic device; and to determine the third relative positional relationship between the light spot and the IMU device based on the multiple images and IMU data.

[0054] Optionally, there are various ways to rigidly connect the calibration body 102 and the electronic device 103. For example, the calibration body 102 and the electronic device 103 can be connected via a rigid metal or similar material. Another example is that the calibration body 102 can be considered as a platform with a fixed support on the ground, which is used to hold the electronic device in place. It is understood that this rigid connection is to ensure that the relative positions of the calibration body 102 and the electronic device 103 do not change during movement, thereby allowing the calibration body 102 to determine the third relative positional relationship between the external light spot and the internal IMU device of the electronic device 103.

[0055] Optionally, the image processing device is specifically used to: determine a first relative positional relationship between the calibration body and the light spot based on multiple images; acquire a second relative positional relationship between the calibration body and the IMU device based on multiple images and IMU data; and determine a third relative positional relationship based on the first and second relative positional relationships.

[0056] Optionally, the image processing device is specifically used to determine the error between the first motion trajectory of the calibration body and the second motion trajectory of the IMU device based on multiple images and IMU data; the image processing device is specifically used to determine the second relative positional relationship between the calibration body and the IMU device based on the error.

[0057] Optionally, the image processing device 104 can also acquire the intrinsic parameters of the IMU device based on multiple images and IMU data.

[0058] Figure 1 In the scenario shown, the image processing device 104 is connected to the camera 101 and the electronic device 103, respectively. Specifically, the image processing device 104 can be connected to the camera 101 and the electronic device 103 via wired or wireless means.

[0059] Typically, the image processing device 104 is connected to the camera 101 via a wired connection, such as fiber optic cable, Universal Serial Bus (USB), or High Definition Multimedia Interface (HDMI). The image processing device 104 is also connected to the electronic device 103 wirelessly, such as via Bluetooth, cellular networks, or Wi-Fi.

[0060] The electronic device in this embodiment contains an internal IMU device and an external light spot. This electronic device can be a controller, aircraft, mobile phone, tablet computer, wearable electronic device, virtual reality (VR) terminal device, augmented reality (AR) terminal device, etc. No specific limitations are specified here.

[0061] The image processing device in this application embodiment can be a server, mobile phone, tablet computer, portable game console, handheld digital assistant (PDA), laptop computer, ultra-mobile personal computer (UMPC), handheld computer, netbook, in-vehicle media playback device, wearable electronic device, virtual reality (VR) terminal device, augmented reality (AR) terminal device, or other device with sufficient computing power. Specific limitations are not specified here.

[0062] The image processing method provided in the embodiments of this application will be described in detail below. This method can be executed by an image processing device. It can also be executed by a component of the image processing device (e.g., a processor, chip, or chip system). Please refer to [link to relevant documentation]. Figure 2 This application provides a schematic flowchart of an image processing method, which may include steps 201 to 205. Steps 201 to 205 are described in detail below. This method can be applied to... Figure 1 The application scenarios shown are illustrated. Of course, it can also be applied to high-precision map downloads and global positioning for terminals such as unmanned vehicles, drones, autonomous robots, AR glasses, and VR headsets.

[0063] Step 201: Acquire multiple images.

[0064] Image processing devices can acquire multiple images by controlling a camera to capture images of electronic devices and calibration objects in motion. This camera can be an internal camera within the image processing device, or it can be a camera device separate from the image processing device (e.g., AR / VR headsets, glasses, etc.).

[0065] Each of the multiple images includes a calibration object and an electronic device with a light spot. The calibration object and the electronic device are rigidly connected, and the electronic device contains an IMU (Induction Unit). The electronic device is in a different position in at least two of the multiple images. These multiple images can also be understood as multiple images captured by a camera during the movement of the electronic device and the calibration object.

[0066] Step 202: Determine the first relative positional relationship between the calibration body and the light spot based on multiple images.

[0067] After acquiring multiple images, the image processing device determines the first relative positional relationship between the calibration body and the electronic device based on the multiple images.

[0068] Specifically, the image processing device can obtain the first coordinates of the calibration object in the image coordinate system based on multiple images. It can also obtain the second coordinates of the light spot in the image coordinate system based on multiple images. Finally, it determines the first relative positional relationship between the calibration object and the electronic device based on the first and second coordinates.

[0069] The above process can also be understood as obtaining the first coordinate and the second coordinate by mapping the calibration body and the electronic device to the same image coordinate system, and then determining the first relative positional relationship between the two coordinates based on the two coordinates in the same coordinate system.

[0070] Optionally, one image can correspond to one coordinate, and multiple images can correspond to a set of coordinates. In this case, the first coordinate of the multiple images mentioned above includes multiple coordinates, which can also be called a first coordinate array. The second coordinate of the multiple images includes multiple coordinates, which can also be called a second coordinate array.

[0071] For example, algorithms such as perspective-three-point (P3P) can be used to solve for the first and second coordinates mentioned above. This allows for the determination of the first relative positional relationship between the first and second coordinates.

[0072] Step 203: Acquire IMU data collected by the IMU device.

[0073] Image processing devices can also acquire IMU data through the IMU device inside the electronic device. This IMU data is related to multiple images. Specifically, the IMU device inside the electronic device collects IMU data during the movement of the electronic device and sends this IMU data to the image processing device.

[0074] The aforementioned IMU data being associated with multiple images can be understood as meaning that both the IMU data and the multiple images were acquired during the movement of the electronic device.

[0075] Of course, to increase the accuracy of subsequent relative position determination, the timestamps of multiple images and IMU data can be aligned. For example, consider an image processing device and an electronic device connected via Bluetooth. The image processing device can control the camera to capture multiple images while simultaneously acquiring IMU data from the electronic device via the Bluetooth module. It then synchronizes the Bluetooth timestamps with the camera, calibrating the camera's time system with the IMU's time system. The timestamps are used to align the timestamps of multiple images and IMU data.

[0076] The relative positional relationships in the embodiments of this application (e.g., the first relative positional relationship, the subsequent second relative positional relationship, and the third relative positional relationship) can be expressed by a rotation and translation (RT) matrix or by a coordinate array, etc., and the specific method is not limited here.

[0077] The IMU data in the embodiments of this application is related to the structure of the IMU device, and the specific embodiments are not limited here. For example, for a three-axis IMU, the IMU data may include three-axis attitude angles and acceleration. For another example, for a six-axis IMU, the IMU data may include three-axis attitude angles and acceleration. For yet another example, for a nine-axis IMU, in addition to three-axis attitude angles and acceleration, the IMU data may also include a three-axis magnetometer.

[0078] Step 204: Obtain the second relative positional relationship between the calibration body and the IMU device based on multiple images and IMU data.

[0079] After acquiring multiple images and IMU data, the image processing device can obtain the second relative positional relationship between the calibration body and the IMU device based on the multiple images and IMU data.

[0080] Specifically, the image processing device can determine the error between the first motion trajectory of the calibration body and the second motion trajectory of the IMU device using multiple images and IMU data. Then, based on this error, it determines the second relative positional relationship between the calibration body and the IMU device.

[0081] The above process can be understood as follows: First, the first motion trajectory of the calibration body is determined based on multiple images generated during the motion. Then, the second motion trajectory is solved by integration (using the IMU's default intrinsic and extrinsic parameters) based on the IMU data corresponding to the multiple images. A loss function is established to represent the difference between the first and second motion trajectories. The IMU's extrinsic parameters are trained with the goal of the loss function value being less than a certain threshold. This yields the corrected IMU extrinsic parameters. In other words, by continuously adjusting the IMU extrinsic parameters, the difference between the first and second motion trajectories becomes smaller and smaller. When the difference is less than a certain threshold or the first and second motion trajectories overlap, the IMU extrinsic parameters at that moment are determined as the second relative positional relationship.

[0082] Step 205: Determine the third relative positional relationship between the light spot and the IMU device based on the first and second relative positional relationships.

[0083] After the image processing device acquires the first relative positional relationship and the second relative positional relationship, it can determine the third relative positional relationship between the light spot and the IMU device based on the first relative positional relationship and the second relative positional relationship.

[0084] This step can also be understood as the image processing device using a calibration body as an intermediate medium or reference to determine the third relative positional relationship between the light spot and the IMU device.

[0085] Specifically, when the relative positional relationship is a matrix, the third relative positional relationship is obtained by multiplying the first and second relative positions.

[0086] Furthermore, there is no time limit between the steps in the embodiments of this application. For example, step 203 can be after step 202 or before step 201, and there is no specific limitation here.

[0087] In this embodiment, a calibration body rigidly connected to an electronic device (e.g., a controller) is constructed, and this calibration body is used as an intermediate medium or reference to determine the third relative positional relationship between the light spot and the IMU device. On one hand, the measurement data output by the IMU can be correctly represented on the coordinate system of the controller's light spot based on this third relative positional relationship, improving the positioning accuracy of the electronic device. On the other hand, the positional deviation between the IMU coordinate system and the electronic device can be determined through the third relative positional relationship, and then the deviation can be corrected, so that the IMU data can directly reflect the movement process of the electronic device.

[0088] Optionally, the IMU intrinsic parameters can be corrected using the error and IMU data from step 204 above. The IMU intrinsic and extrinsic parameters are trained with the goal of the loss function value being less than a certain threshold. This yields the corrected IMU intrinsic and extrinsic parameters. In other words, by continuously adjusting the IMU intrinsic and extrinsic parameters, the difference between the first and second motion trajectories is made smaller and smaller. When the difference is less than a certain threshold or the first and second motion trajectories overlap, the IMU extrinsic parameters at this moment are determined to be the second relative positional relationship. The IMU intrinsic parameters at this moment are then determined to be the adjusted IMU intrinsic parameters. A description of IMU memory can be found in the aforementioned terminology section and will not be repeated here.

[0089] In this approach, on the one hand, high-precision calibration of the IMU intrinsic parameters is achieved through a stereo calibration body. Because the number of LEDs in the electronic device is relatively small, there is significant jitter in calculating the electronic device's pose, making it difficult to calculate the accurate pose. This introduces too much noise during the optimization process with IMU data, resulting in inaccurate IMU intrinsic parameter calibration. This embodiment introduces a stereo calibration body as a stable positioning device. It can accurately determine the motion state of the IMU device, thereby achieving good IMU intrinsic parameter optimization results. On the other hand, the stereo calibration body allows a single camera to capture the pose of the calibration plate within the stereo calibration body from any position of the IMU.

[0090] Furthermore, to improve the calibration efficiency of the above methods, the relative positional relationships between the faces of the calibration body can be determined offline or online. This allows for the calculation of the calibration body's pose even when only one calibration plate is captured in an image.

[0091] For example, taking a hexahedron as the calibration body, different calibration patterns are used on the six faces to facilitate the differentiation of different calibration plates. The determination method mainly includes: taking multiple images of the calibration body using a camera to obtain multiple images, and selecting one calibration plate as the origin of the coordinate system. The calibration plate selected as the origin is called the first calibration plate, and the unselected calibration plates are called the other calibration plates. In each image, the six calibration plates are traversed, and the poses of the calibration plates appearing in the image are calculated, as well as the relative poses between the calibration plates. After processing multiple images, the positional relationship of all calibration plates relative to the origin is calculated. In this way, even when only one calibration plate is captured in the image, the pose of the calibration body can still be calculated.

[0092] The image processing method in the embodiments of this application has been described above. The image processing device in the embodiments of this application is described below. Please refer to [link / reference]. Figure 3 One embodiment of the image processing device in this application includes:

[0093] The acquisition unit 301 is used to acquire multiple images, each image including a calibration body and an electronic device with a light spot. The calibration body and the electronic device are rigidly connected. The electronic device contains an inertial measurement unit (IMU). The position of the electronic device is different in at least two of the multiple images.

[0094] Determining unit 302 is used to determine the first relative positional relationship between the calibration body and the light spot based on multiple images;

[0095] The acquisition unit 301 is also used to acquire IMU data collected by the IMU device, and the IMU data is related to multiple images;

[0096] The acquisition unit 301 is also used to acquire a second relative positional relationship between the calibration body and the IMU device based on multiple images and IMU data;

[0097] The determining unit 302 is also used to determine a third relative positional relationship between the light spot and the IMU device based on the first relative positional relationship and the second relative positional relationship, and the third relative positional relationship is used to locate the electronic device.

[0098] Optionally, the determining unit 302 is specifically used to obtain the first coordinates of the calibration body in the image coordinate system based on multiple images; the determining unit 302 is specifically used to obtain the second coordinates of the light spot in the image coordinate system based on multiple images; the determining unit 302 is specifically used to determine the first relative positional relationship between the calibration body and the light spot based on the first coordinates and the second coordinates.

[0099] Optionally, the acquisition unit 301 is specifically used to determine the error between the first motion trajectory of the calibration body and the second motion trajectory of the IMU device based on multiple images and IMU data; the acquisition unit 301 is specifically used to determine the second relative positional relationship between the calibration body and the IMU device based on the error.

[0100] Optionally, the determining unit 302 is also used to determine the intrinsic parameters of the IMU device based on the error and IMU data.

[0101] In this embodiment, the operations performed by each unit in the image processing device are the same as described above. Figures 1 to 2 The embodiments shown are similar and will not be repeated here.

[0102] In this embodiment, by constructing a calibration body rigidly connected to the electronic device (e.g., a controller, aircraft, etc.), the determining unit 302 uses this calibration body as an intermediate medium or reference to determine the third relative positional relationship between the light spot and the IMU device. Then, based on this third relative positional relationship, the measurement data output by the IMU can be correctly represented on the coordinate system of the controller light spot, improving the positioning accuracy of the electronic device.

[0103] See Figure 4 This application provides a schematic diagram of another image processing device. The image processing device may include a processor 401, a memory 402, and a communication port 403. The processor 401, memory 402, and communication port 403 are interconnected via lines. The memory 402 stores program instructions and data.

[0104] The aforementioned are stored in memory 402. Figure 1 and Figure 2 In the corresponding implementation shown, the program instructions and data corresponding to the steps executed by the image processing device are described.

[0105] Processor 401, for performing the aforementioned Figure 1 and Figure 2 The steps performed by the image processing device are shown in any of the embodiments illustrated.

[0106] Communication port 403 can be used for receiving and sending data, and for performing the aforementioned functions. Figure 1 and Figure 2 The steps related to acquiring, sending, and receiving in any of the embodiments shown.

[0107] In one implementation, the image processing device may include, relative to Figure 4 More or fewer components are merely illustrative in this application and are not intended to limit the scope of the application.

[0108] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image processing method, characterized in that, The method includes: Multiple images are acquired, each image including a calibration body and an electronic device with a light spot. The calibration body and the electronic device are rigidly connected. The electronic device contains an inertial measurement unit (IMU). The electronic device is in a different position in at least two of the multiple images. Based on the multiple images, a first relative positional relationship between the calibration body and the light spot is determined; Acquire IMU data collected by the IMU device, wherein the IMU data is related to the multiple images; Based on the multiple images and the IMU data, a second relative positional relationship between the calibration body and the IMU device is obtained; Based on the first relative positional relationship and the second relative positional relationship, a third relative positional relationship between the light spot and the IMU device is determined, and the third relative positional relationship is used to locate the electronic device.

2. The method according to claim 1, characterized in that, Determining the first relative positional relationship between the calibration body and the light spot based on the plurality of images includes: Based on the multiple images, obtain the first coordinates of the calibration body in the image coordinate system; Based on the multiple images, obtain the second coordinates of the light spot in the image coordinate system; The first relative positional relationship between the calibration body and the light spot is determined based on the first coordinate and the second coordinate.

3. The method according to claim 1 or 2, characterized in that, The step of obtaining the second relative positional relationship between the calibration body and the IMU device based on the multiple images and the IMU data includes: Based on the multiple images and the IMU data, determine the error between the first motion trajectory of the calibration body and the second motion trajectory of the IMU device; The second relative positional relationship between the calibration body and the IMU device is determined based on the error.

4. The method according to claim 3, characterized in that, The method further includes: The intrinsic parameters of the IMU device are determined based on the error and the IMU data.

5. An image processing device, characterized in that, The image processing device includes: An acquisition unit is used to acquire multiple images, each image including a calibration body and an electronic device with a light spot. The calibration body is rigidly connected to the electronic device, and the electronic device contains an inertial measurement unit (IMU). The electronic device is positioned differently in at least two of the multiple images. A determining unit is configured to determine a first relative positional relationship between the calibration body and the light spot based on the plurality of images; The acquisition unit is further configured to acquire IMU data collected by the IMU device, the IMU data being related to the plurality of images; The acquisition unit is further configured to acquire a second relative positional relationship between the calibration body and the IMU device based on the plurality of images and the IMU data; The determining unit is further configured to determine a third relative positional relationship between the light spot and the IMU coordinate system based on the first relative positional relationship and the second relative positional relationship, the third relative positional relationship being used to locate the electronic device.

6. The image processing apparatus according to claim 5, characterized in that, The determining unit is specifically used to obtain the first coordinates of the calibration body in the image coordinate system based on the plurality of images; The determining unit is specifically used to obtain the second coordinates of the light point in the image coordinate system based on the plurality of images; The determining unit is specifically used to determine the first relative positional relationship between the calibration body and the light spot based on the first coordinate and the second coordinate.

7. The image processing apparatus according to claim 5 or 6, characterized in that, The acquisition unit is specifically used to determine the error between the first motion trajectory of the calibration body and the second motion trajectory of the IMU device based on the plurality of images and the IMU data; The acquisition unit is specifically used to determine the second relative positional relationship between the calibration body and the IMU device based on the error.

8. The image processing apparatus according to claim 7, characterized in that, The determining unit is further configured to determine the internal parameters of the IMU device based on the error and the IMU data.

9. A calibration system, characterized in that, The calibration system includes: a camera, an electronic device with a light spot, a calibration body, and an image processing device; wherein, the electronic device is rigidly connected to the calibration body, and the electronic device contains an inertial measurement unit (IMU). The camera is used to capture multiple images of the calibration object and the electronic device during their movement. The calibration body is used to determine the third relative positional relationship between the light spot and the IMU device, and the third relative positional relationship is used to locate the electronic device; The image processing device is used to acquire the plurality of images and the IMU data collected by the IMU device during the motion; and to determine the third relative positional relationship based on the plurality of images and the IMU data; Specifically, the image processing device is used to determine a first relative positional relationship between the calibration body and the light spot based on the plurality of images; The image processing device is specifically used to obtain a second relative positional relationship between the calibration body and the IMU device based on the plurality of images and the IMU data; The image processing device is specifically used to determine the third relative position relationship based on the first relative position relationship and the second relative position relationship.

10. The calibration system according to claim 9, characterized in that, The image processing device is specifically used to determine the error between the first motion trajectory of the calibration body and the second motion trajectory of the IMU device based on the multiple images and the IMU data; The image processing device is specifically used to determine the second relative positional relationship between the calibration body and the IMU device based on the error.

11. The calibration system according to claim 10, characterized in that, The image processing device is also used to determine the intrinsic parameters of the IMU device based on the error and the IMU data.

12. An image processing device, characterized in that, include: A processor coupled to a memory for storing programs or instructions that, when executed by the processor, cause the image processing device to perform the method as described in any one of claims 1 to 4.

13. A computer storage medium, characterized in that, Includes computer instructions that, when executed on an image processing device, cause the image processing device to perform the method as described in any one of claims 1 to 4.

14. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 4.

Images