A multi-camera field of view fusion method, electronic device and storage medium

Abstract

Embodiments of the present invention relate to a multi-camera field of view fusion method, an electronic device, and a storage medium. In the embodiments of the present invention, by using a main camera and multiple side cameras to photograph the same calibration board to obtain a main camera image and multiple side camera images, the positioning points of the main camera image and the multiple side camera images are identified, the true resolutions of the main camera and the multiple side cameras are calculated, the first relative position and multiple second relative positions are calculated and obtained, and based on the first relative position and the multiple second relative positions, the mapping relationship between the main camera FOV and the multiple side camera FOVs is calculated. Based on the mapping relationship, the field of view fusion of the main camera and the multiple side cameras is performed. In this way, the relative positions of the main camera and each side camera can be determined, the field of view fusion of the main camera and the side cameras can be realized, so as to clearly identify the detection objects outside the field of view of the main camera, increase the detection range, improve the detection efficiency, and enhance the user experience.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of optical detection, and in particular, to a multi-camera field of view fusion method, an electronic device, and a storage medium. Background Art

[0002] Most of the optical imaging modules of current AOI devices (optical detection devices) use a configuration of a single top camera to inspect the object to be measured on the bottom track. However, there are some scenarios where only the top camera cannot meet the detection requirements. The field of view of a single top camera is limited, and the size of the captured FOV is limited. When the area of the detection object is large, a side camera needs to be added to take pictures from the side to obtain information that cannot be obtained by the top main camera.

[0003] In order to ensure that the optical imaging module of the AOI device can cover as much detection area as possible, R & D designers often set the main camera and the side cameras at different heights to save space in the horizontal direction. Also, because the side cameras cannot block the field of view of the main camera, the side cameras need to be arranged obliquely outside the optical path of the main camera. These two points result in no connection between the coordinate system of the main camera's field of view and the coordinate systems in the fields of view of each side camera. This drawback makes it impossible to link the defect positions detected in the main camera with the defect positions detected in the side cameras during the later detection process. For example: When a user wants to detect a component with a soldering joint void or other soldering defects in the field of view of the main camera, the user detects multiple components in the side camera at this time, but the user cannot tell which component is the one seen by the user in the field of view of the main camera. Since the field of view of the main camera is not linked to the fields of view of each side camera, the relative positions of the main camera and each side camera cannot be determined, resulting in detection deviation and poor user experience. Summary of the Invention

[0004] In view of this, an object of the embodiments of the present invention is to provide a multi-camera field of view fusion method, an electronic device, and a storage medium, aiming to solve the technical problem in the prior art that component detection is deviated and the user experience is poor due to the inability to fuse the fields of view of the main camera and the side cameras.

[0005] To solve the above technical problem, the embodiments of the present invention provide the following technical solutions:

[0006] In a first aspect, the embodiments of the present invention provide a multi-camera field of view fusion method, including:

[0007] Taking pictures by a main camera and multiple side cameras based on the same calibration board to obtain a main camera image and multiple side camera images;

[0008] Processing of the main camera image:

[0009] Preprocess the main camera image, and after preprocessing, identify the valid dots and dot information of the valid dots in the main camera image, and calculate the true resolution of the main camera;

[0010] Based on the valid dots in the main camera image, identify the positioning points of the main camera image;

[0011] Calculate and obtain the first relative position according to the true resolution of the main camera. The first relative position is the relative position between the positioning point of the main camera image and the center of the main camera FOV;

[0012] Side camera image processing:

[0013] Set the processing order of the side camera image according to requirements, and process each side camera image according to the above main camera image processing steps based on the processing order, obtain the positioning points of the side camera image and obtain the second relative position. The second relative position is the relative position between the positioning point of the side camera image and the center of the side camera FOV;

[0014] Main and side camera field of view fusion:

[0015] Based on the first relative position and multiple different second relative positions, calculate and obtain the mapping relationship between the main camera FOV and multiple different side camera FOVs;

[0016] Perform field of view fusion of the main camera and the side camera based on the mapping relationship.

[0017] In some embodiments, the preprocessing of the main camera image includes:

[0018] Convert the input main camera image into a single-channel grayscale image;

[0019] Perform threshold segmentation on the converted main camera image to convert the main camera image into two colors, black and white.

[0020] In some embodiments, the identifying the valid dots and dot information of the valid dots in the main camera image after preprocessing includes:

[0021] After preprocessing, perform connected component analysis on the main camera image, find the median of the areas of all connected components, and set the area difference threshold according to the requirement of identifying dots;

[0022] Traverse all connected components, calculate the four corner coordinates of the circumscribed rectangle of each connected component, and judge all connected components based on the area difference threshold and the four corner coordinates to identify all valid dots and dot information of the valid dots in the main camera image.

[0023] In some embodiments, the dot information includes the actual diameter of the valid dot, the width and height of the circumscribed rectangle, and the calculating the true resolution of the main camera includes:

[0024] Sum the widths of the circumscribed rectangles of all valid dots to obtain the sum of the circumscribed widths;

[0025] Obtain the dot diameter in the X direction based on the sum of the circumscribed widths and the total number of valid dots, and calculate and obtain the X-axis resolution of the main camera according to the dot diameter in the X direction and the actual diameter;

[0026] Sum the heights of the circumscribed rectangles of all valid dots to obtain the sum of the circumscribed heights;

[0027] Obtain the dot diameter in the Y direction based on the sum of the circumscribed heights and the total number of valid dots, and calculate and obtain the Y-axis resolution of the main camera according to the dot diameter in the Y direction and the actual diameter. The true resolution of the main camera includes the X-axis resolution and the Y-axis resolution of the main camera.

[0028] In some embodiments, the method for identifying the positioning point of the main camera image based on the valid dots in the main camera image includes:

[0029] Set the foreground of the preprocessed main camera image to white, and perform connected component analysis on the main camera image again to obtain the upper left coordinates, width, and height of the circumscribed rectangle of the connected component;

[0030] Calculate the upper right coordinates, lower right coordinates, and lower left coordinates of the circumscribed rectangle of the connected component through the upper left coordinates, width, and height of the circumscribed rectangle of the first connected component;

[0031] Determine whether any one of the upper left coordinates, lower left coordinates, upper right coordinates, and lower right coordinates of the circumscribed rectangle of the connected component touches the edge of the main camera image;

[0032] When the judgment result is yes, it means that the second connected component is the positioning point;

[0033] When the judgment result is no, it means that the first connected component is the positioning point.

[0034] In some embodiments, the method for calculating and obtaining the first relative position according to the true resolution of the main camera includes:

[0035] Obtain the center coordinates of the main camera FOV according to the width and height of the main camera image;

[0036] Obtain the X-axis relative distance of the positioning point relative to the FOV center in pixel units according to the X-axis coordinate of the positioning point and the X-axis coordinate of the main camera FOV center;

[0037] Obtain the Y-axis relative distance of the positioning point relative to the FOV center in pixel units according to the Y-axis coordinate of the positioning point and the Y-axis coordinate of the main camera FOV center;

[0038] Obtain the relative X - axis distance in physical world units of the positioning point relative to the FOV center based on the relative X - axis distance in pixel units and the X - axis resolution of the main camera;

[0039] Obtain the relative Y - axis distance in physical world units of the positioning point relative to the FOV center based on the relative Y - axis distance in pixel units and the Y - axis resolution of the main camera;

[0040] Obtain the first relative position based on the relative X - axis distance in physical world units and the relative Y - axis distance in physical world units.

[0041] In some embodiments, calculating and obtaining the mapping relationship between the FOV of the main camera and the FOVs of multiple different side cameras based on the first relative position and multiple different second relative positions includes:

[0042] Compare the first relative position with the second relative position of the candidate side camera to obtain the relative orientation between the main camera and the candidate side camera, where the candidate side camera is any one of the multiple side cameras;

[0043] Calculate the relative X - axis distance and relative Y - axis distance in physical world units between the main camera and the candidate side camera according to the relative orientation to establish the mapping relationship between the FOV of the main camera and the FOV of the candidate side camera;

[0044] Establish the mapping relationship between the FOV of the main camera and the FOV of each side camera in sequence according to the processing order.

[0045] In some embodiments, before performing the field - of - view fusion between the main camera and the side cameras based on the mapping relationship, the method further includes:

[0046] Compare the widths and heights of the images of multiple side cameras to find the minimum width and minimum height among all the side - camera images;

[0047] Crop the width of each side - camera image to the minimum width and the height to the minimum height.

[0048] In a second aspect, an embodiment of the present invention provides an electronic device, including:

[0049] A processor and a memory communicatively connected to the processor;

[0050] The memory stores computer program instructions executable by the processor, and the processor executes the computer program instructions to enable the electronic device to execute any one of the multi - camera field - of - view fusion methods proposed in the first aspect.

[0051] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer program instructions executable by a processor. When the processor executes the computer program instructions, the computer is caused to execute any one of the multi-camera field-of-view fusion methods proposed in the first aspect.

[0052] The embodiments of the present invention have the following beneficial effects: Different from the prior art, the multi-camera field-of-view fusion method provided by the embodiments of the present invention obtains a main camera image and multiple side camera images by using a main camera and multiple side cameras to photograph the same calibration board, preprocesses the main camera image and the multiple side camera images, and identifies valid dots and dot information of the valid dots in the preprocessed main camera image and multiple side camera images. Based on the valid dots of the main camera image and the multiple side camera images, the positioning points in the main camera image and the multiple side camera images are identified, the true resolutions of the main camera and the multiple side cameras are calculated, the first relative position and multiple second relative positions are calculated based on the calculated true resolutions of the main camera and the multiple side cameras, the mapping relationship between the main camera FOV and the multiple side camera FOVs is calculated according to the first relative position and the multiple second relative positions, and the field-of-view fusion of the main camera and the multiple side cameras is performed based on the mapping relationship. In this way, the relative positions of the main camera and each side camera can be determined, the field-of-view fusion of the main camera and the side cameras can be realized, so that the detection objects outside the field of view of the main camera can be clearly identified, the detection range is increased, the detection efficiency is improved, and the user experience is enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the prior art or embodiments. Obviously, the following-described drawings only show some embodiments of the present invention and should not be regarded as limiting the protection scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0054] Figure 1 is a schematic diagram of an application scenario of the multi-camera field-of-view fusion method in some embodiments of the present invention;

[0055] Figure 2a is a schematic diagram of the positional relationship between the main camera and multiple side cameras in some embodiments of the present invention;

[0056] Figure 2b is a schematic diagram of the relative orientation between the main camera and multiple side cameras in some embodiments of the present invention;

[0057] Figure 3 is a schematic diagram of the positional relationship between the positioning points and calibration points of the calibration pattern array in the calibration board provided in some embodiments of the present invention;

[0058] Figure 4a is the main camera image obtained by the main camera shooting the calibration board in some embodiments of the present invention;

[0059] Figure 4b is the side camera image obtained by the first side camera shooting the calibration board in some embodiments of the present invention, wherein the camera field of view of the side camera image is not corrected;

[0060] Figure 4c is the side camera image obtained by the second side camera shooting the calibration board in some embodiments of the present invention, wherein the camera field of view of the side camera image is not corrected;

[0061] Figure 4d is the side camera image obtained by the third side camera shooting the calibration board in some embodiments of the present invention, wherein the camera field of view of the side camera image is not corrected;

[0062] Figure 4e is the side camera image obtained by the fourth side camera shooting the calibration board in some embodiments of the present invention, wherein the camera field of view of the side camera image is not corrected;

[0063] Figure 5 is the structural schematic diagram of the electronic device provided in some embodiments of the present invention;

[0064] Figure 6 is the flow schematic diagram of the multi-camera field of view fusion method provided in some embodiments of the present invention;

[0065] Figure 7a is Figure 4a the pre-processed main camera image obtained by pre-processing the main camera image shown;

[0066] Figure 7b is Figure 4a the pre-processed and color-inverted main camera image obtained by pre-processing and color-inverting the main camera image shown;

[0067] Figure 8a is Figure 4b the side camera image obtained after performing camera field of view on the side camera image shown;

[0068] Figure 8b is Figure 4c the side camera image obtained after performing camera field of view on the side camera image shown;

[0069] Figure 8c is Figure 4d the side camera image obtained after performing camera field of view on the side camera image shown;

[0070] Figure 8d is Figure 4eThe side camera image obtained after performing the camera field of view on the shown side camera image;

[0071] Figure 9a is Figure 4a the shown main camera image and Figures 8a to 8d a schematic diagram before fusing the shown side camera image;

[0072] Figure 9b is Figure 4a the shown main camera image and Figures 8a to 8d a schematic diagram after fusing the shown side camera image. Detailed implementation manners

[0073] To make the objectives and advantages of the embodiments of the present invention easier to understand, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. The following detailed description of the embodiments of the present invention in the drawings does not limit the scope required to be protected by the present invention, but only represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0074] It should be noted that if there is no conflict, the various technical features involved in the embodiments of the present invention described below can be combined with each other and are all within the protection scope of the present invention. In addition, although functional module division is performed in the device or structure schematic diagram and the logical sequence is shown in the flowchart, in some cases, the steps shown or described can be executed in a different module division in the device or a different order from that in the flowchart. In addition, the "first", "second", "third", and other similar expressions used herein do not limit the data and execution order, but are only for the purpose of easy explanation and to distinguish the same items or similar items with basically the same functions and effects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features.

[0075] Unless otherwise defined, the technical terms and scientific terms used in this specification have the same meanings as those generally understood by those of ordinary skill in the technical field to which the present invention belongs. The terms used in this specification are only for the purpose of describing specific implementation manners and are not used to limit the present invention. It should be understood that the term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0076] In the semiconductor manufacturing industry, it is usually necessary to use Automated Optical Inspection (AOI) equipment to detect and analyze problems such as the appearance, quality, and performance of products. The optical machine and camera are the optical components that make up the AOI equipment. Most of the current optical imaging modules of AOI equipment use a configuration of a single top camera to inspect the objects to be tested on the bottom track. However, there are some scenarios where only relying on the top camera cannot meet the detection requirements. The field of view of a single top camera is limited, and the size of the captured FOV is limited. When the area of the detection object is large, it is necessary to add a side camera to take pictures from the side to obtain information that cannot be obtained by the top main camera.

[0077] In order to ensure that the optical imaging module of the AOI equipment can cover as much detection area as possible, R & D designers often set the main camera and the side cameras at different heights to save space in the horizontal direction. Also, because the side cameras cannot block the field of view of the main camera, the side cameras need to be arranged obliquely outside the optical path of the main camera. These two points result in the coordinate systems in the field of view of the main camera having no connection with the coordinate systems in the fields of view of the respective side cameras. This shortcoming makes it impossible to connect the defect positions detected in the main camera with the defect positions detected in the side cameras during the later detection process. For example: When a user wants to detect a component with a soldering defect such as a cold solder joint or other soldering defects in the field of view of the main camera, at this time, the user detects multiple components in the side camera, but the user cannot tell which component is the one seen by the user in the field of view of the main camera. Since the field of view of the main camera is not connected to the fields of view of the respective side cameras, the relative positions of the main camera and the respective side cameras cannot be determined, resulting in detection deviations and a poor user experience.

[0078] In view of this, the embodiments of the present invention provide a method for fusing the fields of view of multiple cameras. By using the main camera and multiple side cameras to photograph the same calibration board to obtain the main camera image and the images of multiple side cameras, preprocessing the main camera image and the images of multiple side cameras, and identifying the effective dots and the dot information of the effective dots in the preprocessed main camera image and the images of multiple side cameras, based on the effective dots in the main camera image and the images of multiple side cameras, identifying the positioning points in the main camera image and the images of multiple side cameras, calculating the true resolutions of the main camera and multiple side cameras, calculating and obtaining the first relative position and multiple second relative positions according to the calculated true resolutions of the main camera and multiple side cameras, calculating the mapping relationship between the main camera FOV and the FOVs of multiple side cameras according to the first relative position and multiple second relative positions, and performing the field of view fusion of the main camera and multiple side cameras based on the mapping relationship. In this way, the relative positions of the main camera and the respective side cameras can be determined, the field of view fusion of the main camera and the side cameras can be realized, so as to clearly identify the detection objects outside the field of view of the main camera, increase the detection range, improve the detection efficiency, and enhance the user experience.

[0079] Please refer to Figure 1 , Figure 1 which schematically shows an application scenario diagram of the multi-camera field of view fusion method provided by some embodiments of the present invention.

[0080] As Figure 1 shown, the application scenario includes an electronic device 100. It can be understood that, to achieve multi-camera field of view fusion, the electronic device 100 also needs to cooperate with some other software and hardware, such as cameras, image processing software, display devices, and storage devices, etc. Those skilled in the art can add, delete, or change the required software and hardware according to actual needs to cooperate with the electronic device 100 to complete multi-camera field of view fusion. It should be understood that these software and hardware that cooperate can be configured in the electronic device 100 according to actual needs, or exist independently of the electronic device 100.

[0081] It should be understood that Figure 1 in the application scenario shown, the electronic device 100 is a desktop computer, but it does not impose any limitations on the structure, type, quantity, etc. of the electronic devices in other application scenarios. For example, in some other embodiments, the electronic device can also be a laptop computer, a tablet computer, or other suitable types of devices, or it can also be a server, such as a server deployed in the cloud.

[0082] For example, please refer to Figure 2a , Figure 2a which shows the positional relationship between the main camera and the other four side cameras in some embodiments of the present invention. Figure 2a As shown, it includes a calibration board 200, a main camera 300, and four side cameras. Among them, the four side cameras are respectively the first side camera 301, the second side camera 302, the third side camera 303, and the fourth side camera 304. The first side camera 301, the second side camera 302, the third side camera 303, and the fourth side camera 304 are arranged on the lower side of the main camera 300 and are outside the optical path of the main camera 300 to avoid affecting the main camera 300 from photographing the calibration board 200, the component to be detected, or other objects.

[0083] Please refer to Figure 3 , in some embodiments, the calibration board 200 includes a calibration pattern array 210. Among them, the calibration pattern array 210 includes positioning points 211 and calibration points 212. The positioning points 211 are arranged at the center of the calibration pattern array 210 to facilitate subsequent determination of the relative positions of the field of view center of the main camera and the field of view centers of the respective side cameras, and then perform field of view fusion of the main camera and the respective side cameras. It should be understood that Figure 3Only schematically shows the positions of the positioning points 211 and calibration points 212 of the calibration pattern array 210 in the calibration board 200, but it does not impose any limitations on the positions, shapes, arrangements, quantities, etc. of the positioning points and calibration points in other embodiments, as long as the positioning point 211 can be captured by both the main camera and the side cameras.

[0084] When performing multi-camera field-of-view fusion, the main camera and multiple side cameras are used to take pictures based on the same calibration board to obtain the main camera image and multiple side camera images. For example, Figure 2a the calibration board 200 can be photographed by the shown main camera 300, the first side camera 301, the second side camera 302, the third side camera 303, and the fourth side camera 304 to obtain the main camera image and the four side camera images. Please refer to Figures 4a to 4e , Figures 4a to 4e which respectively show the main camera image and the four side camera images obtained by photographing with the main camera 300, the first side camera 301, the second side camera 302, the third side camera 303, and the fourth side camera 304. Among them, Figure 4a the shown main camera image is marked with the center of the camera field of view, and the center of the camera field of view is like the Figure 4a shown gray line.

[0085] It can be understood that there is no distortion in the camera field of view of the main camera image because the main camera is vertically set at the target position for placing the component to be detected, such as the position of the calibration board 200 shown in Figure 2a . However, there will be a depth-of-field problem in the camera fields of view of the four side camera images because the four side cameras are not vertically set at the target position for placing the component to be detected. Therefore, a Scheimpflug lens is installed on the four side cameras. The Scheimpflug lens is installed on the side cameras because it can change the optical axis direction of the lens, making the optical axis of the obliquely arranged side cameras perpendicular to the object surface, so that the entire side image obtained by shooting is in focus and there will be no defocus, solving the depth-of-field problem caused by the tilt of the camera and ensuring that the detection effect is not affected.

[0086] Specifically, as a device providing computing and control capabilities, after the electronic device 100 captures a main camera image and multiple side camera images using the main camera and the multiple side cameras, it preprocesses the main camera image and the multiple side camera images, identifies valid dots in the preprocessed main camera image and the multiple side camera images, then identifies the positioning points in the main camera image and the multiple side camera images based on the valid dots of the main camera image and the multiple side camera images, calculates the true resolutions of the main camera and the multiple side cameras, calculates and obtains a first relative position and multiple second relative positions according to the true resolutions of the main camera and the multiple side cameras. The first relative position is the relative position between the positioning point of the main camera image and the center of the main camera FOV, and the second relative position is the relative position between the positioning point of the side camera image and the center of the side camera FOV. Finally, based on the first relative position and the multiple different second relative positions, it calculates and obtains the mapping relationship between the main camera FOV and the multiple different side camera FOVs, and performs field of view fusion of the main camera and the multiple side cameras based on the mapping relationship. In this way, the relative positions of the main camera and each side camera can be determined, the field of view fusion of the main camera and the side cameras can be achieved, so as to clearly identify the detection objects outside the field of view of the main camera, increase the detection range, improve the detection efficiency, and enhance the user experience.

[0087] To facilitate the understanding of the multi-camera field of view fusion method provided by the embodiments of the present invention, the electronic device provided by the embodiments of the present invention will be introduced in detail first.

[0088] Please refer to Figure 5 , Figure 5 which schematically shows the structural diagram of the electronic device provided by some embodiments of the present invention.

[0089] As Figure 5 shown, the electronic device 100 includes at least one processor 110 and a memory 120 that are communicatively connected, Figure 5 taking one processor connected by a bus system 130 as an example. Among them, the various components in the electronic device 100 are coupled together through the bus system 130, and the bus system 130 is used to realize the connection and communication between the various components. It is easy to understand that in addition to the data bus, the bus system 130 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clear description and concise space, in Figure 5 all kinds of buses are labeled as the bus system 130. It can be understood that Figure 5 the structure shown in the embodiment is only schematic, and it does not impose any limitation on the structure of the above-mentioned electronic device. For example, the above-mentioned electronic device may also include more or fewer components than the Figure 5 shown structure, or have a configuration different from the Figure 5 shown structure.

[0090] Specifically, the processor 110 is configured to provide computing and control capabilities to control the electronic device 100 to perform corresponding tasks. For example, it controls the above-mentioned electronic device 100 to execute any one of the multi-camera field-of-view fusion methods provided by the embodiments of the present invention, or to execute the steps in any possible implementation manner of any one of the multi-camera field-of-view fusion methods provided by the embodiments of the present invention. Those skilled in the art can understand that the processor 110 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0091] The memory 120, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, instructions, and modules. For example, the programs, instructions, and modules corresponding to the multi-camera field-of-view fusion method in the embodiments of the present invention. In some embodiments, the memory 120 may include a program storage area and a data storage area. The program storage area can store an operating system and application programs required for at least one function, and the data storage area can store data created according to the use of the processor 110. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the non-transitory software programs, instructions, and modules stored in the memory 120, so as to implement any one of the multi-camera field-of-view fusion methods provided by the embodiments of the present invention, or to execute the steps in any possible implementation manner of any one of the multi-camera field-of-view fusion methods provided by the embodiments of the present invention. In some embodiments, the memory 120 may include a high-speed random access memory and may also include non-transitory memory. For example, at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 120 may further include a memory remotely disposed relative to the processor 110, and these remotely disposed memories may be connected to the processor 110 through a communication network. It can be understood that examples of the above communication network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0092] As can be understood from the above, the execution subject of any of the multi-camera field-of-view fusion methods provided in the embodiments of the present invention can be any suitable type of electronic device with certain computing and control capabilities. For example, it can be implemented by the above-mentioned electronic device 100. In some feasible implementation manners, the multi-camera field-of-view fusion method provided in the embodiments of the present invention can be implemented by a processor executing computer program instructions stored in a memory.

[0093] Next, the multi-camera field-of-view fusion method provided in the embodiments of the present invention will be described in detail in combination with the exemplary applications and implementations of the electronic device provided in the embodiments of the present invention.

[0094] Please refer to Figure 6 , Figure 6 which schematically shows the flowchart of the multi-camera field-of-view fusion method provided in some embodiments of the present invention.

[0095] Those skilled in the art can understand that the multi-camera field-of-view fusion method provided in the embodiments of the present invention can be applied to the above-mentioned electronic device (for example, electronic device 100). Specifically, the execution subject of the multi-camera field-of-view fusion method is one or at least two processors of the electronic device.

[0096] As Figure 6 shown, the multi-camera field-of-view fusion method includes but is not limited to the following steps S100-S700:

[0097] S100: Shoot through the main camera and multiple side cameras based on the same calibration board to obtain the main camera image and multiple side camera images.

[0098] In the embodiments of the present invention, a positioning block and multiple calibration blocks are arranged on the calibration board. The positioning block and the calibration blocks form a calibration pattern array, and the positioning block is located at the center of the calibration pattern array. The shape of the calibration block can be circular, rectangular, square, etc., and the shape of the positioning block can be annular, diamond-shaped, etc., and the shapes of the positioning block and the calibration block are different. In the embodiments of the present invention, the shape of the calibration block is set to be circular, and the shape of the positioning block is set to be annular, and the outer diameter of the positioning block is the same as the diameter of the calibration block. It is easy to understand that the distribution shape of the positioning block and the calibration blocks on the calibration board (that is, the distribution shape of the calibration pattern array) can be a regular shape such as a rectangle, a circle, or a square, or an irregular shape such as a diamond, an ellipse, or a triangle. In the embodiments of the present invention, the distribution shape of the positioning block and the calibration blocks is set to be a square shape, which is convenient for subsequent calculations.

[0099] Exemplarily, fix the calibration board at the target position of the stable support to ensure that the calibration board does not shake or deform during the shooting process. The size of the calibration board is appropriate and it needs to be within the field of view of the main camera and multiple side cameras at the same time. Set a reasonable light intensity to avoid reflection and shadow, and improve the image clarity and the recognition accuracy of the positioning points and calibration points.

[0100] Specifically, fixedly install the main camera and multiple side cameras at predetermined positions to ensure that each camera does not move during shooting. Send a shooting instruction to the main camera and multiple side cameras. After receiving the shooting instruction, the main camera and multiple side cameras shoot the calibration board set at the target position to obtain the main camera image and multiple side camera images, thereby realizing that both the main camera and multiple side cameras shoot based on the same calibration board to obtain the main camera image and multiple side camera images. The main camera image and multiple side camera images include circular or approximately circular shadow areas (i.e., calibration points) formed by calibration blocks and annular or approximately annular shadow areas (i.e., positioning points) formed by positioning blocks.

[0101] S200: Preprocess the main camera image, and after preprocessing, identify the valid dot points and dot point information in the main camera image, and calculate the true resolution of the main camera.

[0102] In some embodiments, preprocess the main camera image, that is, remove the color information from the main camera image and only retain the brightness information, so as to convert the main camera image into a grayscale image with only one grayscale value for each pixel. Then use Gaussian filtering to smooth the grayscale image and remove the noise points in the grayscale image. Finally, perform binary processing on the grayscale image after removing the noise points to convert the grayscale image after removing the noise points into a pure black-and-white image (i.e., a binary image including only black and white colors), and obtain the preprocessed main camera image. Please refer to Figure 7a , Figure 7a which schematically shows the preprocessed main camera image obtained by preprocessing the Figure 4a shown main camera image.

[0103] In some embodiments, use a circular detection algorithm (such as the Hough transform algorithm) to detect the dot points in the preprocessed main camera image and extract the identified detected dot points. Among them, the Hough transform algorithm is an image processing algorithm used to identify dot points in an image and return the center coordinates and radius of the dot points. In some embodiments, any other suitable algorithm can also be used to detect the dot points in the preprocessed main camera image, and the embodiments of the present invention do not make any limitations in this regard.

[0104] In some embodiments, for each dot extracted, according to a preset rule, it is determined whether the dot is valid. If it is valid, the dot is determined to be a valid dot and screened out as a valid dot. If it is not valid, the dot is determined to be an invalid dot and removed or otherwise processed. The preset rules include that the radius size of the dot is within a preset range, the center position of the dot conforms to the layout of the calibration pattern array, etc. Of course, those skilled in the art can add, modify or delete the preset rules according to actual needs, and the embodiments of the present invention do not make any limitations thereto.

[0105] In some embodiments, after screening out valid dots from the extracted dots, dot information of the valid dots is obtained by using a circular detection algorithm (such as the Hough transform algorithm, etc.). The dot information includes: the centroid and radius of the valid dot. The centroid of the valid dot is the center coordinate of the valid dot. The Hough transform algorithm gives the center coordinate and radius of each valid dot, so that the dot information of the valid dots can be obtained through the Hough transform algorithm.

[0106] In some embodiments, according to the dot information of the valid dots, the true resolution of the main camera is calculated. The true resolution of the main camera includes the X-axis resolution and Y-axis resolution of the main camera. In the embodiments of the present invention, the dot information of the valid dots includes the actual diameter of the valid dot. The circular calibration block on the calibration plate has a known actual size in the actual physical space. The actual size includes the actual diameter of the circular calibration block, and the actual diameter of the circular calibration block is the actual diameter of the valid dot.

[0107] Exemplarily, the circumscribed rectangles of all valid dots are obtained, the widths (in pixels) of the circumscribed rectangles of all valid dots are summed to obtain the total width of the circumscribed rectangles. The total number of all valid dots is calculated. Based on the total width of the circumscribed rectangles and the total number of valid dots, the dot diameter in the X direction is obtained, and the X-axis resolution of the main camera is calculated according to the dot diameter in the X direction and the actual diameter of the valid dot. The heights (in pixels) of the circumscribed rectangles of all valid dots are summed to obtain the total height of the circumscribed rectangles. Based on the total height of the circumscribed rectangles and the total number of valid dots, the dot diameter in the Y direction is obtained, and the Y-axis resolution of the main camera is calculated according to the dot diameter in the Y direction and the actual diameter of the valid dot, so as to obtain the true resolution of the main camera.

[0108] Exemplarily, in some embodiments, preprocessing is performed on the main camera image, which specifically includes but is not limited to the following steps S210 - S220:

[0109] S210: Convert the input main camera image into a single-channel grayscale image.

[0110] Specifically, preprocess the main camera image input to the electronic device, that is, use an image processing library (such as OpenCV, etc.) to remove the color information from the main camera image, convert the number of channels of the main camera image from three channels (such as RGB channels) to a single channel, and obtain a single-channel grayscale image. Each pixel in the single-channel grayscale image has only one grayscale value.

[0111] S220: Perform threshold segmentation on the converted main camera image to convert the main camera image into two colors, black and white.

[0112] In the embodiment of the present invention, a threshold segmentation algorithm is used to perform threshold segmentation on the converted main camera image (i.e., the single-channel grayscale image), so that the main camera image is converted into two colors, black and white, that is, perform binary processing on the converted main camera image, convert the single-channel grayscale image into a pure black and white image (i.e., a binary image including only black and white), and obtain the preprocessed main camera image. The preprocessed main camera image is in the form of a binary image.

[0113] It can be understood that the threshold segmentation algorithm includes but is not limited to: binary threshold segmentation algorithm, OTSU algorithm, and TOZERO algorithm. In some embodiments, Gaussian blur is used to remove the noise points in the binary image, smooth the binary image, make the features in the binary image more obvious, so as to obtain a preprocessed main camera image with higher quality and reduce the interference to feature extraction.

[0114] In some embodiments, after preprocessing, identify the valid dots and the dot information of the valid dots in the main camera image, specifically including but not limited to the following steps S230 - S240:

[0115] S230: After preprocessing, perform connected component analysis on the main camera image, find the median of the areas of all connected components, and set the area difference threshold according to the requirement of identifying dots.

[0116] S240: Traverse all connected components, calculate the four corner coordinates of the circumscribed rectangle of each connected component, and judge all connected components based on the area difference threshold and the four corner coordinates to identify all valid dots and the dot information of the valid dots in the main camera image.

[0117] In the embodiment of the present invention, after preprocessing the main camera image, a connected component analysis algorithm is used to perform connected component analysis on the preprocessed main camera image, detect and identify all connected components in the preprocessed main camera image, and traverse all the identified connected components to obtain connected component information. Among them, the connected component information includes the total number of connected components, the centroid coordinates, area, the upper left corner coordinates of the circumscribed rectangle, width, and height of each connected component. It can be understood that the connected component analysis algorithm includes but is not limited to: Spaghetti algorithm, SAUF algorithm, and BBDT algorithm.

[0118] In some embodiments, the area of the connected component is the total number of pixels within the connected component. The total number of pixels of the connected component is calculated using the cv2.contourArea() function in the OpenCV library or other similar methods to obtain the area of the connected component. According to the areas of each connected component, the areas of all connected components are arranged in ascending order to obtain an area arrangement sequence. The area value located at the middle position is selected as the median of the areas from the area arrangement sequence. If there are two area values at the middle position, the median of the areas is taken as the average of the two area values at the middle position to obtain the median of the areas of all connected components.

[0119] In the embodiments of the present invention, an area difference threshold is set according to the requirements for identifying the circular dots (such as the number of circular dots, the size of the circular dots, and the median of the areas, etc.). The area difference threshold is used to identify and determine whether the identified connected component is qualified. In some embodiments, the area difference threshold can be set according to a preset multiple of the median of the areas of the connected components to distinguish between valid and invalid connected components. For example, the area difference threshold is 1.2 times, 1.5 times, or other appropriate multiples of the median of the areas. Those skilled in the art can set the area difference threshold according to actual needs, and the embodiments of the present invention do not make any limitations thereto.

[0120] In some embodiments, the centroid coordinates of the connected component are the geometric center of the connected component. The moments of the connected component can be calculated using the cv2.moments() function in the OpenCV library to obtain the centroid coordinates of the connected component.

[0121] In some embodiments, the bounding rectangle of the connected component is represented by the upper left corner coordinates and the width and height of the rectangle. The coordinates and dimensions (including the width and height) of the bounding rectangle of the connected component are obtained using the cv2.boundingRect() function of the OpenCV library to obtain the bounding rectangle of the connected component.

[0122] In the embodiments of the present invention, the four corner coordinates of the bounding rectangle include the upper left corner coordinate, the upper right corner coordinate, the lower left corner coordinate, and the lower right corner coordinate of the bounding rectangle.

[0123] Specifically, after traversing all the identified connected components to obtain the upper-left corner coordinates, width, and height of the bounding rectangle of each connected component, for each connected component, based on the upper-left corner coordinates, width, and height of the bounding rectangle of each connected component, calculate the upper-right corner coordinates, lower-left corner coordinates, and lower-right corner coordinates of the bounding rectangle of each connected component to obtain the four-corner coordinates of the bounding rectangle of each connected component. It can be understood that the ordinate of the upper-right corner coordinates of the bounding rectangle is the same as that of the upper-left corner coordinates. Therefore, when calculating the upper-right corner coordinates, add the abscissa of the upper-left corner coordinates to the width of the bounding rectangle to obtain the abscissa of the upper-right corner coordinates, thereby obtaining the upper-right corner coordinates. Similarly, the abscissa of the lower-left corner coordinates of the bounding rectangle is the same as that of the upper-left corner coordinates. Therefore, when calculating the lower-left corner coordinates, subtract the height of the bounding rectangle from the ordinate of the upper-left corner coordinates to obtain the ordinate of the lower-left corner coordinates, thereby obtaining the lower-left corner coordinates. Similarly, after calculating the upper-right corner coordinates and / or the lower-left corner coordinates, the abscissa of the lower-right corner coordinates of the bounding rectangle is the same as that of the upper-right corner coordinates. Therefore, when calculating the lower-right corner coordinates, subtract the height of the bounding rectangle from the ordinate of the upper-right corner coordinates to obtain the ordinate of the lower-right corner coordinates, thereby obtaining the lower-right corner coordinates. Or the ordinate of the lower-right corner coordinates of the bounding rectangle is the same as that of the lower-left corner coordinates. Therefore, when calculating the lower-right corner coordinates, add the width of the bounding rectangle to the abscissa of the lower-left corner coordinates to obtain the abscissa of the lower-right corner coordinates, thereby obtaining the lower-right corner coordinates.

[0124] Compare the upper-left corner coordinates, upper-right corner coordinates, lower-left corner coordinates, and lower-right corner coordinates of the bounding rectangle of each connected component with the edges of the main camera image respectively. If any of the upper-left corner coordinates, upper-right corner coordinates, lower-left corner coordinates, and lower-right corner coordinates of the bounding rectangle exceeds the edge of the main camera image or lies on the edge of the main camera image, then this connected component is an incomplete connected component, and the incomplete connected component is an invalid dot. Filter out this incomplete connected component.

[0125] After filtering out the incomplete connected components, filter the remaining complete connected components according to the area difference threshold to obtain qualified connected components. Among them, the qualified connected components are valid dots, and the connected component information of the qualified connected components is the dot information of the valid dots.

[0126] Specifically, for each complete connected component, subtract the area of each complete connected component from the median of the areas to obtain an area difference. Compare the area difference with the area difference threshold. If the area difference is less than or equal to the area difference threshold, then the connected component is a qualified connected component (i.e., a valid dot). If the area difference is greater than the area difference threshold, then the connected component is an unqualified connected component (i.e., an invalid dot). Use the connected component information of the qualified connected components obtained by filtering as the dot information of the valid dots to obtain the dot information of the valid dots, that is, identify all the valid dots and the dot information of the valid dots in the main camera image.

[0127] Exemplarily, in some embodiments, calculating the true resolution of the main camera specifically includes, but is not limited to, the following steps S250 - S280:

[0128] S250: Sum the widths of the circumscribed rectangles of all valid dots to obtain the sum of the circumscribed widths.

[0129] S260: Obtain the dot diameter in the X - direction based on the sum of the circumscribed widths and the total number of valid dots, and calculate the X - axis resolution of the main camera according to the dot diameter in the X - direction and the actual diameter.

[0130] S270: Sum the heights of the circumscribed rectangles of all valid dots to obtain the sum of the circumscribed heights.

[0131] S280: Obtain the dot diameter in the Y - direction based on the sum of the circumscribed heights and the total number of valid dots, and calculate the Y - axis resolution of the main camera according to the dot diameter in the Y - direction and the actual diameter.

[0132] In the embodiments of the present invention, the dot information includes the actual diameter of the valid dots, the width and height of the circumscribed rectangle, and the true resolution of the main camera includes the X - axis resolution and the Y - axis resolution of the main camera.

[0133] In some embodiments, the true resolution of the main camera is calculated based on the dot information, that is, sum the widths (in pixels) of the circumscribed rectangles of all valid dots to obtain the total sum of the circumscribed rectangle widths, denoted as the sum of the circumscribed widths. Calculate the total number of valid dots, and divide the sum of the circumscribed widths by the total number of valid dots to obtain the dot diameter in the X - direction (i.e., the diameter of the valid dots in the X - axis direction, in pixels), and divide the dot diameter in the X - direction by the actual diameter of the valid dots to obtain the X - axis resolution of the main camera.

[0134] Sum the heights (in pixels) of the circumscribed rectangles of all valid dots to obtain the total sum of the circumscribed rectangle heights, denoted as the sum of the circumscribed heights. Divide the sum of the circumscribed heights by the total number of valid dots to obtain the dot diameter in the Y - direction (i.e., the diameter of the valid dots in the Y - axis direction, in pixels), and divide the dot diameter in the Y - direction by the actual diameter of the valid dots to obtain the Y - axis resolution of the main camera, thereby obtaining the true resolution of the main camera.

[0135] S300: Identify the positioning points of the main camera image based on the valid dots in the main camera image.

[0136] In the embodiments of the present invention, the main camera image includes positioning points and calibration points. After removing the unqualified positioning points and calibration points, qualified positioning points and calibration points (i.e., effective circular dots) are obtained. According to the description of the foregoing embodiments, the shapes of the positioning points and calibration points are different. The positioning points are annular, and the calibration points are circular. Therefore, in the preprocessed main camera image, the colors and shapes of the positioning points and calibration points are different. For example, please refer to Figure 7a , Figure 7a In the preprocessed main camera image shown, the positioning points are white rings, and the calibration points are black dots.

[0137] Specifically, after identifying the effective circular dots in the main camera image, according to the positions, shapes, and colors of each effective circular dot in the image coordinate system, the effective circular dots in the main camera image are identified, the positioning points of the main camera image are identified, and the X-axis coordinates and Y-axis coordinates of the positioning points of the main camera image are obtained.

[0138] Exemplarily, in some embodiments, identifying the positioning points of the main camera image based on the effective circular dots in the main camera image specifically includes, but is not limited to, the following steps S310-S350:

[0139] S310: Set the foreground of the preprocessed main camera image to white, and perform connected component analysis on the main camera image again to obtain the upper left corner coordinates of the circumscribed rectangle of the connected component, the width and height of the circumscribed rectangle.

[0140] S320: Calculate the upper right corner coordinates, lower right corner coordinates, and lower left corner coordinates of the circumscribed rectangle of the connected component through the upper left corner coordinates, width, and height of the circumscribed rectangle of the first connected component.

[0141] S330: Determine whether any one of the upper left corner coordinates, lower left corner coordinates, upper right corner coordinates, and lower right corner coordinates of the circumscribed rectangle of the connected component touches the edge of the main camera image.

[0142] S340: When the judgment result is yes, it means that the second connected component is the positioning point.

[0143] S350: When the judgment result is no, it means that the first connected component is the positioning point.

[0144] In the embodiments of the present invention, in the preprocessed main camera image, the background and the positioning points are white, and the calibration points are black dots. If the connected component analysis algorithm used considers the foreground to be black, then the foreground of the preprocessed main camera image needs to be set to white, and the preprocessed main camera image is subjected to color inversion processing, converting the foreground and the calibration points to white, and the positioning points to black. If the connected component analysis algorithm used considers the foreground to be white, then the preprocessed main camera image does not need to be subjected to color inversion processing. The connected component analysis algorithm used in the embodiments of the present invention considers the foreground to be black. Therefore, the preprocessed main camera image needs to be subjected to color inversion processing, converting the foreground and the calibration points in the preprocessed main camera image to white, and the positioning points to black. At this time, there are only two connected components in the main camera image: the positioning points and the background. Please refer to Figure 7b , Figure 7b which shows Figure 4a the main camera image obtained after preprocessing and color inversion processing of the main camera image shown in Figure 7b where the foreground of the main camera image shown is white.

[0145] Specifically, the connected component analysis algorithm is used to perform connected component analysis on the preprocessed and color-inverted main camera image to obtain the upper left coordinate of the circumscribed rectangle of the connected component, the width and height of the circumscribed rectangle.

[0146] It can be understood that there are only two connected components in the preprocessed and color-inverted main camera image, one connected component is the positioning point, and one connected component is the background area. Therefore, the positioning point can be identified and determined according to the connected component information of these two connected components. Specifically, using the upper left coordinate, width and height of the circumscribed rectangle of the first connected component, calculate the upper right coordinate, lower right coordinate and lower left coordinate of the circumscribed rectangle of this connected component.

[0147] After calculating the upper left coordinate, lower left coordinate, upper right coordinate and lower right coordinate of the circumscribed rectangle of the first connected component, compare the upper left coordinate, lower left coordinate, upper right coordinate and lower right coordinate of the circumscribed rectangle of this connected component with the edge of the main camera image respectively to determine whether any one of the coordinates touches the edge of the main camera image. If any one of the upper left coordinate, upper right coordinate, lower left coordinate and lower right coordinate of the circumscribed rectangle of the first connected component touches the edge of the main camera image, that is, when the judgment result is yes, then the first connected component is not the positioning point, and the second connected component is the positioning point, thereby identifying and determining the positioning point of the main camera image. If there is no coordinate among the upper left coordinate, upper right coordinate, lower left coordinate and lower right coordinate of the circumscribed rectangle of the first connected component that touches the edge of the main camera image, that is, when the judgment result is no, then the first connected component is the positioning point, and the second connected component is not the positioning point, thereby identifying and determining the positioning point of the main camera image, and obtaining the X-axis coordinate and Y-axis coordinate of the positioning point of the main camera image.

[0148] S400: Calculate and obtain the first relative position according to the true resolution of the main camera.

[0149] In this step, the first relative position is the relative position between the positioning point of the main camera image and the center of the main camera FOV. In the embodiments of the present invention, the relative position in the X-axis and the relative position in the Y-axis in physical world units are used to represent the first relative position. The first relative position includes the relative distance in the X-axis in physical world units and the relative distance in the Y-axis in physical world units of the positioning point of the main camera image relative to the center of the main camera FOV.

[0150] In the embodiments of the present invention, before using the main camera to capture the main camera image, set the size (including width and height) of the picture captured by the main camera. In this way, after the main camera captures the main camera image, the width and height of the main camera image can be known.

[0151] Specifically, after using the main camera to capture the main camera image, read the size of the main camera image to obtain the width and height of the main camera image. Then, according to the width and height of the main camera image, calculate the center coordinates of the main camera FOV (Field Of View). The center coordinates of the main camera FOV include the X-axis coordinate and the Y-axis coordinate. According to the X-axis coordinate and the Y-axis coordinate of the positioning point of the main camera image and the X-axis coordinate and the Y-axis coordinate of the center coordinates of the main camera FOV, calculate the relative distance in the X-axis in pixel units and the relative distance in the Y-axis in pixel units of the positioning point relative to the center of the main camera FOV. According to the relative distance in the X-axis in pixel units and the X-axis resolution of the main camera, calculate the relative distance in the X-axis in physical world units of the positioning point relative to the center of the main camera FOV. According to the relative distance in the Y-axis in pixel units and the Y-axis resolution of the main camera, calculate the relative distance in the Y-axis in physical world units of the positioning point relative to the center of the main camera FOV. Combine the relative distance in the X-axis in physical world units and the relative distance in the Y-axis in physical world units of the positioning point relative to the center of the main camera FOV, and the relative position (i.e., the first relative position) between the positioning point of the main camera image and the center of the main camera FOV can be obtained.

[0152] It can be understood that the relative distance in the X-axis in pixel units refers to the X-axis distance between two pixel points (i.e., the positioning point and the center of the main camera FOV) in the pixel coordinate system, and the relative distance in the Y-axis in pixel units refers to the Y-axis distance between two pixel points (i.e., the positioning point and the center of the main camera FOV) in the pixel coordinate system.

[0153] Among them, the relative distance of the X-axis in physical world units refers to the X-axis distance between two points (i.e., the positioning point and the center of the main camera FOV) in the world coordinate system, and the relative distance of the Y-axis in physical world units refers to the Y-axis distance between two points (i.e., the positioning point and the center of the main camera FOV) in the world coordinate system. Among them, the physical world unit is a metric length unit, such as millimeters, centimeters, micrometers, etc.

[0154] In some embodiments, the first relative position is calculated and obtained according to the true resolution of the main camera, specifically including but not limited to the following steps S410 - S460:

[0155] S410: Obtain the coordinates of the center of the main camera FOV according to the width and height of the main camera image.

[0156] In the embodiments of the present invention, divide the width of the main camera image by 2 to obtain the X-axis coordinate of the center of the main camera FOV, divide the height of the main camera image by 2 to obtain the Y-axis coordinate of the center of the main camera FOV, and combine the X-axis coordinate and the Y-axis coordinate of the center of the main camera FOV to obtain the coordinates of the center of the main camera FOV.

[0157] S420: Obtain the relative distance of the X-axis in pixel units of the positioning point relative to the FOV center according to the X-axis coordinate of the positioning point and the X-axis coordinate of the center of the main camera FOV.

[0158] S430: Obtain the relative distance of the Y-axis in pixel units of the positioning point relative to the FOV center according to the Y-axis coordinate of the positioning point and the Y-axis coordinate of the center of the main camera FOV.

[0159] Exemplarily, subtract the X-axis coordinate of the center of the main camera FOV from the X-axis coordinate of the positioning point of the main camera image to obtain the relative distance of the X-axis in pixel units of the positioning point of the main camera image relative to the center of the main camera FOV. And, subtract the Y-axis coordinate of the center of the main camera FOV from the Y-axis coordinate of the positioning point of the main camera image to obtain the relative distance of the Y-axis in pixel units of the positioning point of the main camera image relative to the center of the main camera FOV.

[0160] S440: Obtain the relative distance of the X-axis in physical world units of the positioning point relative to the FOV center according to the relative distance of the X-axis in pixel units and the X-axis resolution of the main camera.

[0161] S450: Obtain the relative distance of the Y-axis in physical world units of the positioning point relative to the FOV center according to the relative distance of the Y-axis in pixel units and the Y-axis resolution of the main camera.

[0162] In an embodiment of the present invention, the relative distance of the positioning point of the main camera image in the X-axis direction in pixel units with respect to the center of the main camera FOV is multiplied by the X-axis resolution of the main camera to obtain the relative distance of the positioning point of the main camera image in the X-axis direction in physical world units with respect to the center of the main camera FOV. Further, the relative distance of the positioning point of the main camera image in the Y-axis direction in pixel units with respect to the center of the main camera FOV is multiplied by the Y-axis resolution of the main camera to obtain the relative distance of the positioning point of the main camera image in the Y-axis direction in physical world units with respect to the center of the main camera FOV.

[0163] S460: Obtain a first relative position based on the relative distance of the X-axis in physical world units and the relative distance of the Y-axis in physical world units.

[0164] In an embodiment of the present invention, by combining the relative distance of the X-axis in physical world units and the relative distance of the Y-axis in physical world units of the positioning point of the main camera image with respect to the center of the main camera FOV, the relative position of the positioning point of the main camera image and the center of the main camera FOV is obtained, that is, the first relative position is obtained.

[0165] S500: Set the processing order of the side camera images according to requirements, and process each side camera image according to the above-mentioned main camera image processing steps based on the processing order to obtain the positioning point of the side camera image and obtain a second relative position.

[0166] It can be understood that since there is distortion in the camera field of view in the side camera images, before processing each side camera image according to the above-mentioned main camera image processing steps based on the processing order, it is necessary to correct the camera field of view in the side camera images. In an embodiment of the present invention, the camera field of view correction method recorded in the invention patent with the patent number ZL202411764896.0, which has been applied for and authorized by the inventor of the present invention, is used to correct the camera field of view in the side camera images to obtain the side camera images with the corrected field of view. Then, each side camera image with the corrected field of view is processed according to the above-mentioned main camera image processing steps based on the processing order. Please refer to Figures 8a to 8d , Figures 8a to 8d respectively show Figures 4b to 4e the side camera images obtained after correcting the camera field of view of the side camera images shown, Figures 8a to 8d the center of the camera field of view is marked on the side camera images with the corrected field of view shown. Among them, the center of the camera field of view is at the position where the colored lines cross as shown in Figures 8a to 8d (that is, the orange line shown in Figure 8a , the red line shown in Figure 8b , the blue line shown in Figure 8c and the green line shown in Figure 8d ).

[0167] In the embodiments of the present invention, the processing order of the side camera images can be set according to actual requirements. For example, the side camera images are as Figures 4b to 4e shown, and the processing order can be set as Figure 4b → Figure 4c → Figure 4d → Figure 4e . Of course, it can also be set to any other suitable processing order, and the embodiments of the present invention do not make any limitations in this regard.

[0168] Exemplarily, the second relative position is the relative position between the positioning point of the side camera image and the center of the side camera FOV. In the embodiments of the present invention, the relative distance of the X-axis and the relative distance of the Y-axis in physical world units are used to characterize the second relative position. The second relative position includes the relative distance of the X-axis and the relative distance of the Y-axis (both in physical world units) of the positioning point of the side camera image relative to the center of the side camera FOV.

[0169] Among them, each side camera image is processed according to the processing steps of the main camera image as described above based on the processing order, the positioning point of the side camera image is obtained, and the second relative position is obtained. Preprocessing the side camera images, identifying the valid dots and the dot information of the valid dots in the side camera images after preprocessing, and calculating the true resolution of the side camera are performed with reference to step S200 and its refinement steps. The positioning point of the side camera image is identified based on the valid dots of the side camera image, and it is performed with reference to step S300 and its refinement steps. The second relative position is calculated and obtained according to the true resolution of the side camera, and it is performed with reference to step S400 and its refinement steps. Finally, each side image corresponds to a second relative position, and multiple different second relative positions are obtained.

[0170] It should be noted that after preprocessing the side camera images, each of the preprocessed side camera images is similar to the preprocessed main camera image. Similarly, after performing color inversion processing on each of the preprocessed side camera images, each of the preprocessed and color-inverted side camera images is similar to the preprocessed and color-inverted main camera image. For the sake of brevity, the embodiments of the present invention do not show each of the preprocessed side camera images and each of the preprocessed and color-inverted side camera images.

[0171] S600: Calculate and obtain the mapping relationship between the main camera FOV and multiple different side camera FOVs based on the first relative position and multiple different second relative positions.

[0172] Specifically, the multiple different second relative positions include the relative positions between the positioning points of the respective side camera images and the centers of the respective side camera FOVs. The mapping relationship refers to a transformation relationship that converts / maps a certain point in the side camera image to a position in the main camera image, and the mapping relationship can be represented in forms such as a projection matrix, a homography matrix, or an affine transformation matrix.

[0173] Specifically, based on the first relative position and the respective second relative positions, the relative orientations of the main camera and the respective side cameras are obtained. Then, based on the relative orientations of the main camera and the respective side cameras, the relative distances along the X-axis and the relative distances along the Y-axis (both in physical world units) between the main camera and the respective side cameras are calculated. Next, based on the relative distances along the X-axis in physical world units and the relative distances along the Y-axis in physical world units between the main camera and the respective side cameras, the mapping relationship between the main camera FOV and the respective side camera FOVs is calculated, so as to obtain the mapping relationship between the main camera FOV and multiple different side camera FOVs. The mapping relationship includes the relative distance along the X-axis in physical world units and the relative distance along the Y-axis in physical world units of the side camera FOV center relative to the main camera FOV center.

[0174] Exemplarily, in some embodiments, based on the first relative position and the multiple different second relative positions, the mapping relationship between the main camera FOV and the multiple different side camera FOVs is calculated and obtained, which specifically includes but is not limited to the following steps S610 - S630:

[0175] S610: Compare the first relative position with the second relative position of the candidate side camera to obtain the relative orientation of the main camera and the candidate side camera.

[0176] In this embodiment, the candidate side camera is any one of the multiple side cameras. Each candidate side camera corresponds to a second relative position, and the main camera corresponds to the first relative position.

[0177] Specifically, by comparing the first relative position with the second relative position corresponding to the candidate side camera, the relative orientation of the main camera and the candidate side camera is obtained. The relative orientation includes that the candidate side camera is located on the right side, left side, upper side, or lower side of the main camera.

[0178] It can be understood that Figure 2a in the schematic diagram of the positional relationship between the main camera and the other four side cameras, the main camera and the other four side cameras are observed from the side, Figure 2a and the positional relationship between the main camera and the other four side cameras shown therein is not the relative orientation of the main camera and the candidate side camera described in this embodiment. The relative orientation of the main camera and the candidate side camera described in this embodiment is the relative orientation of each side camera and the main camera from a top-down perspective. Please refer to Figure 2b , Figure 2bSchematically shows the relative orientation of each side camera and the main camera in a top-down view. Figure 2b Among them, the first side camera 301 is located above the main camera 300, the second side camera 302 is located to the left of the main camera 300, the third side camera 303 is located below the main camera 300, and the fourth side camera 304 is located to the right of the main camera 300. It should be noted that Figure 2b The relative orientation of the shown main camera and the candidate side camera does not impose any limitation on the relative orientation of the main camera and the candidate side camera in other embodiments.

[0179] S620: Calculate the relative X-axis distance and relative Y-axis distance in physical world units between the main camera and the candidate side camera according to the relative orientation, so as to establish the mapping relationship between the main camera FOV and the candidate side camera FOV.

[0180] Specifically, if the candidate side camera is located to the right of the main camera, subtract the relative X-axis distance in physical world units of the positioning point of the main camera image in the first relative position from the center of the main camera FOV from the relative Y-axis distance in physical world units of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera from the center of the side camera FOV, to obtain the relative X-axis distance in physical world units of the center of the candidate side camera FOV relative to the center of the main camera FOV, that is, obtain the relative X-axis distance in physical world units between the main camera and the candidate side camera. And, add the relative Y-axis distance in physical world units of the positioning point of the main camera image in the first relative position from the center of the main camera FOV to the relative X-axis distance in physical world units of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera from the center of the side camera FOV, to obtain the relative Y-axis distance in physical world units of the center of the candidate side camera FOV relative to the center of the main camera FOV, that is, obtain the relative Y-axis distance in physical world units between the main camera and the candidate side camera.

[0181] If the candidate side camera is located on the left side of the main camera, add the relative distance of the positioning point of the main camera image in the first relative position with respect to the X-axis in physical world units from the center of the main camera FOV to the relative distance of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera with respect to the Y-axis in physical world units from the center of the side camera FOV, to obtain the relative distance of the center of the candidate side camera FOV with respect to the center of the main camera FOV in physical world units along the X-axis, that is, obtain the relative distance of the main camera and the candidate side camera in physical world units along the X-axis. Also, subtract the relative distance of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera with respect to the X-axis in physical world units from the center of the side camera FOV from the relative distance of the positioning point of the main camera image in the first relative position with respect to the Y-axis in physical world units from the center of the main camera FOV, to obtain the relative distance of the center of the candidate side camera FOV with respect to the center of the main camera FOV in physical world units along the Y-axis, that is, obtain the relative distance of the main camera and the candidate side camera in physical world units along the Y-axis.

[0182] If the candidate side camera is located above the main camera, add the relative distance of the positioning point of the main camera image in the first relative position with respect to the X-axis in physical world units from the center of the main camera FOV to the relative distance of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera with respect to the X-axis in physical world units from the center of the side camera FOV, to obtain the relative distance of the center of the candidate side camera FOV with respect to the center of the main camera FOV in physical world units along the X-axis, that is, obtain the relative distance of the main camera and the candidate side camera in physical world units along the X-axis. Also, add the relative distance of the positioning point of the main camera image in the first relative position with respect to the Y-axis in physical world units from the center of the main camera FOV to the relative distance of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera with respect to the Y-axis in physical world units from the center of the side camera FOV, to obtain the relative distance of the center of the candidate side camera FOV with respect to the center of the main camera FOV in physical world units along the Y-axis, that is, obtain the relative distance of the main camera and the candidate side camera in physical world units along the Y-axis.

[0183] If the candidate side camera is located below the main camera, subtract the relative X-axis distance in physical world units of the positioning point of the main camera image in the first relative position from the relative X-axis distance in physical world units of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera with respect to the center of the side camera FOV, to obtain the relative X-axis distance in physical world units of the center of the candidate side camera FOV with respect to the center of the main camera FOV, that is, obtain the relative X-axis distance in physical world units between the main camera and the candidate side camera. Also, subtract the relative Y-axis distance in physical world units of the positioning point of the main camera image in the first relative position from the relative Y-axis distance in physical world units of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera with respect to the center of the side camera FOV, to obtain the relative Y-axis distance in physical world units of the center of the candidate side camera FOV with respect to the center of the main camera FOV, that is, obtain the relative Y-axis distance in physical world units between the main camera and the candidate side camera.

[0184] Combine the relative X-axis distance and the relative Y-axis distance in physical world units of the center of the candidate side camera FOV with respect to the center of the main camera FOV, and establish a mapping relationship between the main camera FOV and the candidate side camera FOV. The mapping relationship is used to "re-project" the side camera image content into the main camera image to achieve field of view fusion.

[0185] S630: Establish the mapping relationship between the main camera FOV and each side camera FOV in sequence according to the processing order.

[0186] Specifically, according to the processing order of the side camera images, establish the mapping relationship between the main camera FOV and each side camera FOV in sequence, and obtain the mapping relationship between the main camera FOV and multiple different side camera FOVs.

[0187] S700: Perform field of view fusion between the main camera and the side cameras based on the mapping relationship.

[0188] Specifically, after obtaining the mapping relationship between the main camera FOV and each side camera FOV, for each side camera, according to the mapping relationship between the main camera FOV and the side camera FOV, convert all pixel coordinates in the side camera image captured by the side camera to the coordinate system of the main camera, insert the mapped image content into the corresponding position of the main camera image. Finally, fuse all the side camera images captured by the side cameras into the main camera image to obtain a fused image, thereby achieving field of view fusion between the main camera and each side camera. Please refer to Figure 9a and Figure 9b , Figure 9a shows Figure 4a the main camera image shown and Figures 8a to 8dSchematic diagram before the fusion of the side camera images shown Figure 9b shows Figure 4a the main camera image shown Figures 8a to 8d and the schematic diagram after the fusion of the side camera images shown Figure 9a and Figure 9b both show the center of the camera's field of view, where the center of the camera's field of view is as shown by Figure 9a and Figure 9b the intersecting gray and colored lines shown

[0189] Exemplarily, in some embodiments, before the field of view fusion of the main camera and the side cameras based on the mapping relationship, the multi-camera field of view fusion method further includes but is not limited to the following steps S701 - S702:

[0190] S701: Compare the widths and heights of multiple side camera images, and find the minimum width and the minimum height among all the side camera images

[0191] Exemplarily, obtain the widths and heights of each side camera image, compare the widths of each side camera image, and find the minimum width among all the widths of the side camera images. Compare the heights of each side camera image, and find the minimum height among all the heights of the side camera images

[0192] S702: Crop the width of each side camera image to the minimum width, and crop the height to the minimum height

[0193] Specifically, after obtaining the minimum width and the minimum height, center-crop the width of each side camera image to the minimum width, and center-crop the height of each side camera image to the minimum height to obtain the cropped side camera images. Then, use the cropped side camera images as the side camera images to be fused, and fuse each side camera image to be fused with the main camera image based on the mapping relationship, that is, perform the field of view fusion of the main camera and the side cameras based on the mapping relationship to obtain the fused image

[0194] In summary, the multi-camera field of view fusion method provided by the embodiments of the present invention obtains a main camera image and multiple side camera images by using a main camera and multiple side cameras to photograph the same calibration board, preprocesses the main camera image and the multiple side camera images, and identifies the valid dots and the dot information of the valid dots in the preprocessed main camera image and the multiple side camera images. Based on the valid dots in the main camera image and the multiple side camera images, the positioning points in the main camera image and the multiple side camera images are identified, the true resolutions of the main camera and the multiple side cameras are calculated, the first relative position and multiple second relative positions are calculated according to the calculated true resolutions of the main camera and the multiple side cameras, the mapping relationship between the main camera FOV and the multiple side camera FOVs is calculated according to the first relative position and the multiple second relative positions, and the field of view fusion of the main camera and the multiple side cameras is performed based on the mapping relationship. In this way, the relative positions of the main camera and each side camera can be determined, the field of view fusion of the main camera and the side cameras can be realized, so that the detection object outside the field of view of the detection main camera can be clearly identified, the detection range is increased, the detection efficiency is improved, and the user experience is enhanced.

[0195] The embodiments of the present invention provide a computer-readable storage medium, on which computer program instructions executable by a processor are stored. When the computer program instructions are executed by the processor, the computer is enabled to execute any one of the multi-camera field of view fusion methods provided by the embodiments of the present invention, or execute the steps in any one of the implementation manners of any one of the multi-camera field of view fusion methods provided by the embodiments of the present invention.

[0196] Those skilled in the art can understand that the embodiments provided by the present invention are only illustrative. The writing order of the steps in the method of the embodiments does not mean a strict execution order and does not constitute any limitation on the implementation process. It can be adjusted, combined, and deleted according to actual needs. The modules or sub-modules, units or sub-units, etc. in the device or system of the embodiments can be combined, divided, and deleted according to actual needs. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0197] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Those skilled in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. This computer program can be stored in a computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0198] It should be noted that the above embodiments are for illustrating the technical concept and characteristics of the present invention. The purpose is to enable those familiar with this technology to understand the content of the present invention and implement it accordingly, and it cannot be used to limit the scope of the protection of the rights of the present invention. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, modify them according to the technical solutions recorded in the embodiments of the present invention, or perform equivalent replacements on some of the technical features. It can be understood that these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should be regarded as equivalent changes and modifications made based on the embodiments of the present invention, and should all fall within the scope covered by the claims of the present invention.

Claims

1. A multi-camera field of view fusion method, characterized in that Including: The main camera and multiple side cameras both perform shooting based on the same calibration board to obtain the main camera image and multiple side camera images; Main camera image processing: Preprocess the main camera image, and after preprocessing, identify the valid dots and dot information of the valid dots in the main camera image. The dot information includes the actual diameter of the valid dots, the width and height of the circumscribed rectangle, and calculate the true resolution of the main camera; Calculating the true resolution of the main camera includes: Sum the widths of all valid dot circumscribed rectangles to obtain the sum of the circumscribed widths; Obtain the dot diameter in the X direction based on the sum of the circumscribed widths and the total number of valid dots, and calculate the X-axis resolution of the main camera according to the dot diameter in the X direction and the actual diameter; Sum the heights of all valid dot circumscribed rectangles to obtain the sum of the circumscribed heights; Obtain the dot diameter in the Y direction based on the sum of the circumscribed heights and the total number of valid dots, and calculate the Y-axis resolution of the main camera according to the dot diameter in the Y direction and the actual diameter. The true resolution of the main camera includes the X-axis resolution and Y-axis resolution of the main camera; Identify the positioning point of the main camera image based on the valid dots in the main camera image; Calculate and obtain the first relative position according to the true resolution of the main camera. The first relative position is the relative position between the positioning point of the main camera image and the center of the main camera FOV; Side camera image processing: Set the processing order of the side camera images according to requirements, and process each side camera image according to the above main camera image processing steps based on the processing order to obtain the positioning point of the side camera image and obtain the second relative position. The second relative position is the relative position between the positioning point of the side camera image and the center of the side camera FOV; Main and side camera field of view fusion: Based on the first relative position and multiple different second relative positions, calculate and obtain the mapping relationship between the main camera FOV and multiple different side camera FOVs; Perform field of view fusion of the main camera and the side cameras based on the mapping relationship.

2. The method according to claim 1, characterized in that The preprocessing of the main camera image includes: Convert the input main camera image into a single-channel grayscale image; Perform threshold segmentation on the converted main camera image to convert the main camera image into black and white colors.

3. The method according to claim 1, characterized in that, The identification of the valid dots and dot information of the valid dots in the main camera image after preprocessing includes: After preprocessing, perform connected component analysis on the main camera image, find the median of the areas of all connected components, and set the area difference threshold according to the requirement of identifying dots; Traverse all connected components, calculate the four corner coordinates of the circumscribed rectangle of each connected component, and judge all connected components based on the area difference threshold and the four corner coordinates to identify all valid dots and dot information of the valid dots in the main camera image.

4. The method according to claim 1, wherein The identification of the positioning point of the main camera image based on the valid dots in the main camera image includes: Set the foreground of the preprocessed main camera image to white, and perform connected component analysis on the main camera image again to obtain the upper left corner coordinates, width and height of the circumscribed rectangle of the connected component; Calculate the upper right corner coordinates, lower right corner coordinates and lower left corner coordinates of the circumscribed rectangle of the connected component through the upper left corner coordinates, width and height of the circumscribed rectangle of the first connected component; Determine whether any of the upper-left coordinate, lower-left coordinate, upper-right coordinate, and lower-right coordinate of the circumscribed rectangle of the connected component touches the edge of the main camera image; When the judgment result is yes, it indicates that the second connected component is the positioning point; When the judgment result is no, it indicates that the first connected component is the positioning point.

5. The method according to claim 1, characterized in that, The obtaining the first relative position according to the true resolution of the main camera includes: Obtain the center coordinates of the main camera FOV according to the width and height of the main camera image; Obtain the relative distance of the X-axis of the positioning point relative to the FOV center in pixel units according to the X-axis coordinate of the positioning point and the X-axis coordinate of the main camera FOV center; Obtain the relative distance of the Y-axis of the positioning point relative to the FOV center in pixel units according to the Y-axis coordinate of the positioning point and the Y-axis coordinate of the main camera FOV center; Obtain the relative distance of the X-axis of the positioning point relative to the FOV center in physical world units according to the relative distance of the X-axis in pixel units and the X-axis resolution of the main camera; Obtain the relative distance of the Y-axis of the positioning point relative to the FOV center in physical world units according to the relative distance of the Y-axis in pixel units and the Y-axis resolution of the main camera; Obtain the first relative position based on the relative distance of the X-axis in physical world units and the relative distance of the Y-axis in physical world units.

6. The method according to claim 1, wherein The calculating and obtaining the mapping relationship between the main camera FOV and multiple different side camera FOVs based on the first relative position and multiple different second relative positions includes: Compare the first relative position with the second relative position of the candidate side camera to obtain the relative orientation between the main camera and the candidate side camera, where the candidate side camera is any one of the multiple side cameras; Calculate the relative distance of the X-axis and the relative distance of the Y-axis in physical world units between the main camera and the candidate side camera according to the relative orientation to establish the mapping relationship between the main camera FOV and the candidate side camera FOV; Establish the mapping relationship between the main camera FOV and each side camera FOV in sequence according to the processing order.

7. The method according to claim 1, characterized in that, Before performing the field of view fusion between the main camera and the side camera based on the mapping relationship, the method further includes: Compare the widths and heights of the multiple side camera images, and find the minimum width and the minimum height among all the side camera images; Crop the width of each side camera image to the minimum width and crop the height to the minimum height.

8. An electronic device, characterized in that, including: A processor and a memory communicatively connected to the processor; The memory stores computer program instructions executable by the processor, and the processor executes the computer program instructions to cause the electronic device to execute the multi-camera field of view fusion method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions executable by the processor, and the processor executes the computer program instructions to cause the computer to execute the multi-camera field of view fusion method according to any one of claims 1-7.

Images