Multi-camera view fusion method, electronic equipment and storage medium

Abstract

The embodiment of the invention relates to a multi-camera visual field fusion method, electronic equipment and a storage medium, and the method comprises the steps: shooting a same calibration plate through employing a main camera and a plurality of side cameras, so as to obtain a main camera image and a plurality of side camera images, and recognizing the positioning points of the main camera image and the plurality of side camera images; the method comprises the following steps: calculating and obtaining a first relative position and a plurality of second relative positions by calculating the real resolutions of a main camera and a plurality of side cameras, calculating and obtaining a mapping relation between the FOV of the main camera and the FOV of the plurality of side cameras according to the first relative position and the plurality of second relative positions, and performing view fusion of the main camera and the plurality of side cameras based on the mapping relation. In this way, the relative positions of the main camera and the side cameras can be determined, view fusion of the main camera and the side cameras is achieved, detection objects outside the view of the main camera can be clearly recognized, the detection range is increased, the detection efficiency is improved, and the user experience is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of optical detection, and in particular, to a multi-camera field of view fusion method, an electronic device, and a storage medium. Background Art

[0002] Most of the optical imaging modules of current AOI devices (optical detection devices) use a configuration of a single top camera to inspect objects to be measured on the bottom track. However, there are some scenarios where only the top camera cannot meet the detection requirements. The field of view of a single top camera is limited, and the size of the captured FOV is limited. When the area of the detection object is large, a side camera needs to be added to take pictures from the side to obtain information that cannot be obtained by the top main camera.

[0003] In order to ensure that the optical imaging module of the AOI device can cover as much detection area as possible, R & D designers often set the main camera and the side cameras at different heights to save space in the horizontal direction. Also, because the side cameras cannot block the field of view of the main camera, the side cameras need to be arranged obliquely outside the optical path of the main camera. These two points result in the coordinate systems in the field of view of the main camera having no connection with the coordinate systems in the fields of view of the respective side cameras. This shortcoming makes it impossible to connect the defect positions detected in the main camera with the defect positions detected in the side cameras during the later detection process. For example: When a user wants to detect a component in the field of view of the main camera with a virtual solder joint or other soldering defects, the user detects multiple components in the side camera at this time, but the user cannot tell which component is the one seen by the user in the field of view of the main camera. Since the field of view of the main camera is not connected to the fields of view of the respective side cameras, the relative positions of the main camera and the respective side cameras cannot be determined, resulting in detection deviation and poor user experience. Summary of the Invention

[0004] In view of this, an object of the embodiments of the present invention is to provide a multi-camera field of view fusion method, an electronic device, and a storage medium, aiming to solve the technical problems in the prior art that component detection deviation and poor user experience are caused by the inability to fuse the fields of view of the main camera and the side cameras.

[0005] To solve the above technical problems, the embodiments of the present invention provide the following technical solutions: In a first aspect, the embodiments of the present invention provide a multi-camera field of view fusion method, including: Taking pictures by a main camera and multiple side cameras based on the same calibration board to obtain a main camera image and multiple side camera images; Main camera image processing: Preprocessing the main camera image, and after preprocessing, identifying valid dots and dot information of the valid dots in the main camera image, and calculating the true resolution of the main camera; Identify the positioning point of the main camera image based on the valid dots in the main camera image; Calculate and obtain the first relative position according to the true resolution of the main camera. The first relative position is the relative position between the positioning point of the main camera image and the center of the main camera FOV; Side camera image processing: Set the processing order of the side camera image according to requirements, and process each side camera image according to the above main camera image processing steps based on the processing order to obtain the positioning point of the side camera image and obtain the second relative position. The second relative position is the relative position between the positioning point of the side camera image and the center of the side camera FOV; Main and side camera field of view fusion: Based on the first relative position and multiple different second relative positions, calculate and obtain the mapping relationship between the main camera FOV and multiple different side camera FOVs; Perform field of view fusion of the main camera and the side camera based on the mapping relationship.

[0006] In some embodiments, the preprocessing of the main camera image includes: Convert the input main camera image into a single-channel grayscale image; Perform threshold segmentation on the converted main camera image to convert the main camera image into two colors, black and white.

[0007] In some embodiments, the identification of valid dots and dot information of valid dots in the main camera image after preprocessing includes: Perform connected component analysis on the main camera image after preprocessing, find the median of the areas of all connected components, and set the area difference threshold according to the requirement of identifying dots; Traverse all connected components, calculate the four corner coordinates of the circumscribed rectangle of each connected component, and judge all connected components based on the area difference threshold and the four corner coordinates to identify all valid dots and dot information of valid dots in the main camera image.

[0008] In some embodiments, the dot information includes the actual diameter of the valid dot, the width and height of the circumscribed rectangle. The calculation of the true resolution of the main camera includes: Sum the widths of the circumscribed rectangles of all valid dots to obtain the sum of the circumscribed widths; Obtain the dot diameter in the X direction based on the sum of the circumscribed widths and the total number of valid dots, and calculate and obtain the X-axis resolution of the main camera according to the dot diameter in the X direction and the actual diameter; Sum the heights of the circumscribed rectangles of all valid dots to obtain the sum of the circumscribed heights; Obtain the diameter of the dot in the Y direction based on the sum of the circumscribed heights and the total number of valid dot centers, and calculate and obtain the Y-axis resolution of the main camera according to the diameter of the dot in the Y direction and the actual diameter. The true resolution of the main camera includes the X-axis resolution and the Y-axis resolution of the main camera.

[0009] In some embodiments, the identifying the positioning point of the main camera image based on the valid dots in the main camera image includes: Set the foreground of the preprocessed main camera image to white, and perform connected component analysis on the main camera image again to obtain the upper left coordinate, width, and height of the circumscribed rectangle of the connected component. Calculate the upper right coordinate, lower right coordinate, and lower left coordinate of the circumscribed rectangle of the connected component through the upper left coordinate, width, and height of the circumscribed rectangle of the first connected component. Determine whether any one of the upper left coordinate, lower left coordinate, upper right coordinate, and lower right coordinate of the circumscribed rectangle of the connected component touches the edge of the main camera image. When the judgment result is yes, it means that the second connected component is the positioning point. When the judgment result is no, it means that the first connected component is the positioning point.

[0010] In some embodiments, the calculating and obtaining the first relative position according to the true resolution of the main camera includes: Obtain the center coordinates of the main camera FOV according to the width and height of the main camera image. Obtain the X-axis relative distance of the positioning point relative to the FOV center in pixel units according to the X-axis coordinate of the positioning point and the X-axis coordinate of the main camera FOV center. Obtain the Y-axis relative distance of the positioning point relative to the FOV center in pixel units according to the Y-axis coordinate of the positioning point and the Y-axis coordinate of the main camera FOV center. Obtain the X-axis relative distance of the positioning point relative to the FOV center in the physical world unit according to the X-axis relative distance in pixel units and the X-axis resolution of the main camera. Obtain the Y-axis relative distance of the positioning point relative to the FOV center in the physical world unit according to the Y-axis relative distance in pixel units and the Y-axis resolution of the main camera. Obtain the first relative position based on the X-axis relative distance in the physical world unit and the Y-axis relative distance in the physical world unit.

[0011] In some embodiments, the calculating and obtaining the mapping relationship between the main camera FOV and multiple different side camera FOVs based on the first relative position and multiple different second relative positions includes: Compare the first relative position with the second relative position of a candidate side camera to obtain the relative orientation between the main camera and the candidate side camera, where the candidate side camera is any one of a plurality of side cameras; Calculate the relative X-axis distance and relative Y-axis distance in physical world units between the main camera and the candidate side camera according to the relative orientation, so as to establish a mapping relationship between the FOV of the main camera and the FOV of the candidate side camera; Establish the mapping relationship between the FOV of the main camera and the FOV of each side camera in sequence according to the processing order.

[0012] In some embodiments, before performing the field of view fusion between the main camera and the side camera based on the mapping relationship, the method further includes: Compare the widths and heights of the images of multiple side cameras, and find the minimum width and the minimum height among all the side camera images; Crop the width of each side camera image to the minimum width and crop the height to the minimum height.

[0013] In a second aspect, an embodiment of the present invention provides an electronic device, including: A processor and a memory communicatively connected to the processor; The memory stores computer program instructions executable by the processor, and the processor executes the computer program instructions to cause the electronic device to execute any one of the multi-camera field of view fusion methods proposed in the first aspect.

[0014] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, where the computer-readable storage medium stores computer program instructions executable by a processor, and the processor executes the computer program instructions to cause the computer to execute any one of the multi-camera field of view fusion methods proposed in the first aspect.

[0015] The embodiments of the present invention have the following beneficial effects: Different from the prior art, the multi-camera field-of-view fusion method provided by the embodiments of the present invention obtains a main camera image and a plurality of side camera images by using a main camera and a plurality of side cameras to photograph the same calibration board, preprocesses the main camera image and the plurality of side camera images, and identifies the valid dots and the dot information of the valid dots in the preprocessed main camera image and the plurality of side camera images. Based on the valid dots of the main camera image and the plurality of side camera images, the positioning points in the main camera image and the plurality of side camera images are identified, the true resolutions of the main camera and the plurality of side camera are calculated, the first relative position and a plurality of second relative positions are calculated according to the calculated true resolutions of the main camera and the plurality of side camera, the mapping relationship between the main camera FOV and the plurality of side camera FOVs is calculated according to the first relative position and the plurality of second relative positions, and the field-of-view fusion of the main camera and the plurality of side cameras is performed based on the mapping relationship. In this way, the relative positions of the main camera and each side camera can be determined, the field-of-view fusion of the main camera and the side cameras can be realized, so that the detection objects outside the field of view of the main camera can be clearly identified, the detection range is increased, the detection efficiency is improved, and the user experience is enhanced. Description of the Drawings

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the prior art or embodiments. Obviously, the following-described drawings only show some embodiments of the present invention and should not be regarded as limiting the protection scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0017] Figure 1 is a schematic diagram of the application scenario of the multi-camera field-of-view fusion method in some embodiments of the present invention; Figure 2a is a schematic diagram of the positional relationship between the main camera and the plurality of side cameras in some embodiments of the present invention; Figure 2b is a schematic diagram of the relative orientation between the main camera and the plurality of side cameras in some embodiments of the present invention; Figure 3 is a schematic diagram of the positional relationship between the positioning points and the calibration points in the calibration pattern array on the calibration board provided by some embodiments of the present invention; Figure 4a is the main camera image obtained by the main camera photographing the calibration board provided by some embodiments of the present invention; Figure 4b is the side camera image obtained by the first side camera photographing the calibration board provided by some embodiments of the present invention, wherein the camera field of view of the side camera image is not corrected; Figure 4cIt is a side camera image obtained by a second side camera shooting a calibration board in some embodiments of the present invention, wherein the camera field of view of the side camera image is not corrected; Figure 4d It is a side camera image obtained by a third side camera shooting a calibration board in some embodiments of the present invention, wherein the camera field of view of the side camera image is not corrected; Figure 4e It is a side camera image obtained by a fourth side camera shooting a calibration board in some embodiments of the present invention, wherein the camera field of view of the side camera image is not corrected; Figure 5 It is a schematic structural diagram of an electronic device provided in some embodiments of the present invention; Figure 6 It is a schematic flowchart of a multi-camera field of view fusion method provided in some embodiments of the present invention; Figure 7a It is Figure 4a The preprocessed main camera image obtained by preprocessing the shown main camera image; Figure 7b It is Figure 4a The preprocessed and color-inverted main camera image obtained by preprocessing and color-inverting the shown main camera image; Figure 8a It is Figure 4b The side camera image obtained after the camera field of view of the shown side camera image; Figure 8b It is Figure 4c The side camera image obtained after the camera field of view of the shown side camera image; Figure 8c It is Figure 4d The side camera image obtained after the camera field of view of the shown side camera image; Figure 8d It is Figure 4e The side camera image obtained after the camera field of view of the shown side camera image; Figure 9a It is Figure 4a The schematic diagram before the fusion of the shown main camera image and Figures 8a to 8d The shown side camera image; Figure 9b It is Figure 4a The schematic diagram after the fusion of the shown main camera image and Figures 8a to 8d The shown side camera image. Detailed implementation manners

[0018] To make the objectives and advantages of the embodiments of the present invention more comprehensible, the following will clearly and completely describe the technical solutions of the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part rather than all of the embodiments of the present invention. The following detailed description of the embodiments of the present invention in the accompanying drawings does not limit the scope of the present invention claimed, but only represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0019] It should be noted that if there is no conflict, the various technical features involved in the embodiments of the present invention described below can be combined with each other and are all within the scope of protection of the present invention. In addition, although functional module division is carried out in the device or structure schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from the module division in the device or the flowchart. In addition, the "first", "second", "third" and other similar expressions used in this article do not limit the data and the execution order, but are only for the purpose of convenient explanation and to distinguish the same items or similar items with basically the same functions and effects, and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of technical features.

[0020] Unless otherwise defined, the technical terms and scientific terms used in this specification have the same meanings as those generally understood by those of ordinary skill in the technical field to which the present invention belongs. The terms used in this specification are only for the purpose of describing specific embodiments and are not used to limit the present invention. It should be understood that the term "and / or" used in this specification includes any and all combinations of one or more of the related listed items.

[0021] In semiconductor manufacturing, it is usually necessary to use an Automated Optical Inspection (AOI) device to detect and analyze problems such as the appearance, quality, and performance of products. The light machine and the camera are the optical components that make up the AOI device. Most of the current optical imaging modules of AOI devices use a configuration of a single top camera to inspect the objects to be measured on the bottom track. However, there are some scenarios where only relying on the top camera cannot meet the detection requirements. The field of view of a single top camera is limited, and the size of the FOV captured is limited. When the area of the detection object is large, it is necessary to add a side camera to take pictures from the side to obtain information that cannot be obtained by the top main camera.

[0022] In order to ensure that the optical imaging module of the AOI device can cover as much detection area as possible, R & D designers often set the main camera and the side cameras at different heights to save space in the horizontal direction. Also, since the side cameras cannot block the field of view of the main camera, the side cameras need to be arranged obliquely outside the optical path of the main camera. These two points result in the coordinate systems in the field of view of the main camera having no connection with the coordinate systems in the fields of view of the individual side cameras. This drawback makes it impossible to relate the defect positions detected in the main camera to those detected in the side cameras during the later detection process. For example, when a user wants to detect a component with a solder joint void or other soldering defect in the field of view of the main camera, the user detects multiple components in the side camera at this time, but the user cannot tell which component is the one seen by the user in the field of view of the main camera. Since the field of view of the main camera is not connected to the fields of view of the individual side cameras, the relative positions of the main camera and the individual side cameras cannot be determined, resulting in detection deviations easily occurring and a poor user experience.

[0023] In view of this, the embodiments of the present invention provide a multi-camera field of view fusion method. By using the main camera and multiple side cameras to photograph the same calibration board to obtain the main camera image and multiple side camera images, preprocessing the main camera image and the multiple side camera images, and identifying the valid dots and the dot information of the valid dots in the preprocessed main camera image and the multiple side camera images, identifying the positioning points in the main camera image and the multiple side camera images based on the valid dots of the main camera image and the multiple side camera images, calculating the true resolutions of the main camera and the multiple side cameras, calculating and obtaining the first relative position and multiple second relative positions according to the calculated true resolutions of the main camera and the multiple side cameras, calculating the mapping relationship between the main camera FOV and the multiple side camera FOVs according to the first relative position and the multiple second relative positions, and performing field of view fusion of the main camera and the multiple side cameras based on the mapping relationship. In this way, the relative positions of the main camera and the individual side cameras can be determined, the field of view fusion of the main camera and the side cameras can be achieved, so as to clearly identify the detection objects outside the field of view of the main camera, increase the detection range, improve the detection efficiency, and enhance the user experience.

[0024] Please refer to Figure 1 , Figure 1 which schematically shows an application scenario diagram of the multi-camera field of view fusion method provided by some embodiments of the present invention.

[0025] As Figure 1As shown, the application scenario includes an electronic device 100. It can be understood that to achieve multi-camera field of view fusion, the electronic device 100 also needs to work in coordination with some other software and hardware, such as cameras, image processing software, display devices, and storage devices, etc. Those skilled in the art can add, delete, or change the required software and hardware according to actual needs to cooperate with the electronic device 100 to complete multi-camera field of view fusion. It should be understood that these software and hardware working in coordination can be configured in the electronic device 100 according to actual needs, or exist independently of the electronic device 100.

[0026] It should be understood that in Figure 1 the application scenario shown, the electronic device 100 is a desktop computer, but it does not impose any limitations on the structure, type, quantity, etc. of the electronic devices in other application scenarios. For example, in some other embodiments, the electronic device can also be a laptop computer, a tablet computer, or other suitable types of devices, or it can also be a server, such as a server deployed in the cloud.

[0027] For example, please refer to Figure 2a , Figure 2a which shows the positional relationship between the main camera and four other side cameras in some embodiments of the present invention. Figure 2a As shown, it includes a calibration board 200, a main camera 300, and four side cameras. Among them, the four side cameras are respectively a first side camera 301, a second side camera 302, a third side camera 303, and a fourth side camera 304. The first side camera 301, the second side camera 302, the third side camera 303, and the fourth side camera 304 are arranged on the lower side of the main camera 300 and are located outside the optical path of the main camera 300 to avoid affecting the main camera 300 from photographing the calibration board 200, the element to be detected, or other objects.

[0028] Please refer to Figure 3 , in some embodiments, the calibration board 200 includes a calibration pattern array 210, where the calibration pattern array 210 includes positioning points 211 and calibration points 212. The positioning points 211 are arranged at the center of the calibration pattern array 210 to facilitate subsequent determination of the relative positions of the field of view center of the main camera and the field of view centers of the respective side cameras, and then perform field of view fusion of the main camera and the respective side cameras. It should be understood that Figure 3 only schematically shows the positions of the positioning points 211 and the calibration points 212 of the calibration pattern array 210 on the calibration board 200, but it does not impose any limitations on the positions, shapes, arrangement methods, quantities, etc. of the positioning points and calibration points in other embodiments, as long as both the main camera and the side cameras can photograph the positioning points 211.

[0029] When performing multi-camera field-of-view fusion, the main camera and multiple side cameras both take pictures based on the same calibration board to obtain the main camera image and multiple side camera images. For example, Figure 2a the main camera 300, the first side camera 301, the second side camera 302, the third side camera 303, and the fourth side camera 304 shown in Figures 4a to 4e are used to take pictures of the calibration board 200 to obtain the main camera image and four side camera images. Please refer to Figures 4a to 4e which respectively shows the main camera image and the four side camera images obtained by the main camera 300, the first side camera 301, the second side camera 302, the third side camera 303, and the fourth side camera 304. Among them, Figure 4a the center of the camera field of view in the main camera image is marked, and the center of the camera field of view is like the Figure 4a gray line shown.

[0030] It can be understood that there is no distortion in the camera field of view in the main camera image because the main camera is vertically set at the target position where the component to be detected is placed, such as the position of the calibration board 200 shown in Figure 2a . However, there will be a depth-of-field problem in the camera fields of view in the four side camera images because the four side cameras are not vertically set at the target position where the component to be detected is placed. Therefore, a Scheimpflug lens is installed on the four side cameras. The Scheimpflug lens is installed on the side cameras because it can change the optical axis direction of the lens, making the optical axis of the obliquely arranged side cameras perpendicular to the object surface, so that the entire side image obtained by shooting is in focus, without defocus, solving the depth-of-field problem caused by the camera tilt and ensuring that the detection effect is not affected.

[0031] Specifically, as a device providing computing and control capabilities, after the electronic device 100 captures a main camera image and multiple side camera images using the main camera and multiple side cameras, it preprocesses the main camera image and multiple side camera images, identifies valid dots in the preprocessed main camera image and multiple side camera images, then identifies positioning points in the main camera image and multiple side camera images based on the valid dots of the main camera image and multiple side camera images, calculates the true resolutions of the main camera and multiple side cameras, calculates and obtains a first relative position and multiple second relative positions according to the true resolutions of the main camera and multiple side cameras. The first relative position is the relative position between the positioning point of the main camera image and the center of the main camera FOV, and the second relative position is the relative position between the positioning point of the side camera image and the center of the side camera FOV. Finally, based on the first relative position and multiple different second relative positions, the mapping relationship between the main camera FOV and multiple different side camera FOVs is calculated and obtained, and the field of view fusion of the main camera and multiple side cameras is performed based on the mapping relationship. In this way, the relative positions of the main camera and each side camera can be determined, the field of view fusion of the main camera and side cameras can be realized, so that the detection object outside the main camera field of view can be clearly identified, the detection range is increased, the detection efficiency is improved, and the user experience is enhanced.

[0032] To facilitate the understanding of the multi-camera field of view fusion method provided by the embodiments of the present invention, the electronic device provided by the embodiments of the present invention will be introduced in detail first.

[0033] Please refer to Figure 5 , Figure 5 which schematically shows the structural diagram of the electronic device provided by some embodiments of the present invention.

[0034] As Figure 5 shown, the electronic device 100 includes at least one processor 110 and a memory 120 that are communicatively connected, Figure 5 taking the connection by the bus system 130 and one processor as an example. Among them, each component in the electronic device 100 is coupled together through the bus system 130, and the bus system 130 is used to realize the connection and communication between each component. It is easy to understand that in addition to the data bus, the bus system 130 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clear illustration and concise space, in Figure 5 all kinds of buses are labeled as the bus system 130. It can be understood that Figure 5 the structure shown in the embodiment is only schematic, and it does not impose any limitation on the structure of the above-mentioned electronic device. For example, the above-mentioned electronic device may also include more or fewer components than Figure 5 the structure shown, or have a configuration different from Figure 5 the structure shown.

[0035] Specifically, the processor 110 is configured to provide computing and control capabilities to control the electronic device 100 to perform corresponding tasks. For example, it controls the above-mentioned electronic device 100 to execute any one of the multi-camera field-of-view fusion methods provided by the embodiments of the present invention, or to execute the steps in any possible implementation manner of any one of the multi-camera field-of-view fusion methods provided by the embodiments of the present invention. Those skilled in the art can understand that the processor 110 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0036] The memory 120, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, instructions, and modules. For example, the programs, instructions, and modules corresponding to the multi-camera field-of-view fusion method in the embodiments of the present invention. In some embodiments, the memory 120 may include a program storage area and a data storage area. The program storage area can store an operating system and application programs required for at least one function. The data storage area can store data created according to the use of the processor 110, etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the non-transitory software programs, instructions, and modules stored in the memory 120, so as to implement any one of the multi-camera field-of-view fusion methods provided by the embodiments of the present invention, or to execute the steps in any possible implementation manner of any one of the multi-camera field-of-view fusion methods provided by the embodiments of the present invention. In some embodiments, the memory 120 may include high-speed random access memory and may also include non-transitory memory. For example, at least one disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 120 may further include a memory remotely set relative to the processor 110, and these remotely set memories can be connected to the processor 110 through a communication network. It can be understood that examples of the above communication network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0037] As can be understood from the above, the execution subject of any multi-camera field-of-view fusion method provided by the embodiments of the present invention can be any suitable type of electronic device with certain computing and control capabilities. For example, it can be implemented by the above-mentioned electronic device 100. In some feasible implementation manners, the multi-camera field-of-view fusion method provided by the embodiments of the present invention can be implemented by a processor executing computer program instructions stored in a memory.

[0038] Next, the multi-camera field-of-view fusion method provided by the embodiments of the present invention will be described in detail in combination with the exemplary applications and implementations of the electronic device provided by the embodiments of the present invention.

[0039] Please refer to Figure 6 , Figure 6 which schematically shows the flowchart of the multi-camera field-of-view fusion method provided by some embodiments of the present invention.

[0040] Those skilled in the art can understand that the multi-camera field-of-view fusion method provided by the embodiments of the present invention can be applied to the above-mentioned electronic device (for example, electronic device 100). Specifically, the execution subject of the multi-camera field-of-view fusion method is one or at least two processors of the electronic device.

[0041] As Figure 6 shown, the multi-camera field-of-view fusion method includes but is not limited to the following steps S100-S700: S100: Shoot through the main camera and multiple side cameras based on the same calibration board to obtain the main camera image and multiple side camera images.

[0042] In the embodiments of the present invention, a positioning block and multiple calibration blocks are arranged on the calibration board. The positioning block and the calibration blocks form a calibration pattern array, and the positioning block is located at the center of the calibration pattern array. The shape of the calibration block can be circular, rectangular, square, etc., and the shape of the positioning block can be annular, diamond-shaped, etc., and the shapes of the positioning block and the calibration block are different. In the embodiments of the present invention, the shape of the calibration block is set to be circular, and the shape of the positioning block is set to be annular, and the outer diameter of the positioning block is the same as the diameter of the calibration block. It is easy to understand that the distribution shape of the positioning block and the calibration blocks on the calibration board (that is, the distribution shape of the calibration pattern array) can be regular shapes such as rectangular, circular, or square, and can also be irregular shapes such as diamond-shaped, oval-shaped, or triangular. In the embodiments of the present invention, the distribution shape of the positioning block and the calibration blocks is set to be square, which is convenient for subsequent calculations.

[0043] Exemplarily, fix the calibration board at the target position of the stable bracket to ensure that the calibration board does not shake or deform during the shooting process. The size of the calibration board is appropriate and needs to be within the field of view of both the main camera and multiple side cameras at the same time. Set a reasonable light intensity to avoid reflection and shadow, and improve the image clarity and the recognition accuracy of the positioning points and calibration points.

[0044] Specifically, the main camera and multiple side cameras are fixedly installed at predetermined positions to ensure that each camera does not move during shooting. A shooting instruction is sent to the main camera and the multiple side cameras. After receiving the shooting instruction, the main camera and the multiple side cameras shoot a calibration board set at the target position to obtain the main camera image and the multiple side camera images, thereby realizing shooting by the main camera and the multiple side cameras based on the same calibration board to obtain the main camera image and the multiple side camera images. The main camera image and the multiple side camera images include a circular or approximately circular shadow area (i.e., calibration points) formed by calibration blocks and an annular or approximately annular shadow area (i.e., positioning points) formed by positioning blocks.

[0045] S200: Preprocess the main camera image, and after preprocessing, identify the valid dots and dot information of the valid dots in the main camera image, and calculate the true resolution of the main camera.

[0046] In some embodiments, preprocess the main camera image, that is, remove the color information from the main camera image and only retain the luminance information, so as to convert the main camera image into a grayscale image with only one grayscale value for each pixel. Then, use Gaussian filtering to smooth the grayscale image and remove the noise in the grayscale image. Finally, perform binarization processing on the grayscale image after removing the noise to convert the grayscale image after removing the noise into a pure black-and-white image (i.e., a binary image including only black and white colors), obtaining the preprocessed main camera image. Please refer to Figure 7a , Figure 7a which schematically shows the preprocessed main camera image obtained by preprocessing the Figure 4a shown main camera image.

[0047] In some embodiments, use a circular detection algorithm (such as the Hough transform algorithm) to detect the dots in the preprocessed main camera image and extract the identified detected dots. Among them, the Hough transform algorithm is an image processing algorithm used to identify the dots in an image and returns the center coordinates and radius of the dots. In some embodiments, any other suitable algorithm can also be used to detect the dots in the preprocessed main camera image, and the embodiments of the present invention do not make any limitation in this regard.

[0048] In some embodiments, for each extracted dot, according to a preset rule, determine whether the dot is valid. If it is valid, determine that the dot is a valid dot and screen out the dot as a valid dot. If it is invalid, determine that the dot is an invalid dot and eliminate it or perform other processing. Among them, the preset rules include that the radius size of the dot is within a preset range, the center position of the dot conforms to the layout of the calibration pattern array, etc. Of course, those skilled in the art can add, modify, or delete the preset rules according to actual needs, and the embodiments of the present invention do not make any limitation in this regard.

[0049] In some embodiments, after screening out valid dots from the extracted dots, dot information of the valid dots is obtained by using a circular detection algorithm (such as the Hough transform algorithm, etc.). The dot information includes: the centroid and radius of the valid dot, where the centroid of the valid dot is the central coordinate of the valid dot. The Hough transform algorithm gives the central coordinate and radius of each valid dot, so that the dot information of the valid dots can be obtained through the Hough transform algorithm.

[0050] In some embodiments, according to the dot information of the valid dots, the true resolution of the main camera is calculated. The true resolution of the main camera includes the X-axis resolution and Y-axis resolution of the main camera. In the embodiments of the present invention, the dot information of the valid dots includes the actual diameter of the valid dot. The circular calibration block on the calibration plate has a known actual size in the actual physical space. The actual size includes the actual diameter of the circular calibration block, and the actual diameter of the circular calibration block is the actual diameter of the valid dot.

[0051] Exemplarily, the circumscribed rectangles of all valid dots are obtained, and the widths (in pixels) of the circumscribed rectangles of all valid dots are summed to obtain the total width of the circumscribed rectangles. The total number of all valid dots is calculated, and the dot diameter in the X direction is obtained based on the total width of the circumscribed rectangles and the total number of valid dots. The X-axis resolution of the main camera is calculated according to the dot diameter in the X direction and the actual diameter of the valid dot. The heights (in pixels) of the circumscribed rectangles of all valid dots are summed to obtain the total height of the circumscribed rectangles. The dot diameter in the Y direction is obtained based on the total height of the circumscribed rectangles and the total number of valid dots. The Y-axis resolution of the main camera is calculated according to the dot diameter in the Y direction and the actual diameter of the valid dot, so as to obtain the true resolution of the main camera.

[0052] Exemplarily, in some embodiments, preprocessing is performed on the main camera image, which specifically includes but is not limited to the following steps S210 - S220: S210: Convert the input main camera image into a single-channel grayscale image.

[0053] Specifically, preprocessing is performed on the main camera image input to the electronic device. That is, using an image processing library (such as OpenCV, etc.), the color information of the main camera image is removed, and the number of channels of the main camera image is converted from three channels (such as RGB channels) to a single channel to obtain a single-channel grayscale image. Each pixel in the single-channel grayscale image has only one grayscale value.

[0054] S220: Perform threshold segmentation on the converted main camera image to convert the main camera image into two colors, black and white.

[0055] In the embodiments of the present invention, a threshold segmentation algorithm is used to perform threshold segmentation on the converted main camera image (i.e., a single-channel grayscale image), so that the main camera image is converted into two colors, black and white. That is, the converted main camera image is binarized, and the single-channel grayscale image is converted into a pure black-and-white image (i.e., a binary image including only black and white), obtaining the preprocessed main camera image, and the preprocessed main camera image is in the form of a binary image.

[0056] It can be understood that the threshold segmentation algorithm includes but is not limited to: binary threshold segmentation algorithm, OTSU algorithm, and TOZERO algorithm. In some embodiments, Gaussian blur is used to remove noise points in the binary image, smooth the binary image, and make the features in the binary image more obvious, so as to obtain a preprocessed main camera image with higher quality and reduce interference to feature extraction.

[0057] In some embodiments, after preprocessing, the valid dots and dot information of the valid dots in the main camera image are identified, which specifically includes but is not limited to the following steps S230 - S240: S230: After preprocessing, perform connected component analysis on the main camera image, find the median of the areas of all connected components, and set an area difference threshold according to the requirement of identifying dots.

[0058] S240: Traverse all connected components, calculate the four corner coordinates of the circumscribed rectangle of each connected component, and judge all connected components based on the area difference threshold and the four corner coordinates to identify all valid dots and dot information of the valid dots in the main camera image.

[0059] In the embodiments of the present invention, after preprocessing the main camera image, a connected component analysis algorithm is used to perform connected component analysis on the preprocessed main camera image, detect and identify all connected components in the preprocessed main camera image, and traverse all identified connected components to obtain connected component information. Among them, the connected component information includes the total number of connected components, the centroid coordinates, area, the upper left corner coordinates of the circumscribed rectangle, width, and height of each connected component. It can be understood that the connected component analysis algorithm includes but is not limited to: Spaghetti algorithm, SAUF algorithm, and BBDT algorithm.

[0060] In some embodiments, the area of a connected component is the total number of pixels within the connected component. The cv2.contourArea() function in the OpenCV library or other similar methods are used to calculate the total number of pixels of the connected component to obtain the area of the connected component. According to the area of each connected component, the areas of all connected components are arranged in ascending order to obtain an area arrangement sequence, and the area value located in the middle position is selected as the median of the areas. If there are two area values in the middle position, the median of the areas is taken as the average of the two area values in the middle position to obtain the median of the areas of all connected components.

[0061] In the embodiments of the present invention, an area difference threshold is set according to the requirements for identifying the dot (such as the number of dots, the size of the dots, and the median of the area, etc.), where the area difference threshold is used to identify and determine whether the identified connected component is qualified. In some embodiments, the area difference threshold can be set according to a preset multiple of the median of the area of the connected component, so as to distinguish effective and ineffective connected components. For example, the area difference threshold is 1.2 times, 1.5 times or other appropriate multiples of the median of the area. Those skilled in the art can set the area difference threshold according to actual needs, and the embodiments of the present invention do not make any limitation thereto.

[0062] In some embodiments, the centroid coordinates of the connected component are the geometric center of the connected component. The cv2.moments() function in the OpenCV library can be used to calculate the moments of the connected component to obtain the centroid coordinates of the connected component.

[0063] In some embodiments, the bounding rectangle of the connected component is represented by the upper left corner coordinates and the width and height of the rectangle. The cv2.boundingRect() function of the OpenCV library is used to obtain the coordinates and dimensions (including the width and height) of the bounding rectangle of the connected component, so as to obtain the bounding rectangle of the connected component.

[0064] In the embodiments of the present invention, the four corner coordinates of the bounding rectangle include the upper left corner coordinates, the upper right corner coordinates, the lower left corner coordinates and the lower right corner coordinates of the bounding rectangle.

[0065] Specifically, after traversing all the identified connected components and obtaining the upper-left corner coordinates, width, and height of the bounding rectangle of each connected component, for each connected component, based on the upper-left corner coordinates, width, and height of the bounding rectangle of each connected component, calculate the upper-right corner coordinates, lower-left corner coordinates, and lower-right corner coordinates of the bounding rectangle of each connected component to obtain the four-corner coordinates of the bounding rectangle of each connected component. It can be understood that the ordinate of the upper-right corner coordinates of the bounding rectangle is the same as that of the upper-left corner coordinates. Therefore, when calculating the upper-right corner coordinates, add the abscissa of the upper-left corner coordinates to the width of the bounding rectangle to obtain the abscissa of the upper-right corner coordinates, thereby obtaining the upper-right corner coordinates. Similarly, the abscissa of the lower-left corner coordinates of the bounding rectangle is the same as that of the upper-left corner coordinates. Therefore, when calculating the lower-left corner coordinates, subtract the height of the bounding rectangle from the ordinate of the upper-left corner coordinates to obtain the ordinate of the lower-left corner coordinates, thereby obtaining the lower-left corner coordinates. Similarly, after calculating the upper-right corner coordinates and / or lower-left corner coordinates, the abscissa of the lower-right corner coordinates of the bounding rectangle is the same as that of the upper-right corner coordinates. Therefore, when calculating the lower-right corner coordinates, subtract the height of the bounding rectangle from the ordinate of the upper-right corner coordinates to obtain the ordinate of the lower-right corner coordinates, thereby obtaining the lower-right corner coordinates. Or the ordinate of the lower-right corner coordinates of the bounding rectangle is the same as that of the lower-left corner coordinates. Therefore, when calculating the lower-right corner coordinates, add the width of the bounding rectangle to the abscissa of the lower-left corner coordinates to obtain the abscissa of the lower-right corner coordinates, thereby obtaining the lower-right corner coordinates.

[0066] Compare the upper-left corner coordinates, upper-right corner coordinates, lower-left corner coordinates, and lower-right corner coordinates of the bounding rectangle of each connected component with the edges of the main camera image respectively. If any of the upper-left corner coordinates, upper-right corner coordinates, lower-left corner coordinates, and lower-right corner coordinates of the bounding rectangle exceeds the edge of the main camera image or is located on the edge of the main camera image, then this connected component is an incomplete connected component, and the incomplete connected component is an invalid dot. Filter out this incomplete connected component.

[0067] After filtering out the incomplete connected components, filter the remaining complete connected components according to the area difference threshold to obtain qualified connected components. Among them, the qualified connected components are valid dots, and the connected component information of the qualified connected components is the dot information of the valid dots.

[0068] Specifically, for each complete connected component, subtract the area of each complete connected component from the median of the areas to obtain an area difference. Compare the area difference with the area difference threshold. If the area difference is less than or equal to the area difference threshold, then the connected component is a qualified connected component (i.e., a valid dot). If the area difference is greater than the area difference threshold, then the connected component is an unqualified connected component (i.e., an invalid dot). Use the connected component information of the qualified connected components obtained by filtering as the dot information of the valid dots to obtain the dot information of the valid dots, that is, identify all the valid dots and the dot information of the valid dots in the main camera image.

[0069] Exemplarily, in some embodiments, calculating the true resolution of the main camera specifically includes, but is not limited to, the following steps S250 - S280: S250: Sum the widths of the circumscribed rectangles of all valid dots to obtain the sum of the circumscribed widths.

[0070] S260: Obtain the dot diameter in the X - direction based on the sum of the circumscribed widths and the total number of valid dots, and calculate the X - axis resolution of the main camera according to the dot diameter in the X - direction and the actual diameter.

[0071] S270: Sum the heights of the circumscribed rectangles of all valid dots to obtain the sum of the circumscribed heights.

[0072] S280: Obtain the dot diameter in the Y - direction based on the sum of the circumscribed heights and the total number of valid dots, and calculate the Y - axis resolution of the main camera according to the dot diameter in the Y - direction and the actual diameter.

[0073] In the embodiments of the present invention, the dot information includes the actual diameter of the valid dots, the width and height of the circumscribed rectangle, and the true resolution of the main camera includes the X - axis resolution and the Y - axis resolution of the main camera.

[0074] In some embodiments, the true resolution of the main camera is calculated based on the dot information, that is, sum the widths (in pixels) of the circumscribed rectangles of all valid dots to obtain the total sum of the circumscribed rectangle widths, denoted as the sum of the circumscribed widths. Calculate the total number of valid dots, and divide the sum of the circumscribed widths by the total number of valid dots to obtain the dot diameter in the X - direction (i.e., the diameter of the valid dots in the X - axis direction, in pixels). Divide the dot diameter in the X - direction by the actual diameter of the valid dots to obtain the X - axis resolution of the main camera.

[0075] Sum the heights (in pixels) of the circumscribed rectangles of all valid dots to obtain the total sum of the circumscribed rectangle heights, denoted as the sum of the circumscribed heights. Divide the sum of the circumscribed heights by the total number of valid dots to obtain the dot diameter in the Y - direction (i.e., the diameter of the valid dots in the Y - axis direction, in pixels). Divide the dot diameter in the Y - direction by the actual diameter of the valid dots to obtain the Y - axis resolution of the main camera, thereby obtaining the true resolution of the main camera.

[0076] S300: Identify the positioning points of the main camera image based on the valid dots in the main camera image.

[0077] In the embodiments of the present invention, the main camera image includes positioning points and calibration points. After removing the unqualified positioning points and calibration points, qualified positioning points and calibration points (i.e., valid dots) are obtained. According to the description of the foregoing embodiments, the shapes of the positioning points and calibration points are different, where the positioning points are annular and the calibration points are circular. Therefore, in the pre - processed main camera image, the colors and shapes of the positioning points and calibration points are different. For example, please refer toFigure 7a , Figure 7a In the preprocessed main camera image shown, the positioning point is a white ring and the calibration point is a black dot.

[0078] Specifically, after identifying the valid dots in the main camera image, the valid dots in the main camera image are identified according to the position, shape, and color of each valid dot in the image coordinate system, the positioning point of the main camera image is identified, and the X-axis coordinate and Y-axis coordinate of the positioning point of the main camera image are obtained.

[0079] Exemplarily, in some embodiments, identifying the positioning point of the main camera image based on the valid dots in the main camera image specifically includes, but is not limited to, the following steps S310-S350: S310: Set the foreground of the preprocessed main camera image to white, and perform connected component analysis on the main camera image again to obtain the upper left coordinate of the circumscribed rectangle of the connected component, the width and height of the circumscribed rectangle.

[0080] S320: Calculate the upper right coordinate, lower right coordinate, and lower left coordinate of the circumscribed rectangle of the connected component through the upper left coordinate, width, and height of the circumscribed rectangle of the first connected component.

[0081] S330: Determine whether any one of the upper left coordinate, lower left coordinate, upper right coordinate, and lower right coordinate of the circumscribed rectangle of the connected component touches the edge of the main camera image.

[0082] S340: When the judgment result is yes, it means that the second connected component is the positioning point.

[0083] S350: When the judgment result is no, it means that the first connected component is the positioning point.

[0084] In the embodiment of the present invention, the background and the positioning point in the preprocessed main camera image are white, and the calibration point is a black dot. If the connected component analysis algorithm used considers the foreground to be black, then the foreground of the preprocessed main camera image needs to be set to white, and the preprocessed main camera image is subjected to color inversion processing to convert the foreground and the calibration point to white and the positioning point to black. If the connected component analysis algorithm used considers the foreground to be white, then the preprocessed main camera image does not need to be subjected to color inversion processing. The connected component analysis algorithm used in the embodiment of the present invention considers the foreground to be black. Therefore, the preprocessed main camera image needs to be subjected to color inversion processing to convert the foreground and the calibration point in the preprocessed main camera image to white and the positioning point to black. At this time, there are only two connected components in the main camera image: the positioning point and the background. Please refer to Figure 7b , Figure 7b shows Figure 4a the main camera image obtained after preprocessing and color inversion processing of the shown main camera image,Figure 7b The foreground of the main camera image shown is white.

[0085] Specifically, the connected component analysis algorithm is used to perform connected component analysis on the preprocessed and inverted main camera image to obtain the upper left corner coordinates of the circumscribed rectangle of the connected component, the width and height of the circumscribed rectangle.

[0086] It can be understood that there are only two connected components in the preprocessed and inverted main camera image, one connected component is the positioning point and the other connected component is the background area. Therefore, the positioning point can be identified and determined according to the connected component information of these two connected components. Specifically, using the upper left corner coordinates, width and height of the circumscribed rectangle of the first connected component, calculate the upper right corner coordinates, lower right corner coordinates and lower left corner coordinates of the circumscribed rectangle of this connected component.

[0087] After calculating the upper left corner coordinates, lower left corner coordinates, upper right corner coordinates and lower right corner coordinates of the circumscribed rectangle of the first connected component, compare the upper left corner coordinates, lower left corner coordinates, upper right corner coordinates and lower right corner coordinates of the circumscribed rectangle of this connected component with the edge of the main camera image respectively to determine whether any one of the coordinates touches the edge of the main camera image. If any one of the upper left corner coordinates, upper right corner coordinates, lower left corner coordinates and lower right corner coordinates of the circumscribed rectangle of the first connected component touches the edge of the main camera image, that is, when the judgment result is yes, then the first connected component is not the positioning point and the second connected component is the positioning point, so as to identify and determine the positioning point of the main camera image. If none of the upper left corner coordinates, upper right corner coordinates, lower left corner coordinates and lower right corner coordinates of the circumscribed rectangle of the first connected component touches the edge of the main camera image, that is, when the judgment result is no, then the first connected component is the positioning point and the second connected component is not the positioning point, so as to identify and determine the positioning point of the main camera image and obtain the X-axis coordinate and Y-axis coordinate of the positioning point of the main camera image.

[0088] S400: Calculate and obtain the first relative position according to the true resolution of the main camera.

[0089] In this step, the first relative position is the relative position between the positioning point of the main camera image and the center of the main camera FOV. In the embodiment of the present invention, the relative distance on the X-axis and the relative distance on the Y-axis in physical world units are used to represent the first relative position. The first relative position includes the relative distance on the X-axis in physical world units and the relative distance on the Y-axis in physical world units of the positioning point of the main camera image relative to the center of the main camera FOV.

[0090] In the embodiment of the present invention, before using the main camera to capture the main camera image, set the size (including width and height) of the picture captured by the main camera. In this way, after the main camera captures the main camera image, the width and height of the main camera image can be known.

[0091] Specifically, after obtaining the main camera image by using the main camera, read the size of the main camera image to obtain the width and height of the main camera image. Then, based on the width and height of the main camera image, calculate the central coordinates of the main camera FOV (Field Of View), and the central coordinates of the main camera FOV include the X-axis coordinate and the Y-axis coordinate. According to the X-axis coordinate and Y-axis coordinate of the positioning point in the main camera image and the X-axis coordinate and Y-axis coordinate of the central coordinates of the main camera FOV, calculate the relative distance of the X-axis in pixel units and the relative distance of the Y-axis in pixel units of the positioning point relative to the central of the main camera FOV. Calculate the relative distance of the X-axis in the physical world unit of the positioning point relative to the central of the main camera FOV according to the relative distance of the X-axis in pixel units and the X-axis resolution of the main camera, and calculate the relative distance of the Y-axis in the physical world unit of the positioning point relative to the central of the main camera FOV according to the relative distance of the Y-axis in pixel units and the Y-axis resolution of the main camera. Combine the relative distance of the X-axis in the physical world unit and the relative distance of the Y-axis in the physical world unit of the positioning point relative to the central of the main camera FOV, and then obtain the relative position (i.e., the first relative position) between the positioning point in the main camera image and the central of the main camera FOV.

[0092] It can be understood that the relative distance of the X-axis in pixel units refers to the X-axis distance between two pixel points (i.e., the positioning point and the central of the main camera FOV) in the pixel coordinate system, and the relative distance of the Y-axis in pixel units refers to the Y-axis distance between two pixel points (i.e., the positioning point and the central of the main camera FOV) in the pixel coordinate system.

[0093] Among them, the relative distance of the X-axis in the physical world unit refers to the X-axis distance between two points (i.e., the positioning point and the central of the main camera FOV) in the world coordinate system, and the relative distance of the Y-axis in the physical world unit refers to the Y-axis distance between two points (i.e., the positioning point and the central of the main camera FOV) in the world coordinate system. Among them, the physical world unit is a metric length unit, such as millimeters, centimeters, micrometers, etc.

[0094] In some embodiments, the first relative position is calculated and obtained according to the true resolution of the main camera, specifically including but not limited to the following steps S410 - S460: S410: Obtain the central coordinates of the main camera FOV according to the width and height of the main camera image.

[0095] In the embodiments of the present invention, divide the width of the main camera image by 2 to obtain the X-axis coordinate of the central of the main camera FOV, divide the height of the main camera image by 2 to obtain the Y-axis coordinate of the central of the main camera FOV, and combine the X-axis coordinate and the Y-axis coordinate of the central of the main camera FOV to obtain the central coordinates of the main camera FOV.

[0096] S420: Obtain the relative distance of the X-axis in pixel units of the positioning point relative to the central of the FOV according to the X-axis coordinate of the positioning point and the X-axis coordinate of the central coordinates of the main camera FOV.

[0097] S430: Obtain the relative Y-axis distance in pixel units of the positioning point with respect to the FOV center based on the Y-axis coordinate of the positioning point and the Y-axis coordinate of the center of the main camera's FOV.

[0098] Exemplarily, subtract the X-axis coordinate of the center of the main camera's FOV from the X-axis coordinate of the positioning point in the main camera image to obtain the relative X-axis distance in pixel units of the positioning point in the main camera image with respect to the center of the main camera's FOV. Also, subtract the Y-axis coordinate of the center of the main camera's FOV from the Y-axis coordinate of the positioning point in the main camera image to obtain the relative Y-axis distance in pixel units of the positioning point in the main camera image with respect to the center of the main camera's FOV.

[0099] S440: Obtain the relative X-axis distance in physical world units of the positioning point with respect to the FOV center based on the relative X-axis distance in pixel units and the X-axis resolution of the main camera.

[0100] S450: Obtain the relative Y-axis distance in physical world units of the positioning point with respect to the FOV center based on the relative Y-axis distance in pixel units and the Y-axis resolution of the main camera.

[0101] In an embodiment of the present invention, multiply the relative X-axis distance in pixel units of the positioning point in the main camera image with respect to the center of the main camera's FOV by the X-axis resolution of the main camera to obtain the relative X-axis distance in physical world units of the positioning point in the main camera image with respect to the center of the main camera's FOV. Also, multiply the relative Y-axis distance in pixel units of the positioning point in the main camera image with respect to the center of the main camera's FOV by the Y-axis resolution of the main camera to obtain the relative Y-axis distance in physical world units of the positioning point in the main camera image with respect to the center of the main camera's FOV.

[0102] S460: Obtain the first relative position based on the relative X-axis distance in physical world units and the relative Y-axis distance in physical world units.

[0103] In an embodiment of the present invention, combine the relative X-axis distance in physical world units and the relative Y-axis distance in physical world units of the positioning point in the main camera image with respect to the center of the main camera's FOV to obtain the relative position of the positioning point in the main camera image and the center of the main camera's FOV, that is, obtain the first relative position.

[0104] S500: Set the processing order of the side camera images according to requirements, and process each side camera image according to the above main camera image processing steps based on the processing order to obtain the positioning point of the side camera image and obtain the second relative position.

[0105] It can be understood that due to the distortion of the camera field of view in the side camera image, before processing each side camera image according to the above main camera image processing steps in the processing order, it is necessary to correct the camera field of view in the side camera image. In the embodiments of the present invention, the camera field of view correction method recorded in the invention patent with the patent number ZL202411764896.0 previously applied for and authorized by the inventors of the present invention is used to correct the camera field of view in the side camera image, and the side camera image after the camera field of view correction is obtained. Then, based on the processing order, each side camera image after the camera field of view correction is processed according to the above main camera image processing steps. Please refer to Figures 8a to 8d , Figures 8a to 8d respectively show Figures 4b to 4e the side camera image obtained after correcting the camera field of view of the side camera image shown Figures 8a to 8d The center of the camera field of view is marked on the side camera image after the camera field of view correction shown, where the center of the camera field of view is at the position where the colored lines cross as shown in Figures 8a to 8d shown (i.e., Figure 8a the orange line shown in Figure 8b the red line shown in Figure 8c the blue line shown in Figure 8d and the green line shown).

[0106] In the embodiments of the present invention, the processing order of the side camera images can be set according to actual needs. For example, as shown in the side camera image Figures 4b to 4e shown, the processing order can be set as Figure 4b → Figure 4c → Figure 4d → Figure 4e , and of course, it can also be set to any other suitable processing order, and the embodiments of the present invention do not make any limitations on this.

[0107] Exemplarily, the second relative position is the relative position between the positioning point of the side camera image and the center of the side camera FOV. In the embodiments of the present invention, the relative distance on the X-axis and the relative distance on the Y-axis in physical world units are used to characterize the second relative position. The second relative position includes the relative distance on the X-axis and the relative distance on the Y-axis of the positioning point of the side camera image relative to the center of the side camera FOV (both in physical world units).

[0108] Among them, each side camera image is processed according to the processing steps of the main camera image as described above in the order of processing to obtain the positioning points of the side camera image and obtain the second relative position. The preprocessing of the side camera image, the identification of the valid dots and the dot information of the valid dots of the side camera image after preprocessing, and the calculation of the true resolution of the side camera are performed with reference to step S200 and its refinement steps. The positioning points of the side camera image are identified based on the valid dots of the side camera image, and are performed with reference to step S300 and its refinement steps. The second relative position is calculated and obtained according to the true resolution of the side camera, and is performed with reference to step S400 and its refinement steps. Finally, a second relative position is obtained for each side image, and a plurality of different second relative positions are obtained.

[0109] It should be noted that after preprocessing the side camera images, each preprocessed side camera image is similar to the preprocessed main camera image. Similarly, after performing color inversion processing on each preprocessed side camera image, each preprocessed and color-inverted side camera image is similar to the preprocessed and color-inverted main camera image. For the sake of brevity, the embodiments of the present invention do not show each preprocessed side camera image and each preprocessed and color-inverted side camera image.

[0110] S600: Based on the first relative position and a plurality of different second relative positions, calculate and obtain the mapping relationship between the main camera FOV and a plurality of different side camera FOVs.

[0111] Specifically, the plurality of different second relative positions include the relative positions of the positioning points of each side camera image and the centers of each side camera FOV. The mapping relationship refers to a transformation relationship that converts / maps a certain point in the side camera image to the position in the main camera image, and can be represented in the form of a projection matrix, a homography matrix, or an affine transformation matrix, etc.

[0112] Specifically, according to the first relative position and each second relative position, the relative orientations of the main camera and each side camera are obtained. Then, according to the relative orientations of the main camera and each side camera, the relative X-axis distance and relative Y-axis distance (both in physical world units) between the main camera and each side camera are calculated. Next, according to the relative X-axis distance in physical world units and the relative Y-axis distance in physical world units between the main camera and each side camera, the mapping relationship between the main camera FOV and each side camera FOV is calculated, so as to obtain the mapping relationship between the main camera FOV and a plurality of different side camera FOVs. The mapping relationship includes the relative X-axis distance in physical world units and the relative Y-axis distance in physical world units of the side camera FOV center relative to the main camera FOV center.

[0113] Exemplarily, in some embodiments, based on the first relative position and multiple different second relative positions, the mapping relationship between the main camera FOV and multiple different side camera FOVs is calculated and obtained, specifically including but not limited to the following steps S610 - S630: S610: Compare the first relative position with the second relative position of the candidate side camera to obtain the relative orientation between the main camera and the candidate side camera.

[0114] In this embodiment, the candidate side camera is any one of the multiple side cameras. Each candidate side camera corresponds to a second relative position, and the main camera corresponds to the first relative position.

[0115] Specifically, compare the first relative position with the second relative position corresponding to the candidate side camera to obtain the relative orientation between the main camera and the candidate side camera. The relative orientation includes that the candidate side camera is located on the right side, left side, upper side, and lower side of the main camera.

[0116] It can be understood that Figure 2a in the schematic diagram of the positional relationship between the main camera and the other four side cameras, the main camera and the other four side cameras are observed from the side, Figure 2a and the positional relationship between the main camera and the other four side cameras shown is not the relative orientation between the main camera and the candidate side camera described in this embodiment. The relative orientation between the main camera and the candidate side camera described in this embodiment is the relative orientation between each side camera and the main camera from the top-down perspective. Please refer to Figure 2b , Figure 2b which schematically shows the relative orientation between each side camera and the main camera from the top-down perspective. Figure 2b In, the first side camera 301 is located on the upper side of the main camera 300, the second side camera 302 is located on the left side of the main camera 300, the third side camera 303 is located on the lower side of the main camera 300, and the fourth side camera 304 is located on the right side of the main camera 300. It should be noted that Figure 2b the relative orientation between the main camera and the candidate side camera shown does not impose any limitation on the relative orientation between the main camera and the candidate side camera in other embodiments.

[0117] S620: Calculate the relative distance of the X-axis and the relative distance of the Y-axis in physical world units between the main camera and the candidate side camera according to the relative orientation to establish the mapping relationship between the main camera FOV and the candidate side camera FOV.

[0118] Specifically, if the candidate side camera is located on the right side of the main camera, subtract the relative distance of the positioning point of the main camera image in the first relative position from the center of the main camera FOV along the X-axis in physical world units from the relative distance of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera from the center of the side camera FOV along the Y-axis in physical world units, to obtain the relative distance of the center of the candidate side camera FOV from the center of the main camera FOV along the X-axis in physical world units, that is, obtain the relative distance of the main camera and the candidate side camera along the X-axis in physical world units. Also, add the relative distance of the positioning point of the main camera image in the first relative position from the center of the main camera FOV along the Y-axis in physical world units to the relative distance of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera from the center of the side camera FOV along the X-axis in physical world units, to obtain the relative distance of the center of the candidate side camera FOV from the center of the main camera FOV along the Y-axis in physical world units, that is, obtain the relative distance of the main camera and the candidate side camera along the Y-axis in physical world units.

[0119] If the candidate side camera is located on the left side of the main camera, add the relative distance of the positioning point of the main camera image in the first relative position from the center of the main camera FOV along the X-axis in physical world units to the relative distance of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera from the center of the side camera FOV along the Y-axis in physical world units, to obtain the relative distance of the center of the candidate side camera FOV from the center of the main camera FOV along the X-axis in physical world units, that is, obtain the relative distance of the main camera and the candidate side camera along the X-axis in physical world units. Also, subtract the relative distance of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera from the center of the side camera FOV along the X-axis in physical world units from the relative distance of the positioning point of the main camera image in the first relative position from the center of the main camera FOV along the Y-axis in physical world units, to obtain the relative distance of the center of the candidate side camera FOV from the center of the main camera FOV along the Y-axis in physical world units, that is, obtain the relative distance of the main camera and the candidate side camera along the Y-axis in physical world units.

[0120] If the candidate side camera is located above the main camera, add the relative X-axis distance in physical world units of the positioning point of the main camera image in the first relative position with respect to the center of the main camera FOV to the relative X-axis distance in physical world units of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera with respect to the center of the side camera FOV, to obtain the relative X-axis distance in physical world units of the center of the candidate side camera FOV with respect to the center of the main camera FOV, that is, obtain the relative X-axis distance in physical world units between the main camera and the candidate side camera. Also, add the relative Y-axis distance in physical world units of the positioning point of the main camera image in the first relative position with respect to the center of the main camera FOV to the relative Y-axis distance in physical world units of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera with respect to the center of the side camera FOV, to obtain the relative Y-axis distance in physical world units of the center of the candidate side camera FOV with respect to the center of the main camera FOV, that is, obtain the relative Y-axis distance in physical world units between the main camera and the candidate side camera.

[0121] If the candidate side camera is located below the main camera, subtract the relative X-axis distance in physical world units of the positioning point of the main camera image in the first relative position with respect to the center of the main camera FOV from the relative X-axis distance in physical world units of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera with respect to the center of the side camera FOV, to obtain the relative X-axis distance in physical world units of the center of the candidate side camera FOV with respect to the center of the main camera FOV, that is, obtain the relative X-axis distance in physical world units between the main camera and the candidate side camera. Also, subtract the relative Y-axis distance in physical world units of the positioning point of the main camera image in the first relative position with respect to the center of the main camera FOV from the relative Y-axis distance in physical world units of the positioning point of the side camera image in the second relative position corresponding to the candidate side camera with respect to the center of the side camera FOV, to obtain the relative Y-axis distance in physical world units of the center of the candidate side camera FOV with respect to the center of the main camera FOV, that is, obtain the relative Y-axis distance in physical world units between the main camera and the candidate side camera.

[0122] Combine the relative X-axis distance and the relative Y-axis distance in physical world units of the center of the candidate side camera FOV with respect to the center of the main camera FOV to establish a mapping relationship between the main camera FOV and the candidate side camera FOV. The mapping relationship is used to "reproject" the content of the side camera image into the main camera image to achieve field of view fusion.

[0123] S630: Establish the mapping relationship between the main camera FOV and each side camera FOV in sequence according to the processing order.

[0124] Specifically, according to the processing sequence of the side camera images, the mapping relationships between the main camera FOV and each side camera FOV are established in sequence, and the mapping relationships between the main camera FOV and multiple different side camera FOVs are obtained.

[0125] S700: Perform the field of view fusion of the main camera and the side cameras based on the mapping relationships.

[0126] Specifically, after obtaining the mapping relationships between the main camera FOV and each side camera FOV, for each side camera, according to the mapping relationship between the main camera FOV and this side camera FOV, all pixel coordinates in the side camera image captured by this side camera are converted to the coordinate system of the main camera, and the mapped image content is inserted into the corresponding position of the main camera image. Finally, all the side camera images captured by the side cameras are fused into the main camera image to obtain a fused image, thereby realizing the field of view fusion of the main camera and each side camera. Please refer to Figure 9a and Figure 9b , Figure 9a shows Figure 4a the main camera image shown in Figures 8a to 8d and the schematic diagram before the fusion of the side camera image shown in Figure 9b shows Figure 4a the main camera image shown in Figures 8a to 8d and the schematic diagram after the fusion of the side camera image shown in Figure 9a and Figure 9b both show the center of the camera field of view, where the center of the camera field of view is like Figure 9a and Figure 9b the intersecting gray lines and colored lines shown.

[0127] Exemplarily, in some embodiments, before performing the field of view fusion of the main camera and the side cameras based on the mapping relationships, the multi-camera field of view fusion method further includes but is not limited to the following steps S701 - S702: S701: Compare the widths and heights of multiple side camera images, and find the minimum width and minimum height among all side camera images.

[0128] Exemplarily, obtain the widths and heights of each side camera image, compare the widths of each side camera image, and find the minimum width among all the widths of the side camera images. Compare the heights of each side camera image, and find the minimum height among all the heights of the side camera images.

[0129] S702: Crop the width of each side camera image to the minimum width, and crop the height to the minimum height.

[0130] Specifically, after obtaining the minimum width and the minimum height, center-crop the width of each side camera image to the minimum width, and center-crop the height of each side camera image to the minimum height, to obtain the cropped side camera images. Then, use the cropped side camera images as the side camera images to be fused, and fuse each side camera image to be fused with the main camera image based on the mapping relationship, that is, perform the field of view fusion of the main camera and the side cameras according to the mapping relationship, to obtain the fused image.

[0131] In summary, the multi-camera field of view fusion method provided by the embodiments of the present invention obtains the main camera image and multiple side camera images by using the main camera and multiple side cameras to photograph the same calibration board, preprocesses the main camera image and the multiple side camera images, and identifies the valid dots and the dot information of the valid dots in the preprocessed main camera image and multiple side camera images. Based on the valid dots in the main camera image and the multiple side camera images, identify the positioning points in the main camera image and the multiple side camera images, calculate the true resolutions of the main camera and the multiple side cameras, calculate and obtain the first relative position and multiple second relative positions according to the calculated true resolutions of the main camera and the multiple side cameras, calculate the mapping relationship between the main camera FOV and the multiple side camera FOV according to the first relative position and the multiple second relative positions, and perform the field of view fusion of the main camera and the multiple side cameras based on the mapping relationship. In this way, the relative positions of the main camera and each side camera can be determined, the field of view fusion of the main camera and the side cameras can be realized, so that the detection object outside the field of view of the detection main camera can be clearly identified, the detection range can be increased, the detection efficiency can be improved, and the user experience can be enhanced.

[0132] The embodiments of the present invention provide a computer-readable storage medium, on which computer program instructions executable by a processor are stored. When the computer program instructions are executed by the processor, the computer is caused to execute any one of the multi-camera field of view fusion methods provided by the embodiments of the present invention, or execute the steps in any one of the implementation manners of any one of the multi-camera field of view fusion methods provided by the embodiments of the present invention.

[0133] Those skilled in the art can understand that the embodiments provided by the present invention are only illustrative. The writing order of the steps in the method of the embodiments does not mean a strict execution order and does not constitute any limitation on the implementation process. It can be adjusted, combined, and deleted according to actual needs. The modules or sub-modules, units or sub-units, etc. in the device or system of the embodiments can be combined, divided, and deleted according to actual needs. For example, the division of units is only a logical function division, and there may be other division methods in actual implementation. For another example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0134] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Those skilled in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program, and this computer program can be stored in a computer-readable storage medium. When the computer program is executed, it can include the processes of the above method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM), etc.

[0135] It should be noted that the above embodiments are for explaining the technical concept and characteristics of the present invention. The purpose is to enable those familiar with this technology to understand the content of the present invention and implement it accordingly, and it cannot be used to limit the scope of the patent protection of the present invention. Those of ordinary skill in the art can understand all or part of the processes of implementing the above embodiments, modify them according to the technical solutions recorded in the embodiments of the present invention, or perform equivalent replacements on some of the technical features. It can be understood that these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should be regarded as equal changes and modifications based on the embodiments of the present invention, and should all fall within the scope covered by the claims of the present invention.

Claims

1. A multi-camera field of view fusion method, characterized in that: include: The main camera and the multiple side camera images are acquired by shooting with the main camera and the multiple side camera images based on the same calibration plate; Main camera image processing: Preprocessing the main camera image, identifying valid dots and dot information of valid dots in the main camera image after the preprocessing, and calculating the real resolution of the main camera; Identify the positioning points of the main camera image based on the valid dots in the main camera image; Obtain a first relative position according to the actual resolution of the main camera, where the first relative position is the relative position between the positioning point of the main camera image and the center of the FOV of the main camera; Side camera image processing: The processing order of the side camera images is set according to the requirements, and each side camera image is processed according to the above main camera image processing steps based on the processing order to obtain the positioning point of the side camera image and obtain the second relative position, where the second relative position is the relative position between the positioning point of the side camera image and the center of the FOV of the side camera; Main side camera field of view fusion: Based on the first relative position and the multiple different second relative positions, a mapping relationship between the main camera FOV and the multiple different side camera FOVs is calculated and obtained; The field of view of the main camera and the side camera are fused based on the mapping relationship.

2. The method according to claim 1, characterized in that The preprocessing of the main camera image includes: Convert the input main camera image into a single-channel grayscale image; The converted main camera image is subjected to threshold segmentation so as to convert the main camera image into black and white.

3. The method according to claim 1, characterized in that The step of identifying valid dots and dot information of valid dots in the main camera image after preprocessing includes: After preprocessing, the main camera image is analyzed for connected domains, the median of the area of ​​all connected domains is found, and the area difference threshold is set according to the need to identify dots; All connected domains are traversed, the four-corner coordinates of the circumscribed rectangle of each connected domain are calculated, and all connected domains are judged based on the area difference threshold and the four-corner coordinates to identify all valid dots and dot information of valid dots in the main camera image.

4. The method according to claim 1, characterized in that: The dot information includes the actual diameter of the effective dot, the width and height of the circumscribed rectangle, and the calculation of the actual resolution of the main camera includes: Sum the widths of the circumscribed rectangles of all valid points to obtain the sum of the circumscribed widths; The diameter of the dots in the X direction is obtained based on the sum of the circumscribed widths and the total number of valid dots, and the X-axis resolution of the main camera is calculated based on the diameter of the dots in the X direction and the actual diameter; Sum the heights of all valid circle point circumscribed rectangles to obtain the sum of the circumscribed heights; The Y-axis diameter of the dots is obtained based on the sum of the circumscribed heights and the total number of valid dots, and the Y-axis resolution of the main camera is calculated based on the Y-axis diameter of the dots and the actual diameter. The actual resolution of the main camera includes the X-axis resolution and Y-axis resolution of the main camera.

5. The method according to claim 1, characterized in that The identifying the positioning point of the main camera image based on the valid circle point in the main camera image includes: The foreground of the preprocessed main camera image is set to white, and a connected domain analysis is performed on the main camera image to obtain the coordinates of the upper left corner of the connected domain circumscribed rectangle, the width and height of the circumscribed rectangle; Calculate the upper right corner coordinates, lower right corner coordinates and lower left corner coordinates of the circumscribed rectangle of the first connected domain through the upper left corner coordinates, width and height of the circumscribed rectangle of the first connected domain; Determine whether any of the upper left corner coordinates, lower left corner coordinates, upper right corner coordinates and lower right corner coordinates of the circumscribed rectangle of the connected domain touches the edge of the main camera image; When the judgment result is yes, it means that the second connected domain is the positioning point; When the judgment result is no, it means that the first connected domain is the positioning point.

6. The method according to claim 1, characterized in that The step of calculating and acquiring the first relative position according to the real resolution of the main camera includes: Get the main camera FOV center coordinates according to the width and height of the main camera image; Obtain the X-axis relative distance of the positioning point relative to the FOV center in pixel units according to the X-axis coordinate of the positioning point and the X-axis coordinate of the main camera FOV center coordinate; According to the Y-axis coordinate of the positioning point and the Y-axis coordinate of the main camera FOV center coordinate, the Y-axis relative distance of the positioning point relative to the pixel unit of the FOV center is obtained; Obtain the X-axis relative distance of the positioning point relative to the center of the FOV in physical world units based on the X-axis relative distance in pixel units and the X-axis resolution of the main camera; Get the Y-axis relative distance of the positioning point relative to the center of the FOV in physical world units based on the Y-axis relative distance in pixel units and the Y-axis resolution of the main camera; The first relative position is obtained based on the X-axis relative distance in physical world units and the Y-axis relative distance in physical world units.

7. The method according to claim 1, characterized in that The calculating and obtaining the mapping relationship between the main camera FOV and the multiple side camera FOVs based on the first relative position and the multiple different second relative positions includes: Comparing the first relative position with the second relative position of the candidate side camera to obtain the relative position of the main camera and the candidate side camera, where the candidate side camera is any one of the multiple side cameras; Calculate the relative X-axis distance and Y-axis distance of the physical world unit between the main camera and the candidate side camera according to the relative orientation to establish a mapping relationship between the main camera FOV and the candidate side camera FOV; The mapping relationship between the main camera FOV and each side camera FOV is established in sequence according to the processing order.

8. The method according to claim 1, characterized in that Before fusing the fields of view of the main camera and the side cameras based on the mapping relationship, the method further includes: Compare the widths and heights of multiple side camera images and find the minimum width and height among all side camera images; The width of each side camera image is cropped to the width minimum value, and the height is cropped to the height minimum value.

9. An electronic device, characterized in that: include: A processor and a memory communicatively connected to the processor; The memory stores computer program instructions executable by the processor, and the processor executes the computer program instructions to enable the electronic device to perform the multi-camera field of view fusion method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions executable by a processor, and the processor executes the computer program instructions to enable the computer to execute the multi-camera field of view fusion method according to any one of claims 1 to 8.

Images