Fisheye image correction method and its device, equipment and medium

Abstract

The present application relates to a fisheye image correction method and its device, equipment, and medium in the field of image processing, the method comprising: performing boundary fixing on the image instance segmentation result of the display fisheye image according to the boundary fixing rule, obtaining the corresponding regional boundary point information, determining the corresponding regional point information according to the regional boundary point information and the regional point mapping relationship, subdividing multiple target subdivided areas in the display fisheye image based on the regional boundary point information and the regional point information, applying the fisheye correction algorithm to perform image correction on the subdivided images corresponding to each target subdivided area, and obtaining the corresponding corrected subdivided images. The present application can efficiently and accurately correct the image area in the display fisheye image.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a fisheye image correction method and its corresponding device, computer equipment, and computer-readable storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the significant progress in deep learning algorithms and neural processing unit (NPU) chip technology, the application of artificial intelligence has gradually expanded to the consumer electronics market. In this field, fisheye image correction technology has become an important application technology. Fisheye cameras are widely used in panoramic monitoring, vehicle-mounted surround view systems and other fields because they can provide a wide field of view. However, due to the lens characteristics of fisheye cameras, the captured images are often accompanied by significant distortion, which limits their effectiveness in certain applications. Therefore, developing effective fisheye image correction methods to improve image quality and user experience has become a key requirement for technological development.

[0003] Traditional fisheye image correction methods often rely on complex mathematical models and algorithms. Although these methods can correct distortion to a certain extent, they have high computational costs and slow processing speeds, making it difficult to meet the needs of real-time processing.

[0004] In view of the shortcomings of traditional approaches, the applicant has been engaged in research in related fields for a long time and has taken a different approach to solve industry problems in the field of image processing. Summary of the invention

[0005] The primary purpose of the present application is to solve at least one of the above problems and to provide a fisheye image correction method and its corresponding device, computer equipment, and computer program product.

[0006] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0007] A fisheye image correction method provided for one of the purposes of this application comprises the following steps:

[0008] Obtain a fisheye image of the display, perform boundary fixing on the image instance segmentation result of the fisheye image of the display according to a preset boundary fixing rule, and obtain the boundary point position information of the area corresponding to the fisheye image of the display;

[0009] According to the regional boundary point information and the preset regional point mapping relationship, the regional point information corresponding to the display fisheye image is determined, and the display fisheye image is regionally subdivided based on the regional boundary point information and the regional point information to obtain multiple target subdivided areas;

[0010] A preset fisheye correction algorithm is applied to perform image correction on the subdivided images corresponding to the multiple target subdivided areas in the fisheye image of the display to obtain corresponding corrected subdivided images.

[0011] On the other hand, a fisheye image correction device is provided to meet one of the purposes of the present application, including a boundary point determination module, a fisheye image subdivision module and an image correction module, wherein the boundary point determination module obtains a display fisheye image, performs boundary fixing on an image instance segmentation result of the display fisheye image according to a preset boundary fixing rule, and obtains regional boundary point information corresponding to the display fisheye image; the fisheye image subdivision module is used to determine the regional point information corresponding to the display fisheye image according to the regional boundary point information and a preset regional point mapping relationship, and performs regional subdivision on the display fisheye image based on the regional boundary point information and the regional point information to obtain multiple target subdivision areas; the image correction module is used to apply a preset fisheye correction algorithm to perform image correction on the subdivided images corresponding to the multiple target subdivision areas in the display fisheye image to obtain corresponding corrected subdivided images.

[0012] On the other hand, a computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the fisheye image correction method described in the present application.

[0013] On the other hand, a computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any embodiment of the present application.

[0014] The technical solution of this application has many advantages, including but not limited to the following aspects:

[0015] This application performs image instance segmentation and then performs regular point positioning of the boundary of the display fisheye image on the corresponding results, thereby obtaining the regional boundary point information corresponding to each point surrounding the boundary, ensuring real-time and accurate capture of the display content of the display. Then, according to the point mapping relationship between the points on the boundary of the display fisheye image and the points in the image, the corresponding points in the display fisheye image are located, and the display fisheye image is divided based on these points to obtain multiple target subdivision areas and their subdivision images, and the visual distortion of each subdivision image is corrected. It can accurately, efficiently and finely divide the target subdivision areas and their subdivision images in the display fisheye image, and ensure the accuracy and reliability of the image content in the subdivision image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0017] Figure 1 A schematic diagram of the circuit principle of an atmosphere light control device according to a typical embodiment of the present invention;

[0018] Figure 2 A schematic diagram of a typical embodiment of the fisheye image correction method of the present application;

[0019] Figure 3 A schematic diagram of a subdivision of a display fisheye image as an example of the present application;

[0020] Figure 4 A schematic diagram of the position distribution information corresponding to the display predetermined point area where each predetermined point is located in the anchor frame area of ​​the example of the present application;

[0021] Figure 5 This is a principle block diagram of the fisheye image correction device of the present application;

[0022] Figure 6 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0023] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and cannot be interpreted as limiting the present invention.

[0024] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0025] Those skilled in the art will understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0026] In an exemplary embodiment of the present invention, the fisheye image correction method is implemented based on the atmosphere light control device 10, combined with Figure 1 The atmosphere light control device 10 includes a camera device 11, a control unit 12 and an atmosphere light device 13.

[0027] The camera device 11, the control unit 12 and the atmosphere light device 13 can be connected by wire, and the camera device 11, the control unit 12 and the atmosphere light device 13 can also be connected by a network. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network. In some embodiments, the camera device 11, the control unit 12 and the atmosphere light device 13 can also communicate and transmit through a specific communication protocol, and the communication protocol includes but is not limited to a BLE (Bluetooth low energy) protocol, a WLAN (Wireless Local Area Network) protocol, a Bluetooth protocol, a ZigBee protocol or a Wi Fi (Wireless Fidelity) protocol, etc. The network can also include network entities such as routers and gateways, which are not shown in the figure.

[0028] The control unit 12 is connected to the camera device 11 so that the control unit 12 can read the video stream generated by the camera device, or the camera device sends the video stream generated by the camera device to the control unit. The control unit is connected to the atmosphere light device 13 so that the control unit 12 can control the lighting effect of the atmosphere light device.

[0029] The camera device 11 may be a single fisheye camera, which takes pictures by using a short focal length and a 180-degree viewing angle.

[0030] The control unit 12 can adopt SOC (System on Chip), CPU (central processing unit), MPU (Microcontroller Unit), NPU (neural-network processing unit), etc., and can be set according to actual use needs, and this application does not impose any restrictions on this.

[0031] In one embodiment, the atmosphere light device 13 may be equipped with a WS2811 chip to receive and process the lighting effect control signal output by the corresponding control unit to emit light according to the lighting effect control signal.

[0032] In some embodiments, the atmosphere light device 13 may be a strip light, a bar light, a lamp post, or a flat lamp (containing a plurality of lamp beads).

[0033] In some embodiments, the atmosphere light device 13 may include one or more light units, which may be combined accordingly according to actual use requirements.

[0034] In some embodiments, each lighting unit may include one or more lamp beads, and the control unit 12 may control the brightness, color, etc. of the lamp beads of the lighting unit, and may also control the number of lights on, thereby achieving different lighting effects of the atmosphere lighting device 13.

[0035] In some implementations, the position and number of the atmosphere light devices 13 can be set according to actual use needs. According to the position of the atmosphere light devices 13, the length, height, and area of ​​the lighting effect can also be controlled.

[0036] See also Figure 2 A fisheye image correction method of the present application, in its typical embodiment, comprises the following steps:

[0037] Step S1100, obtaining a fisheye image of the display, and performing boundary fixing on the image instance segmentation result of the fisheye image of the display according to a preset boundary fixing rule to obtain the boundary point position information of the region corresponding to the fisheye image of the display;

[0038] The camera device in the atmosphere light control device is used to capture images within its visible range and generate a corresponding video stream. The control unit is connected to the camera device so that the control unit can read the video stream generated by the camera device when capturing the display. Therefore, it can be understood that when the camera device of the single fisheye camera captures the display, the control unit obtains the video stream generated by the camera device and then performs frame processing on it, that is, splits the video stream into multiple image frames, thereby obtaining image frames arranged in chronological order.

[0039] The frame processing can be implemented by ffmpeg audio and video processing tools, OpenCV library, etc. For example, the VideoCapture class of OpenCV can be used to read the video stream, and the read method can be used to read the image frames in the video stream frame by frame.

[0040] A preset image segmentation model is used to take the image frame as input to perform target detection and image instance segmentation on the image frame. The image segmentation model extracts image semantic features corresponding to the image frame by performing convolution operations on the image frame, including color, texture, boundary, deep complex and abstract semantic information of the image frame, etc., and generates a corresponding multi-scale feature map. The display fisheye image belonging to the imaging range of the display in the image frame is used as the target detection object, and multiple candidate frame areas are predicted on the feature map. The overlapping candidate frame areas are filtered through a non-maximum suppression algorithm to obtain the anchor frame area of ​​the target detection object in the frame. In addition, the image segmentation model classifies each pixel point in the image frame, constructs a pixel-level mask of the display fisheye image, thereby segmenting the display fisheye image, and uses the Sobel algorithm to perform boundary detection on the display fisheye image to determine the coordinate information of each boundary point of the boundary of the display fisheye image to constitute the boundary part point information.

[0041] The image segmentation model is pre-trained to a convergence state, and learns to determine the display fisheye image and its boundary point information that belong to the imaging range of the display in the input image, as well as the anchor frame area of ​​the display fisheye image in the frame. The image segmentation model is recommended to be selected from the YOLO series model such as YOLOV8, and other models that are also suitable for target detection and image instance segmentation can also be used, such as U-Net, DeepLab series, BiSeNet, etc., and those skilled in the art can select and implement as needed.

[0042] Further, the image segmentation model usually determines the boundary point information of the display fisheye image, and the corresponding boundary points cannot completely surround the display fisheye image, so a linear interpolation algorithm is used to linearly complement the boundary point information to obtain the complete boundary point information, so that the corresponding boundary points completely surround the display fisheye image, presenting the complete boundary of the display fisheye image. Further, the complete boundary point information is subjected to noise reduction processing to obtain the boundary frame point information. According to the preset subdivided area positioning feature, the position distribution information of the display predetermined point area where each predetermined point is located in the anchor frame area can be determined in the anchor frame area. Then, according to the horizontal coordinate range and the vertical coordinate range corresponding to each display predetermined point area, the coordinate information of each boundary point in the boundary frame point information is traversed to determine the display predetermined point area to which each boundary point belongs. Then, for each display predetermined point area, each boundary point belonging to the display predetermined point area is respectively used as a single area boundary point, and the coordinate information of these area boundary points is collected to form the predetermined area point information of the display predetermined point area. According to the preset regional fixed-point features and the respective predetermined regional point information, the coordinate information of the preset points in each display predetermined point area is determined to constitute the predetermined point information. The coordinate information includes the horizontal coordinate and the vertical coordinate corresponding to the corresponding predetermined point in the anchor frame area. For every three adjacent predetermined point information, the center point information corresponding to the two target predetermined point information in the three predetermined point information is adaptively distorted, and the center point information after adaptive distortion adjustment is obtained as the first regional subdivision point information. From all the predetermined point information, the second regional subdivision point information matching the preset regional subdivision point features is determined, and the regional boundary point information is constituted by all the second regional subdivision point information and all the first regional subdivision point information.

[0043] The camera device uses a single fisheye camera, and sets 9 predetermined points around the boundary of the display fisheye image, corresponding to the four corner points of the display frame (such as Figure 3 The middle number is 2, 4, 6, 8), a midpoint of the lower long side (such as Figure 3 5) and the midpoints of the two broad sides (such as Figure 3 3, 7), and the two points where the long side of the display intersects with the display fisheye image (such as Figure 3 Winning bid numbers 1, 9).

[0044] The position distribution information includes the horizontal coordinate range and the vertical coordinate range corresponding to the distribution of the corresponding display predetermined point area in the anchor frame area. Those skilled in the art can set the subdivided area positioning features according to prior knowledge or experimental data, or implement them according to the relevant disclosures in the subsequent embodiments. The regional fixed-point features include locating the coordinate distribution rules of the predetermined points in each display predetermined point area. Those skilled in the art can set the regional fixed-point features according to prior knowledge or experimental data, or implement them according to the relevant disclosures in the subsequent embodiments.

[0045] Step S1200, determining the point information in the region corresponding to the fisheye image of the display according to the region boundary point information and the preset region point mapping relationship, and performing region segmentation on the fisheye image of the display based on the region boundary point information and the region point information to obtain a plurality of target segmented regions;

[0046] The regional point mapping relationship includes the relative coordinate offset relationship between the coordinate information of the points in each region and the coordinate information of at least one point corresponding to the regional boundary point information, so that the coordinate information of the points in each region can be determined according to the coordinate information of at least one point in the regional boundary point corresponding to each relative coordinate offset relationship. The relative coordinate offset relationship includes the horizontal coordinate and the vertical coordinate of the corresponding point in the region relative to the horizontal coordinate and the vertical coordinate of at least one point corresponding to the regional boundary point information, and the corresponding mathematical expression of the horizontal value and the mathematical expression of the vertical value. The points in each region belong to the corner points of the corresponding region obtained by dividing the display fisheye image except the points corresponding to the regional boundary point information. A person skilled in the art may obtain the corresponding display fisheye image and its regional boundary information by using a camera device in advance according to the above-mentioned relevant disclosure, and then divide the display fisheye image into regions according to the regional boundary information of the display fisheye image by using a straight line according to their own needs, obtain all regions corresponding to the display fisheye image, and the coordinate information of each corner point of each region, and use the corner points of these different corner points except any one point corresponding to the regional boundary information as the regional points of the display fisheye image, and then determine at least one point corresponding to the regional boundary point position information connected to the point in the region for each point in the region, and according to the coordinate information of these points and the coordinate information of the point in the region, the coordinate information of the point in the region, the corresponding horizontal value mathematical expression and vertical value mathematical expression can be determined according to the coordinate information of these points. For example, the horizontal value mathematical expression is represented by the horizontal coordinate in the coordinate information of the point in the region, which is obtained by calculating the horizontal coordinate of at least one point relative to the point in the region multiplied by the respective weights and then adding the value obtained; the vertical value mathematical expression is represented by the vertical coordinate in the coordinate information of the point in the region, which is obtained by calculating the vertical coordinate of at least one point relative to the point in the region multiplied by the respective weights and then adding the value obtained.

[0047] According to the regional boundary point information and the regional point information, the corresponding points in the display fisheye image can be determined, and some or all of these points are connected by straight lines according to the preset point-to-point connection relationship set to complete the regional subdivision of the display fisheye image and obtain corresponding multiple regions. Further, multiple regions corresponding to the edges of the display frame are determined from these regions as target subdivision regions. The point-to-point connection relationship set includes each pair of two connected points, which can be pre-set by those skilled in the art according to their own needs.

[0048] The exemplary display fisheye image division effect is as follows Figure 3 As shown, Figure 3 The winning bids 18 to 23 are Figure 3 Points in each area of ​​the display fisheye image, Figure 3 The numbers 1 to 17 are the points corresponding to the regional boundary point information of the fisheye image of the display. The multiple target subdivision areas are: Figure 3 The numbers 2, 10, 11, and 18 are the areas formed by the corner points. Figure 3 The numbers 10, 1, 18, and 20 are the areas formed by the corner points. Figure 3 The number 11, 18, 12, and 19 are the areas formed by the corner points. Figure 3 The numbers 12, 19, 4, and 13 are the areas formed by the corner points. Figure 3 The numbers 2, 10, 11, and 18 are the areas formed by the corner points. Figure 3 The number 19, 21, 13, and 5 are the areas formed by the corner points. Figure 3 The numbers 21, 5, 14, and 22 are the areas formed by the corner points. Figure 3 The numbers 14, 22, 6, and 15 are the areas formed by the corner points. Figure 3 The number 22, 15, 23, and 16 are the areas formed by the corner points. Figure 3 The number 23, 17, 8, and 16 are the areas formed by the corner points. Figure 3 The numbers 9, 17, 23, and 20 are respectively regarded as the areas formed by the corner points.

[0049] Step S1300, applying a preset fisheye correction algorithm to perform image correction on the subdivided images corresponding to the multiple target subdivided areas in the fisheye image of the display, to obtain corresponding corrected subdivided images;

[0050] For the segmented image corresponding to each target segmented area in the fisheye image of the display, the cv2.getPerspectiveTransform function in the opencv library is called, and the corner point information corresponding to each corner point of the target segmented area and the preset regional resolution are used as the function input parameters to obtain the perspective matrix parameters corresponding to the matrix image of the preset regional resolution of the segmented image of the target segmented area. The preset regional resolution can be set as required by those skilled in the art. Further, the cv2.warpPerspective function of the opencv library is called, and the perspective matrix parameters and the segmented image are used as the function input parameters to obtain the matrix image, and complete the image correction of the segmented image.

[0051] It can be known from the typical embodiments of the present application that the technical solution of the present application has many advantages, including but not limited to the following aspects:

[0052] This application performs image instance segmentation and then performs regular point positioning of the boundary of the display fisheye image on the corresponding results, thereby obtaining the regional boundary point information corresponding to each point surrounding the boundary, ensuring real-time and accurate capture of the display content of the display. Then, according to the point mapping relationship between the points on the boundary of the display fisheye image and the points in the image, the corresponding points in the display fisheye image are located, and the display fisheye image is divided based on these points to obtain multiple target subdivision areas and their subdivision images, and the visual distortion of each subdivision image is corrected. It can accurately, efficiently and finely divide the target subdivision areas and their subdivision images in the display fisheye image, and ensure the accuracy and reliability of the image content in the subdivision image.

[0053] In a further embodiment, step S1100, performing boundary positioning on the image instance segmentation result of the display fisheye image according to a preset boundary positioning rule to obtain the region boundary point information corresponding to the display fisheye image, includes the following steps:

[0054] Step S1110: Use a preset image segmentation model to perform image instance segmentation on the display fisheye image to obtain point information of the boundary portion of the display fisheye image and an anchor frame area of ​​the display fisheye image in the frame;

[0055] The image segmentation model is pre-trained to a convergence state, and learns to determine the display fisheye image and its boundary point information that belong to the imaging range of the display in the input image, as well as the anchor frame area of ​​the display fisheye image in the frame. The image segmentation model is recommended to be selected from the YOLO series model such as YOLOV8, and other models that are also suitable for target detection and image instance segmentation can also be used, such as U-Net, DeepLab series, BiSeNet, etc., and those skilled in the art can select and implement as needed.

[0056] The image segmentation model is used to take the image frame as input to perform target detection and image instance segmentation on the image frame. The image segmentation model extracts the image semantic features corresponding to the image frame by performing convolution operations on the image frame, including the color, texture, boundary, deep complex and abstract semantic information of the image frame, etc., and generates a corresponding multi-scale feature map. The display fisheye image belonging to the imaging range of the display in the image frame is used as the target detection object, and multiple candidate frame areas are predicted on the feature map. The overlapping candidate frame areas are filtered through a non-maximum suppression algorithm to obtain the anchor frame area of ​​the target detection object in the frame. In addition, the image segmentation model classifies each pixel point in the image frame, constructs a pixel-level mask of the display fisheye image, thereby segmenting the display fisheye image, and uses the sobel algorithm to perform boundary detection on the display fisheye image to determine the coordinate information of each boundary point of the boundary of the display fisheye image to constitute the boundary part point information.

[0057] Step S1120: after linearly completing the part of the boundary point information, performing noise reduction processing on the corresponding obtained complete boundary point information to obtain boundary framework point information;

[0058] For the coordinate information of every two adjacent boundary points in the partial boundary point position information, and taking the boundary point with a smaller horizontal coordinate and / or vertical coordinate among the two boundary points as the starting point, and taking the other boundary point as the ending point, the Euclidean distance, the absolute value of the horizontal coordinate difference, and the absolute value of the vertical coordinate difference of the two boundary points are calculated. When the Euclidean distance is greater than or equal to the preset spacing, the Euclidean distance is divided by the preset spacing and rounded down to obtain the corresponding arithmetic result. The arithmetic result minus 1 is the total number of boundary points required to be added between the two boundary points. Then, a series of parameters uniformly distributed between 0 and 1 (excluding 0 and 1) are generated according to the arithmetic result. For each parameter, the parameter is multiplied by the absolute value of the horizontal coordinate difference and the absolute value of the vertical coordinate difference, and then added to the horizontal coordinate and vertical coordinate of the starting point to obtain the coordinate information of a single boundary point required to be added between the starting point and the ending point; when the Euclidean distance is less than the preset spacing, it means that no additional boundary points need to be added between the two adjacent boundary points. The preset spacing can be set as needed by those skilled in the art. For ease of understanding, an illustrative example is given. The coordinate information of the starting point is (x1:0, y1:20), the coordinate information of the end point is (x2:50, y2:40), and the preset spacing is 5. The Euclidean distance between the two boundary points is approximately 53.86, the absolute value of the difference in the horizontal coordinate is 50, and the absolute value of the difference in the vertical coordinate is 20. The Euclidean distance is divided by the preset spacing and rounded down to obtain the corresponding arithmetic result of 10. The total number of boundary points that need to be added between the two boundary points is 9, which is between 0 and 1 (excluding The parameters of the uniform distribution (including 0 and 1) are 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9 respectively, and the coordinate information of each boundary point that needs to be filled in is (x:5, y:22), (x:10, y:24), (x:15, y:26), (x:20, y:28), (x:25, y:30), (x:30, y:32), (x:35, y:34), (x:40, y:36), and (x:45, y:38).

[0059] Furthermore, the Ramer-Douglas-Peucker algorithm (full name Douglas Peucker algorithm, abbreviated as RDP algorithm) is used to perform noise reduction processing on the complete boundary point information, remove the coordinate information of redundant boundary points in the complete boundary point information, retain the coordinate information of multiple boundary points that constitute the boundary of the display fisheye image and do not affect its true shape, and obtain the boundary framework point information.

[0060] Step S1130, determining each display predetermined point area and its predetermined area point information in the anchor frame area according to the preset subdivided area positioning features and the boundary frame point information;

[0061] The subdivided area positioning feature includes the position distribution information of the display predetermined point area where each predetermined point is located in the anchor frame area, and the position distribution information includes the corresponding horizontal coordinate range and vertical coordinate range of the corresponding display predetermined point area distribution in the anchor frame area. Those skilled in the art can set the subdivided area positioning feature according to prior knowledge or experimental data, and can also implement it according to the relevant disclosure in the subsequent embodiments.

[0062] The width of the anchor frame area is the maximum horizontal coordinate minus the minimum horizontal coordinate in the edge complete point information, and the height of the anchor frame area is the maximum vertical coordinate minus the minimum vertical coordinate in the edge complete point information.

[0063] In one embodiment, nine predetermined points are set according to the display fisheye image of the complete imaging of the surrounding display, which correspond to the four corner points of the display frame, a midpoint of the lower long side and two midpoints of the wide sides, and two points where the long side of the display intersects with the display fisheye image, and the position distribution information corresponding to the display predetermined point area where each predetermined point is located in the anchor frame area is preset, see Figure 4 They are {horizontal axis range: [x min , 1 / 8*x max ], vertical coordinate range: [1 / 3*y max ,y max ]}(P1), {horizontal axis range: [1 / 32*x max ,7 / 32*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max ]}(P2), {horizontal axis range: [1 / 8*x max , 1 / 4*x max ], vertical coordinate range: [2 / 3*y max ,y max ]}(P3), {horizontal axis range: [3 / 8*x max , 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}(P4), {horizontal axis range: [5 / 8*x max , 3 / 4*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}(P5), {horizontal axis range: [21 / 32*x max , 31 / 32*x max ], vertical coordinate range: [3 / 12*y max , 11 / 12*y max ]}(P6), {horizontal axis range: [7 / 8*xmax , x max ], vertical coordinate range: [1 / 6*y max ,y max ]}(P7), {horizontal axis range: [x min , 3 / 8*x max ], vertical coordinate range: [y min , 1 / 6*y max ]}(P8), {horizontal axis range: [5 / 8*x max , x max ], vertical coordinate range: [y min , 1 / 6*y max ]}(P9), all the location distribution information is collected to form the location feature of the subdivided area. max 、x min ,y min ,y max They are the minimum horizontal coordinate, maximum horizontal coordinate, minimum vertical coordinate, and maximum vertical coordinate in the complete edge point information.

[0064] It is not difficult to understand that, according to the position distribution information of each display predetermined point area in the subdivided area positioning feature, each display predetermined point area can be determined in the anchor frame area, and then, according to the horizontal coordinate range and the vertical coordinate range corresponding to each display predetermined point area, the coordinate information of each boundary point in the boundary frame point information is traversed to determine the display predetermined point area to which each boundary point belongs, and then for each display predetermined point area, all boundary points belonging to the display predetermined point area are respectively regarded as single area boundary points, and the coordinate information of these area boundary points is collected to constitute the predetermined area point information of the display predetermined point area.

[0065] Step S1140, determining the predetermined point information corresponding to each display predetermined point area according to the preset regional fixed point features and the point information of each predetermined area;

[0066] The regional fixed-point features include locating the coordinate distribution patterns of predetermined points in each display predetermined point area. Those skilled in the art may set the regional fixed-point features based on prior knowledge or experimental data, or may implement them according to the relevant disclosures in some subsequent embodiments.

[0067] For {horizontal axis range: [x min , 3 / 8*x max ], vertical coordinate range: [y min , 1 / 6*y max]}, determine the predetermined point area of ​​the display predetermined point area, the ordinate is the minimum ordinate in the predetermined area point information, the coordinate information of each corresponding area boundary point, and then determine the coordinate information of the area boundary point with the minimum abscissa from the coordinate information of these area boundary points, as the coordinate information of the predetermined point of the display predetermined point area, to form the predetermined point information; for {abscissa range: [5 / 8*x max , x max ], vertical coordinate range: [y min , 1 / 6*y max ]}, determine the predetermined point area of ​​the display predetermined point area, the ordinate is the minimum ordinate in the predetermined area point information, the coordinate information of each corresponding area boundary point, and then determine the coordinate information of the area boundary point with the largest horizontal coordinate from the coordinate information of these area boundary points, as the coordinate information of the predetermined point of the display predetermined point area, to form the predetermined point information; for {horizontal coordinate range: [3 / 8*x max , 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}, determine the display predetermined point area, in the area contour information of the display predetermined point area, the horizontal coordinate is the value obtained by multiplying the width of the anchor frame area by the first preset ratio and adding the minimum horizontal coordinate in the edge complete point information, and the vertical coordinate is the area boundary point corresponding to the maximum vertical coordinate in the predetermined area point information, and the coordinate information of the corresponding area boundary point is used as the coordinate information of the predetermined point of the display predetermined point area, constituting the predetermined point information; for {horizontal coordinate range: [x min , 1 / 8*x max ], vertical coordinate range: [1 / 3*y max ,y max ]}, in one embodiment, the coordinate information of the area boundary point with the smallest horizontal coordinate in the area outline information of the display predetermined point area is determined, or the coordinate information of the area boundary point with the largest vertical coordinate among the N area boundary points with smaller horizontal coordinates in the area outline information is determined as the coordinate information of the predetermined point of the display predetermined point area, constituting the predetermined point position information. The N can be set as needed by those skilled in the art, for example, 5; for {horizontal coordinate range: [7 / 8*x max , x max ], vertical coordinate range: [1 / 6*y max ,y max]}, in one embodiment, the coordinate information of the area boundary point with the largest horizontal coordinate in the area outline information of the display predetermined point area is determined, or the coordinate information of the area boundary point with the largest vertical coordinate among the N area boundary points with larger horizontal coordinates in the area outline information is determined as the coordinate information of the predetermined point of the display predetermined point area, constituting the predetermined point position information; for {horizontal coordinate range: [1 / 8*x max , 1 / 4*x max ], vertical coordinate range: [2 / 3*y max ,y max ]} display predetermined point area, {horizontal axis range: [5 / 8*x max , 3 / 4*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}, firstly, the coordinate information of each boundary point in the boundary frame point information is sorted in a clockwise manner around the boundary of the display fisheye image, and a boundary point sequence is obtained. From the boundary point sequence, the vectors formed by each boundary point in the boundary point sequence and its next boundary point are determined in sequence, and each two vectors obtained are adjacent in position. Thus, the arc cosine value between each two vectors is calculated to be the angle between the two vectors, and then for each display predetermined point area, the coordinate information of the intersection between the two adjacent vectors with the largest angle is determined from the predetermined area point information of the display predetermined point area, as the predetermined point information of the display predetermined point area; for {horizontal coordinate range: [1 / 32*x max ,7 / 32*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max ]} display predetermined point area, {horizontal axis range: [21 / 32*x max , 31 / 32*x max ], vertical coordinate range: [3 / 12*y max , 11 / 12*y max ]}, for each display predetermined point area, determining the coordinate information of the area boundary point with the same distance as two predetermined points adjacent to the display predetermined point area as the coordinate information of the predetermined point in the display predetermined point area.

[0068] Step S1150: for every three adjacent predetermined point information, performing adaptive distortion adjustment on the center point information corresponding to two target predetermined point information among the three predetermined point information, and obtaining the center point information after adaptive distortion adjustment as the first area subdivision point information;

[0069] The predetermined point information located in the middle of the three predetermined point information is used as the single target predetermined point information, and then the horizontal coordinate or vertical coordinate of the corresponding predetermined point is determined from the three predetermined point information, and the corresponding predetermined point information is used as another target predetermined point information if the horizontal coordinate or vertical coordinate is smaller than the horizontal coordinate or vertical coordinate of the predetermined point in the target predetermined point information. The coordinate information of the midpoint between the two predetermined points corresponding to the two target predetermined point information is obtained as the center point information, and then the horizontal distortion offset value and the vertical distortion offset value of the midpoint corresponding to the two predetermined points are obtained from the preset distortion adjustment parameter set, so that the horizontal coordinate and the vertical coordinate in the center point information are added to the horizontal distortion offset value and the vertical distortion offset value to complete the adaptive distortion adjustment of the center point information, and the center point information after adaptive distortion adjustment is obtained as the first area subdivision point information.

[0070] The distortion adjustment parameter set includes the lateral distortion offset value and the longitudinal distortion offset value of the midpoint between two predetermined points corresponding to each two adjacent predetermined point information, which is used to adapt the fisheye imaging distortion and make corresponding adjustments. Those skilled in the art can flexibly set the distortion adjustment parameter set according to the disclosure herein.

[0071] Step S1160: Determine second area subdivision point information matching the preset area subdivision point feature from all the predetermined point information, and form area boundary point information with all the second area subdivision point information and all the first area subdivision point information.

[0072] The regional subdivision point feature includes the feature that each predetermined point required for dividing the display fisheye image corresponds to a point on the display frame to which it belongs. Those skilled in the art can flexibly set the regional subdivision point feature for the division of the display fisheye image according to their own needs. In one embodiment, the regional subdivision point feature includes the feature that each predetermined point required for dividing the display fisheye image corresponds to four corner points on the display frame to which it belongs (such as Figure 3 The middle number is 2, 4, 6, 8), a midpoint of the lower long side (such as Figure 3 5), and the two points where the long side of the display intersects with the display fisheye image (such as Figure 3 Thus, from all the predetermined point information, each predetermined point information matching the regional subdivision point feature is used as the second regional subdivision point information, and then all the second regional subdivision point information and all the first regional subdivision point information are combined to form the regional boundary point information.

[0073] In this embodiment, the regional boundary point information corresponding to the boundary of the display fisheye image is disclosed, which can ensure the accuracy and reliability of the coordinate information of each predetermined point pointed out from each display predetermined point area and the intermediate point information after adaptive distortion adjustment between each two adjacent predetermined point information based on the boundary points of the complete boundary of the display fisheye image.

[0074] In a further embodiment, step S1140, determining the predetermined point information corresponding to each display predetermined point area according to the preset regional fixed point feature and the point information of each predetermined area, includes any one or more of the following steps:

[0075] Step S1141: for each predetermined display point region including a predetermined point belonging to an intersection point of a long side of the display and a fisheye image of the display, or a midpoint of a long side of the display, determine the predetermined point position information of the predetermined display point region from the predetermined region point position information of the predetermined display point region according to the position of the predetermined display point region in the anchor frame region;

[0076] The two display predetermined point areas including the predetermined point belonging to the intersection point of the long side of the display and the display fisheye image are: {horizontal coordinate range: [x min , 3 / 8*x max ], vertical coordinate range: [y min , 1 / 6*y max ]}, determine the predetermined point area of ​​the display predetermined point area, the ordinate is the minimum ordinate in the predetermined area point information, the coordinate information of each corresponding area boundary point, and then determine the coordinate information of the area boundary point with the minimum abscissa from the coordinate information of these area boundary points, as the coordinate information of the predetermined point of the display predetermined point area, to form the predetermined point information; {abscissa range: [5 / 8*x max , x max ], vertical coordinate range: [y min , 1 / 6*y max ]}, determine the coordinate information of each area boundary point whose ordinate is the minimum ordinate in the predetermined area point information of the predetermined area point, and then determine the coordinate information of the area boundary point with the minimum abscissa from the coordinate information of these area boundary points as the coordinate information of the predetermined point of the predetermined area, to form the predetermined point information;

[0077] The display predetermined point area containing the predetermined point belonging to the midpoint of the long side of the display {horizontal coordinate range: [3 / 8*x max , 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,ymax ]}, determine that in the area contour information of the display predetermined point area, the horizontal coordinate is the value obtained by multiplying the width of the anchor frame area by the first preset ratio and adding the minimum horizontal coordinate in the edge complete point information, and the vertical coordinate is the area boundary point corresponding to the maximum vertical coordinate in the predetermined area point information, and the coordinate information of the corresponding area boundary point is used as the coordinate information of the predetermined point of the display predetermined point area, constituting the predetermined point information.

[0078] Step S1142: for each display predetermined point area including the predetermined point belonging to the corner point of the display, determine the vector formed by every two adjacent boundary points and the angle between every two adjacent vectors according to the boundary frame point position information, and determine the coordinate information of the intersection between two adjacent vectors with the largest angle from the predetermined area point position information of the display predetermined point area as the predetermined point position information of the display predetermined point area;

[0079] The display predetermined point regions containing the predetermined points belonging to the corner points of the display are: [x min , 1 / 8*x max ], vertical coordinate range: [1 / 3*y max ,y max ]}, {horizontal axis range: [1 / 8*x max , 1 / 4*x max ], vertical coordinate range: [2 / 3*y max ,y max ]}, {horizontal axis range: [5 / 8*x max , 3 / 4*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}, {horizontal axis range: [7 / 8*x max , x max ], vertical coordinate range: [1 / 6*y max ,y max ]}, firstly, the coordinate information of each boundary point in the boundary frame point information is sorted in a clockwise direction around the boundary of the display fisheye image to obtain a boundary point sequence, and from the boundary point sequence, the vector formed by each boundary point in the boundary point sequence and its next boundary point is determined in order, and each two vectors obtained are adjacent in position, thereby calculating the arc cosine value between each two vectors, which is the angle between the two vectors, and then for each display predetermined point area, the coordinate information of the intersection between two adjacent vectors with the largest angle is determined from the predetermined area point information of the display predetermined point area, as the predetermined point information of the display predetermined point area.

[0080] Step S1143, for each display predetermined point area that includes a predetermined point belonging to the midpoint of the wide side of the display, based on two predetermined point position information adjacent to the position of the display predetermined point area, determine from the predetermined area point information of the display predetermined point area the coordinate information of the area boundary point whose ratio of distance to the two predetermined point position information meets a preset condition, as the predetermined point information of the display predetermined point area.

[0081] The display predetermined point areas containing the predetermined point belonging to the midpoint of the wide side of the display are: {horizontal coordinate range: [1 / 32*x max ,7 / 32*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max ]}, {horizontal axis range: [21 / 32*x max , 31 / 32*x max ], vertical coordinate range: [3 / 12*y max , 11 / 12*y max ]}, for each display predetermined point area, the coordinate information of the area boundary point with the same distance as two predetermined points adjacent to the display predetermined point area is determined from the predetermined area point position information of the display predetermined point area, as the coordinate information of the color picking point in the color picking point area.

[0082] In this embodiment, the implementation process of determining the predetermined point information corresponding to each display predetermined point area is disclosed, ensuring that each predetermined point corresponds to the four corner points of the display frame, a lower long side midpoint and two wide side midpoints, and the two points where the long side of the display intersects with the display fisheye image.

[0083] In a further embodiment, step S1150, for every three adjacent predetermined point information, performing adaptive distortion adjustment on the center point information corresponding to two target predetermined point information in the three predetermined point information, and obtaining the adaptive distortion adjusted center point information as the first area subdivision point information, includes the following steps:

[0084] Step S1151: for every three adjacent preset point information, determine the distortion type and distortion ratio corresponding to the distortion imaging effect of the three preset point information;

[0085] The predetermined point information located in the middle of the three predetermined point information is taken as the target predetermined point information, the sum of the vertical coordinates in the remaining two predetermined point information is calculated and then divided by two to obtain the mean of the vertical coordinates. If the vertical coordinate in the target predetermined point information is less than the mean of the vertical coordinates, it is determined that the distortion type corresponding to the distortion imaging effect of the three preset point information is concave; if the vertical coordinate in the target predetermined point information is greater than the mean of the vertical coordinates, it is determined that the distortion type corresponding to the distortion imaging effect of the three preset point information is convex.

[0086] According to the remaining two predetermined point information, the slope and the slope-intercept in the mathematical expression of the straight line formed by the corresponding two predetermined points can be calculated, so as to obtain the mathematical expression of the straight line. The distance between the predetermined point corresponding to the target predetermined point information and the straight line can be calculated by applying the point-to-straight line distance formula according to the target predetermined point information and the mathematical expression of the straight line.

[0087] The distance between the two corresponding predetermined points is calculated based on the remaining two predetermined point information by applying the Euclidean formula. The distance between the predetermined point corresponding to the target predetermined point information and the straight line is divided by the distance between the two predetermined points corresponding to the two predetermined point information to obtain the distortion ratio corresponding to the distortion imaging effect of the three preset point information.

[0088] Step S1152: Determine the corresponding center point information, the distance between the two points and the slope according to the two target preset point information among the three preset point information;

[0089] Further, from the three predetermined point information, the horizontal coordinate or vertical coordinate of the corresponding predetermined point is determined, which is correspondingly smaller than the horizontal coordinate or vertical coordinate of the predetermined point in the target predetermined point information, and the corresponding predetermined point information is used as another target predetermined point information.

[0090] According to the preset point information of the two targets, the coordinate information of the midpoint between the corresponding two preset points is calculated as the center point information, and the slope of the straight line connecting the two preset points is calculated, and the distance between the two preset points is calculated as the distance between the two points.

[0091] Step S1153, determining the point offset information corresponding to the center point information according to the distortion type, distortion ratio, distance between two points and slope;

[0092] The arc tangent value corresponding to the absolute value of the slope is obtained to obtain the oblique angle. The distance between the two points is calculated and multiplied by the distortion ratio to obtain the distortion distance. The sine value corresponding to the oblique angle is obtained and multiplied by the distortion distance to obtain the horizontal axis offset. The cosine value corresponding to the oblique angle is obtained and multiplied by the distortion distance to obtain the vertical axis offset.

[0093] If the distortion type is convex and the slope is greater than zero, determine that the incremental value corresponding to the horizontal coordinate in the center point information is the horizontal axis offset, and the incremental value corresponding to the vertical coordinate in the center point information is the vertical axis offset. The incremental values ​​corresponding to the horizontal coordinate and the vertical coordinate are used as the point offset information.

[0094] If the distortion type is convex and the slope is less than zero, determine that the incremental value corresponding to the horizontal coordinate in the center point information is the negative value of the horizontal axis offset, and the incremental value corresponding to the vertical coordinate in the center point information is the vertical axis offset. The incremental values ​​corresponding to the horizontal coordinate and the vertical coordinate are used as the point offset information.

[0095] If the distortion type is concave and the slope is greater than zero, determine that the incremental value corresponding to the horizontal coordinate in the center point information is the negative value of the horizontal axis offset, and the incremental value corresponding to the vertical coordinate in the center point information is the negative value of the vertical axis offset. The incremental values ​​corresponding to the horizontal coordinate and the vertical coordinate are used as the point offset information.

[0096] If the distortion type is concave and the slope is less than zero, determine that the incremental value corresponding to the horizontal coordinate in the center point information is the horizontal axis offset, and the incremental value corresponding to the vertical coordinate in the center point information is the negative value of the vertical axis offset. The incremental values ​​corresponding to the horizontal coordinate and the vertical coordinate are used as the point offset information.

[0097] Step S1154: adjusting the center point information according to the point offset information, and obtaining the adjusted center point information as the first area subdivision point information.

[0098] The horizontal coordinate and the vertical coordinate in the center point information are added to their respective incremental values ​​in the point offset information, and the center point information formed by the obtained horizontal coordinate and vertical coordinate, that is, the adjusted center point information, is used as the first area subdivision point information.

[0099] In this embodiment, for every three adjacent preset point information, according to the distortion imaging effect of the three preset point information, the center point information corresponding to two target preset point information in the three preset point information is adjusted accordingly as the first area subdivision point information, thereby ensuring the accuracy and reliability of the first area subdivision point information.

[0100] In a further embodiment, step S1300, applying a preset fisheye correction algorithm to perform image correction on the subdivided images corresponding to the multiple target subdivided areas in the fisheye image of the display to obtain the corresponding corrected subdivided images, includes the following steps:

[0101] Step S1310: for each subdivided image corresponding to each target subdivided area in the fisheye image of the display, according to the corner point position information corresponding to each corner point of the target subdivided area, determining the perspective matrix parameters corresponding to the matrix image of the preset regional resolution for perspective transformation of the subdivided image of the target subdivided area;

[0102] For the segmented image corresponding to each target segmented area in the display fisheye image, the cv2.getPerspectiveTransform function in the opencv library is called, and the corner point information corresponding to each corner point of the target segmented area and the preset regional resolution are used as the function input parameters to obtain the perspective matrix parameters corresponding to the matrix image of the preset regional resolution after the perspective transformation of the segmented image of the target segmented area. The preset regional resolution can be set as needed by those skilled in the art.

[0103] Step S1320: Perform perspective transformation correction on the subdivided image according to the perspective matrix parameters to obtain the matrix image as the image after the subdivided image is corrected.

[0104] The cv2.warpPerspective function of the opencv library is called, and the perspective matrix parameters and the subdivided image of the target subdivided area are used as input parameters of the function to obtain the matrix image and complete the image correction of the subdivided image.

[0105] In this embodiment, the implementation process of image correction of the subdivided image of the target subdivided area is disclosed, and the subdivided image can be corrected very efficiently.

[0106] In a further embodiment, step S1300, applying a preset fisheye correction algorithm to perform image correction on the subdivided images corresponding to the multiple target subdivided areas in the fisheye image of the display, and obtaining the corresponding corrected subdivided images, comprises the following steps:

[0107] Step S1400, determining the core color of each corrected subdivided image;

[0108] In one embodiment, for each corrected subdivided image, full pixel sampling is performed on the subdivided image to determine the color of each pixel in the subdivided image, and the color is usually expressed in RGB values. The colors of all pixels are added and then divided by the total number of these pixels to obtain the average color of the subdivided image. Since the fisheye image of the display captured by the camera device is darker in color than the actual display image, in order to approach the real visual effect, the average color is enhanced, and the incremental value of any one component of R, G, and B preset by the user or the incremental value of any multiple components is added to the value of the same component of the average color to obtain the core color of the subdivided image. For the sake of understanding the exemplary example, the average color is R: 100, G: 100, B: 100, the incremental value of R preset by the user is 20, the incremental value of B is 5, and the core color is R: 120, G: 100, B: 105.

[0109] Step S1410: Control the lighting unit group corresponding to the corrected segmented image in the ambient lighting device to display colored light of the corresponding core color.

[0110] The control device can predetermine the mapping relationship between each of the subdivided images and the corresponding light unit in the atmosphere light device. Specifically, since the atmosphere light device and each subdivided image are arranged around the display, all the light units in the atmosphere light device can be divided into each part of the total number of light units in the lighting effect color picking area as each light unit group, so that each light unit group and each subdivided image have an equivalent positional relationship, and thus a one-to-one mapping relationship is established between each light unit group and each subdivided image based on this equivalent relationship. Therefore, the control device can encapsulate the lighting effect control signal based on the signal format of the WS2811 chip according to the core color corresponding to each subdivided image and the mapping relationship between each subdivided image and the corresponding light unit group in the atmosphere light device, so as to control each light unit group of the atmosphere light device to emit the color light corresponding to the core color of the mapped associated subdivided image.

[0111] The control device is connected to the atmosphere light device and outputs the lighting effect control signal to the atmosphere light device, so that the lighting effect control circuit of the atmosphere light device can control the light-emitting element in the atmosphere light device to emit light accordingly and present the corresponding lighting effect according to the lighting effect control signal. The light-emitting element can be a light-emitting element capable of emitting monochromatic light or three light-emitting elements capable of emitting RGB (red, green, blue) three colors of light.

[0112] In this embodiment, by determining the core color corresponding to each corrected segmented image, the corresponding light unit group in the atmosphere lighting device is controlled to display the corresponding color light, so as to achieve an atmosphere lighting effect that changes synchronously with the display contents of multiple parts of the display, thereby significantly enhancing the user's immersion and visual experience.

[0113] In a further embodiment, step S1400, determining the core color of each corrected subdivided image, includes the following steps:

[0114] Step S1401: for each corrected subdivided image, determine the colors of a plurality of pixels belonging to the core hue interval of the subdivided image and whose corresponding hue values ​​meet a preset condition, and calculate the corresponding average color;

[0115] Taking a single subdivided image as an example, the subdivided image is sampled at every other pixel to determine the color of each corresponding pixel, which is usually represented by an RGB value. The color represented by the RGB value of each pixel is converted into a corresponding color represented by an HSV value. A plurality of preset hue intervals are obtained. The plurality of preset hue intervals can be divided into [0°, 360°] as required by a person skilled in the art, for example, divided into 6 hue intervals, namely [0°, 60°), [60°, 120°), [120°, 180°), [180°, 240°), [240°, 300°), and [300°, 360°]. According to the hue value of the color represented by the HSV value of each pixel, the hue interval to which the color represented by the HSV value of each pixel belongs can be determined. For each hue interval, the total number of pixels belonging to the hue interval is determined, so as to select the hue interval with the largest total number of pixels as the core hue area. The colors represented by HSV of all pixels belonging to the core hue interval are sorted in descending order of hue value, and the colors represented by HSV of the top N pixels are selected. The colors represented by HSV of these pixels are converted to colors represented by RGB and then the average is calculated as the average color of the segmented image. The N can be set as needed by those skilled in the art, for example, 10.

[0116] Step S1402: Perform color enhancement on the average color to obtain the core color of the segmented image.

[0117] Since the fisheye image of the display captured by the camera equipment is darker in color than the actual display image, in order to approach the real visual effect, the average color is enhanced, and the incremental value of any one component of R, G, B preset by the user or the incremental values ​​of any multiple components are added to the value of the same component of the average color to obtain the target color of the segmented image.

[0118] In this embodiment, each subdivided image is sampled at alternate pixels, and the average color corresponding to the core hue interval of the subdivided image is obtained and then the color is enhanced to obtain the target color. On the one hand, since the color of the pixels separated by the sampling interval can reduce the amount of calculation, while still being able to obtain enough colors to represent the entire subdivided image, on the other hand, it can make up for the error between the camera shooting and the actual imaging, so that the subsequent lighting effect display effect is better.

[0119] In a further embodiment, before step S1100, obtaining the fisheye image of the display, the following steps are included:

[0120] Step S1000, obtaining a preset training set, training the image segmentation model to a convergent state, so that it can learn to determine the display fisheye image and its boundary point information belonging to the imaging range of the display in the input image, as well as the anchor frame area of ​​the display fisheye image in the frame, the training set includes multiple training samples and their supervisory labels, the training samples are images containing display fisheye images, the supervisory labels mark the target edge detection information corresponding to the display fisheye images in the corresponding training samples, and the training samples in the training set are obtained by photographing the display using a camera device with a single fisheye camera.

[0121] In the recommended embodiment, multiple images of the display produced by a camera device using a single fisheye camera in different environments are pre-collected. It is not difficult to understand that each collected image contains a fisheye image of the display corresponding to the imaging range of the display. For example, the fisheye image of the display in the image produced by the single fisheye camera is a complete imaging of the display. The different environments can be any one or any multiple environments at different times, with different objects placed next to the display, with the display placed on different objects, with the display attached to or hung on different objects, and with the display in different spaces, so as to enhance the generalization ability, reliability, and accuracy of the image segmentation model.

[0122] Each image is used as a single training sample. Further, labelme and labelimg marking tools are used to mark each training sample. Taking a single training sample as an example, the display fisheye image in the image of the training sample is used as the target detection object. The labelimg marking tool is used to frame the specific position and size of the target detection object in the image. In addition, the labelme marking tool is used to set corresponding boundary points along the boundary of the target detection object in the image. The boundary formed by the connection between the boundary points just encloses the target detection object. After the marking is completed, the labelme and labelimg marking tools generate a corresponding marking file containing the target edge detection information of the display fisheye image as the supervision label of the training sample. The target edge detection information includes the coordinate information of the center pixel of the display fisheye image in the corresponding image (also the coordinate information of the center pixel of the anchor frame of the display fisheye image in the frame in the image), the width and height of the display fisheye image in the image (also the width and height of the anchor frame in the image), and the coordinate information of each boundary point of the edge contour of the display fisheye image in the image.

[0123] The image segmentation model is recommended to be a YOLO series model such as YOLOV8, or other models that are also suitable for target detection and instance segmentation, such as U-Net, DeepLab series, BiSeNet, etc. Those skilled in the art can select and implement them as needed.

[0124] A single training sample in the training set and its supervision label are called, and the image segmentation model uses the training sample as input to perform target detection and instance segmentation on the training sample. The image segmentation model extracts the image semantic features corresponding to the training sample by performing convolution operations on the training sample, including the color, texture, edge, contour, deep complex and abstract semantic information of the training sample, and generates a corresponding multi-scale feature map. The display fisheye image in the training sample is used as the target detection object to predict multiple candidate box areas on the feature map, and the overlapping candidate box areas are filtered by non-maximum suppression to predict the predicted anchor box area of ​​the target detection object in the box. In addition, the image segmentation model classifies each pixel in the training sample and constructs a pixel-level mask of the display fisheye image to segment the predicted display fisheye image, and performs edge detection on the predicted display fisheye image to determine the coordinate information of each boundary point surrounding the edge of the predicted display fisheye image to form the predicted boundary point information. A cross entropy loss function is used to calculate the loss value corresponding to the cross entropy loss of the predicted anchor frame area, the predicted display fisheye image, and the predicted boundary point information based on the supervisory label of the training sample. When the loss value is lower than a preset threshold, it indicates that the image segmentation model has been trained to a convergence state, thereby terminating the training of the image segmentation model. When the loss value is greater than or equal to the preset threshold, it indicates that the image segmentation model has not converged, and then the image segmentation model is gradient updated according to the loss value. Usually, the weight parameters of each link in the image segmentation model are corrected by back propagation to make the image segmentation model further close to convergence. Then, other training samples in the training set and their supervisory labels are continuously called to perform iterative training on the image segmentation model until the image segmentation model is trained to a convergence state.

[0125] In this embodiment, the training process of the image segmentation model is disclosed to ensure the accuracy and reliability of the image segmentation model in identifying the display fisheye image and its boundary point information.

[0126] See also Figure 5, a fisheye image correction device provided to meet one of the purposes of the present application is a functional embodiment of the fisheye image correction method of the present application. On the other hand, the device is a fisheye image correction device provided to meet one of the purposes of the present application, including a boundary point determination module, a fisheye image subdivision module 1200 and an image correction module 1300, wherein the boundary point determination module 1100 is used to obtain a display fisheye image, perform boundary fixing on the image instance segmentation result of the display fisheye image according to a preset boundary fixing rule, and obtain the regional boundary point information corresponding to the display fisheye image; the fisheye image subdivision module 1200 is used to determine the boundary point information of the region according to the regional boundary point information and The preset regional point mapping relationship is used to determine the point information in the region corresponding to the fisheye image of the display, and the fisheye image of the display is regionally subdivided based on the regional boundary point information and the point information in the region to obtain multiple target subdivided regions; the image correction module 1300 is used to apply the preset fisheye correction algorithm to perform image correction on the subdivided images corresponding to the multiple target subdivided regions in the fisheye image of the display to obtain corresponding corrected subdivided images; the lighting effect display module 1400 is used to determine the core color of each corrected subdivided image, and control the lighting unit group corresponding to the corrected subdivided image in the atmosphere lighting device to display the color light of the corresponding core color.

[0127] In a further embodiment, the boundary point determination module 1100 includes: an instance segmentation submodule, which is used to perform image instance segmentation on the display fisheye image using a preset image segmentation model to obtain the boundary point information of the display fisheye image and the anchor frame area of ​​the display fisheye image in the frame; a linear completion submodule, which is used to linearly complete the partial boundary point information and then perform noise reduction processing on the corresponding obtained complete boundary point information to obtain boundary frame point information; a region and point determination submodule, which is used to determine each display predetermined point region and its predetermined region point information in the anchor frame region according to the preset subdivided region positioning features and the edge complete point information; a predetermined point determination submodule A block is used to determine the predetermined point information corresponding to each display predetermined point area according to the preset regional fixed point feature and the predetermined regional point information; an adaptive distortion adjustment submodule is used to perform adaptive distortion adjustment on the center point information corresponding to two target predetermined point information in the three predetermined point information for every three adjacent predetermined point information, and obtain the center point information after adaptive distortion adjustment as the first regional subdivision point information; an information construction submodule is used to determine the second regional subdivision point information matching the preset regional subdivision point feature from all the predetermined point information, and form the regional boundary point information with all the second regional subdivision point information and all the first regional subdivision point information.

[0128] In a further embodiment, the predetermined point position determination submodule includes any one or more of the following units: an area information determination unit, for determining, for each display predetermined point area including a predetermined point belonging to the intersection point of the long side of the display and the fisheye image of the display, or belonging to the midpoint of the long side of the display, the predetermined point position information of the display predetermined point area from the predetermined area point position information of the display predetermined point area according to the position of the display predetermined point area in the anchor frame area; a predetermined point positioning unit, for each display predetermined point area including a predetermined point belonging to a corner point of the display, determining, according to the boundary frame point position information, a vector formed by every two adjacent boundary points, and The angle between every two adjacent vectors is determined from the predetermined area point information of the display predetermined point area, and the coordinate information of the intersection point between the two adjacent vectors with the largest angle is determined as the predetermined point information of the display predetermined point area; the first point information determination unit is used to, for each display predetermined point area containing the predetermined point belonging to the midpoint of the wide side of the display, determine, from the predetermined area point information of the display predetermined point area, the coordinate information of the area boundary point whose distance ratio with the two predetermined point information meets the preset conditions, according to two predetermined point information adjacent to the position of the display predetermined point area, as the predetermined point information of the display predetermined point area.

[0129] In a further embodiment, the adaptive distortion adjustment submodule includes: a distortion determination unit, including determining, for each three adjacent preset point information, the distortion type and distortion ratio corresponding to the distortion imaging effect of the three preset point information; an intermediate information determination unit, including determining the corresponding center point information, the distance between the two points and the slope based on the two target preset point information among the three preset point information; an offset information determination unit, including determining the point offset information corresponding to the center point information based on the distortion type, distortion ratio, the distance between the two points and the slope; a second point information determination unit, including adjusting the center point information according to the point offset information, and obtaining the adjusted center point information as the first area subdivision point information.

[0130] In a further embodiment, the image correction module 1300 includes: a parameter determination submodule, which is used to determine, for each target subdivision area in the fisheye image of the display, perspective matrix parameters corresponding to the perspective transformation of the subdivision image of the target subdivision area into a matrix image of a preset regional resolution according to the corner point position information corresponding to each corner point of the target subdivision area; and a perspective transformation submodule, which is used to perform perspective transformation correction on the subdivision image according to the perspective matrix parameters to obtain the matrix image as the image after the subdivision image is corrected.

[0131] In a further embodiment, the image correction module 1300 includes: a core color determination submodule, which is used to determine the core color of each corrected segmented image; and a color light display submodule, which is used to control the light unit group corresponding to the corrected segmented image in the atmosphere light device to display the color light of the corresponding core color.

[0132] In a further embodiment, the color light display submodule includes: an average color determination submodule, which is used to determine, for each corrected subdivided image, the core hue interval belonging to the subdivided image, and the colors of multiple pixels whose corresponding hue values ​​meet preset conditions, and obtain the corresponding average color; a color enhancement submodule, which is used to enhance the average color to obtain the core color of the subdivided image.

[0133] In a further embodiment, before the boundary point determination module 1100, it includes: a model training submodule, which is used to obtain a preset training set, train the image segmentation model to a convergence state, so that it can learn to determine the display fisheye image and its boundary part point information belonging to the imaging range of the display in the input image, and the anchor frame area of ​​the display fisheye image in the frame. The training set includes multiple training samples and their supervisory labels, and the training samples are images containing display fisheye images. The supervisory labels mark the target edge detection information corresponding to the display fisheye images in the corresponding training samples. The training samples in the training set are obtained by photographing the display using a camera device with a single fisheye camera.

[0134] In order to solve the above technical problems, the present application also provides a computer device. Figure 6 As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a fisheye image correction method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the fisheye image correction method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0135] In this embodiment, the processor is used to execute Figure 5 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the fisheye image correction device of this application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0136] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the fisheye image correction method of any embodiment of the present application.

[0137] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0138] In summary, the present application can efficiently and accurately correct the image area in the fisheye image of the display.

[0139] It will be understood by those skilled in the art that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, changed, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, changed, rearranged, decomposed, combined, or deleted. Furthermore, the steps, measures, and schemes in the various operations, methods, and processes in the prior art that are open source and in this application may also be alternated, changed, rearranged, decomposed, combined, or deleted.

[0140] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A fisheye image correction method, characterized in that: The steps include: Acquire a display fisheye image, perform boundary fixing on the image instance segmentation result of the display fisheye image according to a preset boundary fixing rule, and obtain the regional boundary point information corresponding to the display fisheye image, which includes: using a preset image segmentation model to perform image instance segmentation on the display fisheye image, and obtain the boundary part point information of the display fisheye image, and the anchor frame area of ​​the display fisheye image in the frame; After linearly completing the boundary point information, performing noise reduction processing on the corresponding complete boundary point information to obtain boundary frame point information; Determine each display predetermined point area and its predetermined area point information in the anchor frame area according to the preset subdivided area positioning features and the boundary frame point information; Determine the predetermined point information corresponding to each display predetermined point area according to the preset regional fixed point features and the point information of each predetermined area; For every three adjacent predetermined point information, adaptive distortion adjustment is performed on the center point information corresponding to two target predetermined point information among the three predetermined point information, and the center point information after adaptive distortion adjustment is obtained as the first area subdivision point information; Determine second area subdivision point information matching the preset area subdivision point feature from all the predetermined point information, and form area boundary point information with all the second area subdivision point information and all the first area subdivision point information; According to the regional boundary point information and the preset regional point mapping relationship, the regional point information corresponding to the display fisheye image is determined, and the display fisheye image is regionally subdivided based on the regional boundary point information and the regional point information to obtain multiple target subdivided areas; A preset fisheye correction algorithm is applied to perform image correction on the subdivided images corresponding to the multiple target subdivided areas in the fisheye image of the display to obtain corresponding corrected subdivided images.

2. The fisheye image correction method according to claim 1, characterized in that: Determining the predetermined point information corresponding to each display predetermined point area according to the preset regional fixed point features and the point information of each predetermined area includes any one or more of the following steps: For each predetermined display point region including a predetermined point belonging to an intersection point of a long side of the display and a fisheye image of the display, or a predetermined point belonging to a midpoint of a long side of the display, according to a position of the predetermined display point region in the anchor frame region, the predetermined point position information of the predetermined display point region is determined from the predetermined area point position information of the predetermined display point region; For each display predetermined point area including the predetermined point belonging to the corner point of the display, the vector formed by each two adjacent boundary points and the angle between each two adjacent vectors are determined according to the boundary frame point position information, and the coordinate information of the intersection point between two adjacent vectors with the largest angle is determined from the predetermined area point position information of the display predetermined point area as the predetermined point position information of the display predetermined point area; For each display predetermined point area that includes a predetermined point belonging to the midpoint of the wide side of the display, based on two predetermined point position information adjacent to the position of the display predetermined point area, the coordinate information of the area boundary point whose ratio of distance to the two predetermined point position information meets the preset conditions is determined from the predetermined area point information of the display predetermined point area as the predetermined point information of the display predetermined point area.

3. The fisheye image correction method according to claim 1, characterized in that: For every three adjacent predetermined point information, adaptive distortion adjustment is performed on the center point information corresponding to two target predetermined point information in the three predetermined point information, and the center point information after adaptive distortion adjustment is obtained as the first area subdivision point information, including the following steps: For every three adjacent preset point information, determine the distortion type and distortion ratio corresponding to the distortion imaging effect of the three preset point information; According to two target preset point information among the three preset point information, the corresponding center point information, the distance between the two points and the slope are determined; Determine the point offset information corresponding to the center point information according to the distortion type, distortion ratio, distance between two points and slope; The center point information is adjusted according to the point offset information, and the adjusted center point information is obtained as the first area subdivision point information.

4. The fisheye image correction method according to claim 1, characterized in that: Applying a preset fisheye correction algorithm to perform image correction on the subdivided images corresponding to the multiple target subdivided areas in the fisheye image of the display to obtain corresponding corrected subdivided images includes the following steps: For each subdivided image corresponding to each target subdivided area in the fisheye image of the display, according to the corner point position information corresponding to each corner point of the target subdivided area, determining the perspective matrix parameters corresponding to the perspective transformation of the subdivided image of the target subdivided area into a matrix image of a preset regional resolution; The subdivided image is subjected to perspective transformation correction according to the perspective matrix parameters to obtain the matrix image as the image after the subdivided image is corrected.

5. The fisheye image correction method according to claim 1, characterized in that: Applying a preset fisheye correction algorithm to perform image correction on the subdivided images corresponding to the multiple target subdivided areas in the fisheye image of the display to obtain the corresponding corrected subdivided images includes the following steps: Determine the core color of each corrected segmented image; The light unit group corresponding to the corrected subdivided image in the ambient light device is controlled to display colored light of the corresponding core color.

6. The fisheye image correction method according to claim 5, characterized in that: Determining the core color of each corrected segmented image includes the following steps: For each corrected subdivided image, determine the colors of multiple pixels belonging to the core hue interval of the subdivided image and whose corresponding hue values ​​meet preset conditions, and calculate the corresponding average color; The average color is enhanced to obtain the core color of the segmented image.

7. The fisheye image correction method according to claim 1, characterized in that: Before obtaining the fisheye image of the display, include: A preset training set is obtained, and an image segmentation model is trained to a convergent state so that the model can learn to determine the display fisheye image and its boundary point information belonging to the imaging range of the display in the input image, as well as the anchor frame area of ​​the display fisheye image in the frame. The training set includes multiple training samples and their supervisory labels. The training samples are images containing display fisheye images. The supervisory labels mark target edge detection information corresponding to the display fisheye images in the corresponding training samples. The training samples in the training set are obtained by photographing the display using a camera device with a single fisheye camera.

8. A fisheye image correction device, characterized in that: include: A boundary point determination module obtains a display fisheye image, performs boundary fixing on the image instance segmentation result of the display fisheye image according to a preset boundary fixing rule, and obtains the boundary point information of the area corresponding to the display fisheye image, which includes: using a preset image segmentation model to perform image instance segmentation on the display fisheye image, and obtains the boundary part point information of the display fisheye image, and the anchor frame area of ​​the display fisheye image in the frame; After linearly completing the boundary point information, performing noise reduction processing on the corresponding complete boundary point information to obtain boundary frame point information; Determine each display predetermined point area and its predetermined area point information in the anchor frame area according to the preset subdivided area positioning features and the boundary frame point information; Determine the predetermined point information corresponding to each display predetermined point area according to the preset regional fixed point features and the point information of each predetermined area; For every three adjacent predetermined point information, adaptive distortion adjustment is performed on the center point information corresponding to two target predetermined point information among the three predetermined point information, and the center point information after adaptive distortion adjustment is obtained as the first area subdivision point information; Determine second area subdivision point information matching the preset area subdivision point feature from all the predetermined point information, and form area boundary point information with all the second area subdivision point information and all the first area subdivision point information; A fisheye image segmentation module is used to determine the point information in the region corresponding to the fisheye image of the display according to the region boundary point information and the preset region point mapping relationship, and to perform regional segmentation on the fisheye image of the display based on the region boundary point information and the region point information to obtain a plurality of target segmentation regions; The image correction module is used to apply a preset fisheye correction algorithm to perform image correction on the subdivided images corresponding to the multiple target subdivided areas in the fisheye image of the display to obtain corresponding corrected subdivided images.

9. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

Images