Ambient lighting effect control method and device and equipment

Abstract

The present application relates to an atmosphere lighting control method and its device and equipment in the field of lighting control, the method comprising: obtaining image frames in a video stream generated by a camera device shooting a display, determining the display area belonging to the imaging range of the display, and the color picking point information of each color picking point set around the display area; partitioning the display area according to the color picking point information to obtain multiple subdivided areas, and using the multiple refined areas arranged along the circumference of the display area as lighting effect color picking areas; color sampling each lighting effect color picking area to determine the corresponding target color; constructing a lighting effect control signal according to the target color corresponding to each lighting effect color picking area, and outputting the lighting effect control signal to the atmosphere light device to control the atmosphere light device to emit light according to the lighting effect control signal. The present application realizes an atmosphere lighting effect that is synchronized with the display content.

Description

Technical Field

[0001] The present application relates to the field of lighting control, and in particular to an atmosphere lighting effect control method and its corresponding device and computer equipment. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the significant progress of deep learning algorithms and neural processing unit (NPU) chip technology, the application of artificial intelligence has gradually expanded to the consumer electronics market. In this field, the camera-equipped display back light strip product represents an innovative application example. The core function of the product needs to capture the color displayed on the display screen through the camera to display the corresponding lighting effect on the back light strip. The operation of this function usually depends on the positioning of the display frame. In the existing technology, the user is usually required to manually locate the display frame, which generally includes identifying the key points on some edges of the display frame, such as the starting point, midpoint and end point. This process is not only cumbersome, but also may be difficult for users, because it requires users to have certain operating skills in order to accurately identify and locate the TV frame in the picture on the TV screen. This manual positioning process not only affects the user experience, but may also cause the lighting effect to be incompletely synchronized with the screen content due to inaccurate positioning, thereby reducing the practicality and market competitiveness of the product.

[0003] It can be seen that the traditional light screen equipment has not achieved good results in lighting effect display, which affects the promotion of light screen equipment. It is necessary to upgrade the technology of light screen equipment. Summary of the invention

[0004] The primary purpose of the present application is to solve at least one of the above problems and provide an atmosphere lighting effect control method and its corresponding device and computer equipment.

[0005] In order to meet the various objectives of this application, this application adopts the following technical solutions:

[0006] An ambient lighting effect control method provided to meet one of the purposes of this application includes the following steps:

[0007] Acquire an image frame in a video stream generated by a camera device shooting a display, determine a display area in the image frame that belongs to an imaging range of the display, and color picking point information of each color picking point set around the display area;

[0008] Partitioning the display area according to the color picking point information to obtain a plurality of subdivided areas, and using the plurality of subdivided areas arranged along the circumference of the display area as lighting effect color picking areas;

[0009] Perform color sampling on each lighting effect color picking area to determine the target color corresponding to each lighting effect color picking area;

[0010] A lighting effect control signal is constructed according to the target color corresponding to each lighting effect color picking area, and the lighting effect control signal is output to the atmosphere light device to control the atmosphere light device to emit light according to the lighting effect control signal.

[0011] On the other hand, an atmosphere lighting effect control device provided to meet one of the purposes of the present application includes an image processing module, a color determination module, a color sampling module and a lighting effect playback module, wherein the image processing module is used to obtain image frames in a video stream generated by a camera device shooting a display, and determine the display area in the image frame that belongs to the imaging range of the display, and the color picking point information of each color picking point set around the display area; the color determination module is used to partition the display area according to the color picking point information to obtain a plurality of subdivided areas, and use the plurality of refined areas arranged along the circumference of the display area as lighting effect color picking areas; the color sampling module is used to perform color sampling on each lighting effect color picking area, and determine the target color corresponding to each lighting effect color picking area; the lighting effect playback module is used to construct a lighting effect control signal according to the target color corresponding to each lighting effect color picking area, and output the lighting effect control signal to the atmosphere lighting device to control the atmosphere lighting device to emit light according to the lighting effect control signal.

[0012] On the other hand, a computer device provided to meet one of the purposes of the present application includes a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the atmosphere lighting effect control method described in the present application.

[0013] On the other hand, a computer program product provided to meet another purpose of the present application includes a computer program / instruction, which, when executed by a processor, implements the steps of the method described in any embodiment of the present application.

[0014] The technical solution of this application has many advantages, including but not limited to the following aspects:

[0015] This application first extracts image frames from the video stream generated by the camera device shooting the display, and determines the display area where the display emits light and the color picking point information of each color picking point set around the display area for each image frame, to ensure real-time and accurate capture of the display content of the display, and determine the color picking points on the edge of the corresponding display area, laying the foundation for subsequent lighting effect color picking.

[0016] Next, the display area is divided into multiple sub-areas based on the color picking point information, and the multiple sub-areas arranged along the circumference of the display area are respectively used as lighting effect color picking areas, and the target color corresponding to each lighting effect color picking area is sampled, so as to construct lighting effect control information and output it to the atmosphere lighting device, control the atmosphere lighting device to emit light accordingly, and realize the conversion of the target color corresponding to the display content to the corresponding atmosphere lighting effect, extending the atmosphere lighting effect to the environment outside the display, and being able to dynamically adapt to the changes in the display content in real time to present matching atmosphere lighting effects, creating a natural and beautiful viewing atmosphere, and significantly improving the user's visual experience and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0018] Figure 1 A schematic diagram of the circuit principle of an atmosphere light control device according to a typical embodiment of the present invention;

[0019] Figure 2 A flowchart of a typical embodiment of the ambient lighting effect control method of the present application;

[0020] Figure 3 The image frame of the example of the present application is taken by a camera device using a single fisheye camera, and is a schematic diagram of the position distribution information corresponding to the color picking point area where 8 color picking points are located in the anchor frame area;

[0021] Figure 4 The image frame of the example of the present application is taken by the left fisheye camera of the camera device of the dual fisheye camera, and is a schematic diagram of the position distribution information corresponding to the color picking point area where the five color picking points are located in the anchor frame area;

[0022] Figure 5 The image frame of the example of the present application is taken by the right fisheye camera of the camera device of the dual fisheye camera, and is a schematic diagram of the position distribution information corresponding to the color picking point area where the five color picking points are located in the anchor frame area;

[0023] Figure 6 This is a principle block diagram of the ambient lighting effect control device of this application;

[0024] Figure 7 A schematic diagram of the structure of a computer device used in this application. DETAILED DESCRIPTION

[0025] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and cannot be interpreted as limiting the present invention.

[0026] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0027] Those skilled in the art will understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0028] In a typical embodiment of the present invention, the atmosphere lighting effect control method is implemented based on the atmosphere lighting effect control device 10, combined with Figure 1 The atmosphere lighting effect control device 10 includes a camera device 11, a control unit 12 and an atmosphere lighting device 13.

[0029] The camera device 11, the control unit 12 and the atmosphere light device 13 can be connected by wire, and the camera device 11, the control unit 12 and the atmosphere light device 13 can also be connected by a network. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network. In some embodiments, the camera device 11, the control unit 12 and the atmosphere light device 13 can also communicate and transmit through a specific communication protocol, and the communication protocol includes but is not limited to a BLE (Bluetooth low energy) protocol, a WLAN (Wireless Local Area Network) protocol, a Bluetooth protocol, a ZigBee protocol or a Wi Fi (Wireless Fidelity) protocol, etc. The network can also include network entities such as routers and gateways, which are not shown in the figure.

[0030] The control unit 12 is connected to the camera device 11 so that the control unit 12 can read the video stream generated by the camera device, or the camera device sends the video stream generated by the camera device to the control unit. The control unit is connected to the atmosphere light device 13 so that the control unit 12 can control the lighting effect of the atmosphere light device.

[0031] The camera device 11 can be a single fisheye camera or a dual fisheye camera, which can be installed and fixed just above (as shown) or just below (not shown) the display. The single fisheye camera shoots by using a short focal length and a 180-degree viewing angle. The left fisheye camera and the right fisheye camera of the dual fisheye camera simultaneously use the same short focal length and a 90-degree viewing angle to shoot.

[0032] The control unit 12 can adopt SOC (System on Chip), CPU (central processing unit), MPU (Microcontroller Unit), NPU (neural-network processing unit), etc., and can be set according to actual use needs, and this application does not impose any restrictions on this.

[0033] In one embodiment, the atmosphere light device 13 may be equipped with a WS2811 chip to receive and process the lighting effect control signal output by the corresponding control unit to emit light according to the lighting effect control signal.

[0034] In some embodiments, the atmosphere light device 13 may be a strip light, a bar light, a lamp post, or a flat lamp (containing a plurality of lamp beads).

[0035] In some embodiments, the atmosphere light device 13 may include one or more light units, which may be combined accordingly according to actual use requirements.

[0036] In some embodiments, each lighting unit may include one or more lamp beads, and the control unit 12 may control the brightness, color, etc. of the lamp beads of the lighting unit, and may also control the number of lights on, thereby achieving different lighting effects of the atmosphere lighting device 13.

[0037] In some implementations, the position and number of the atmosphere light devices 13 can be set according to actual use needs. According to the position of the atmosphere light devices 13, the length, height, and area of ​​the lighting effect can also be controlled.

[0038] See also Figure 2 In a typical embodiment of the present application, an atmosphere lighting effect control method comprises the following steps:

[0039] Step S1100, obtaining an image frame in a video stream generated by a camera device shooting a display, determining a display area in the image frame that belongs to an imaging range of the display, and color picking point information of each color picking point set around the display area;

[0040] For the camera device using a single fisheye camera, 8 color picking points are set for the display area of ​​the complete imaging of the surround display, corresponding to the four corner points, two long side midpoints, and two wide side midpoints located at the edge of the display; for the camera device using a double fisheye camera, 10 color picking points are set for the display area of ​​the complete imaging of the surround display, specifically, 5 color picking points are set for the display area of ​​the left half of the surround display, corresponding to the two corner points located at the left edge of the display, a midpoint of the upper long side, a midpoint of the left wide side, a point located to the left of the midpoint of the lower long side and at a preset distance from the point, and 5 color picking points are set for the display area of ​​the right half of the surround display, corresponding to the two corner points located at the right edge of the display, a midpoint of the upper long side, a midpoint of the right wide side, a point located to the right of the midpoint of the lower long side and at a preset distance from the point. The preset distance can be flexibly set by those skilled in the art.

[0041] The camera device in the atmosphere lighting control device is used to shoot images within its visible range and generate corresponding video streams. The control unit is connected to the camera device so that the control unit can read the video stream generated by the camera device when shooting the display. Therefore, it can be understood that when a camera device with a single fisheye camera is used to shoot the display, the control unit obtains the video stream generated by the camera device and then performs frame processing on it, that is, the video stream is split into multiple image frames, so as to obtain each image frame arranged in chronological order; when a camera device with a dual fisheye camera is used to shoot the display, the control unit obtains the video stream generated by the camera device, and the video stream includes sub-video streams generated by the left fisheye camera and the right fisheye camera in the camera device shooting simultaneously, and then the two sub-video streams are respectively frame processed, so as to obtain each image frame in each sub-video stream arranged in chronological order. The left fisheye camera shoots the left half of the display for imaging, and the right fisheye camera shoots the right half of the display for imaging.

[0042] The frame processing can be implemented by using ffmpeg audio and video processing tools, OpenCV library, etc. For example, the VideoCapture class of OpenCV can be used to read the video stream, and the read method can be used to read the image frames in the video stream frame by frame.

[0043] A preset image segmentation model is used to take the image frame as input to perform target detection and instance segmentation on the image frame. The image segmentation model extracts image semantic features corresponding to the image frame by performing convolution operations on the image frame, including color, texture, edge, contour, deep complex and abstract semantic information of the image frame, etc., and generates a corresponding multi-scale feature map. The display area in the image frame that belongs to the imaging range of the display is used as the target detection object, and multiple candidate frame areas are predicted on the feature map. The overlapping candidate frame areas are filtered through a non-maximum suppression algorithm to obtain the anchor frame area of ​​the target detection object in the frame. In addition, the image segmentation model classifies each pixel in the image frame and constructs a pixel-level mask of the display area to segment the display area, and uses the Sobel algorithm to perform edge detection on the display area to determine the coordinate information of each edge point surrounding the edge of the display area to form edge point information.

[0044] The image segmentation model is pre-trained to a convergence state, and learns to determine the display area and its edge point information in the input image that belongs to the imaging range of the display, as well as the anchor frame area of ​​the display area in the frame. The image segmentation model is recommended to be a YOLO series model such as YOLOV8, or other models that are also suitable for target detection and instance segmentation, such as U-Net, DeepLab series, BiSeNet, etc., and those skilled in the art can select and implement as needed.

[0045] Further, usually the image segmentation model determines the edge point information surrounding the edge of the display area, and the corresponding edge point cannot completely surround the display area, so the edge point information is regarded as the edge point information to be completed, and then the linear interpolation algorithm is used to complete the edge point information to be completed, and the edge contour point information is obtained, so that the corresponding edge point completely surrounds the display area, presenting a complete edge contour of the display area. Further, the edge contour point information is subjected to noise reduction processing to obtain edge frame point information. According to the preset regional positioning feature, the position distribution information of the color picking point area where each color picking point is located in the anchor frame area can be determined in the anchor frame area, and then, according to the horizontal coordinate range and vertical coordinate range corresponding to each color picking point area, the coordinate information of each edge point in the edge frame point information is traversed to determine the color picking point area to which each edge point belongs, and then for each color picking point area, all edge points belonging to the color picking point area are respectively used as single area contour points, and the coordinate information of these area contour points is collected to form the area contour point information of the color picking point area. According to the preset regional fixed point features and the contour point information of each region, the coordinate information of the color picking point in each color picking point region is determined to form the color picking point information. The coordinate information includes the corresponding horizontal coordinate and vertical coordinate of the corresponding color picking point in the anchor frame region.

[0046] The position distribution information includes the horizontal coordinate range and the vertical coordinate range corresponding to the distribution of the corresponding color picking point area in the anchor frame area. Those skilled in the art can set the regional positioning features according to prior knowledge or experimental data, or implement them according to the relevant disclosures in the subsequent embodiments. The regional fixed-point features include locating the coordinate distribution rules of the color picking points in each color picking point area. Those skilled in the art can set the regional fixed-point features according to prior knowledge or experimental data, or implement them according to the relevant disclosures in the subsequent embodiments.

[0047] Step S1200, partitioning the display area according to the color picking point information to obtain a plurality of subdivided areas, and using the plurality of subdivided areas arranged along the circumference of the display area as lighting effect color picking areas;

[0048] In one embodiment, each color picking point can be located in the display area based on the color picking point information. This can be achieved by connecting pairs of color picking points that are relatively positioned left and right and relatively up and down on the display area with a preset distortion angle and a preset connection method between corresponding points. After that, every two adjacent color picking points are connected with a straight line to partition the display area and obtain a corresponding plurality of subdivided areas.

[0049] In other embodiments, each color picking point can be located in the display area according to the color picking point information. On this basis, the partition points near each color picking point can also be located for some color picking points, and some or all color picking points and these partition points are connected according to the relative left-right position or up-down position relationship between points, and with the preset distortion angle and preset connection method between the corresponding points. After that, every two adjacent points are connected with a straight line to partition the display area and obtain corresponding multiple subdivided areas. The coordinate information of the partition points near the color picking points can be flexibly set by a technician in this field according to prior knowledge or experimental data to locate the partition points.

[0050] The preset connection mode can be any one of a straight line, a curve, a broken line, or any combination of these. The preset distortion angle can be flexibly set by a person skilled in the art based on prior knowledge or experimental data.

[0051] Step S1300, performing color sampling on each lighting effect color picking area to determine a target color corresponding to each lighting effect color picking area;

[0052] In one embodiment, for each lighting effect color picking area, all pixel points of the lighting effect color picking area are sampled to determine the color of each pixel point in the lighting effect color picking area. The color is usually expressed in RGB values. The colors of all pixels are added and then divided by the total number of these pixels to obtain the average color of the lighting effect color picking area. Since the display area photographed by the camera device is darker than the actual display area, in order to approach the real visual effect, the average color is enhanced, and the incremental value of any one component of R, G, and B preset by the user or the incremental value of any multiple components is added to the value of the same component of the average color to obtain the target color of the lighting effect color picking area. For the sake of understanding the exemplary example, the average color is R: 100, G: 100, B: 100, the incremental value of R preset by the user is 20, the incremental value of B is 5, and the target color is R: 120, G: 100, B: 105.

[0053] In another embodiment, for each lighting effect color picking area, the lighting effect color picking area is sampled at every other pixel to determine the color of each corresponding pixel, the color is usually represented by RGB values, and the color of each pixel represented by RGB values ​​is converted into a corresponding color represented by HSV values. A plurality of preset hue intervals are obtained, and the plurality of preset hue intervals can be divided into [0°, 360°] as required by a technician in this field, for example, divided into 6 hue intervals, namely [0°, 60°), [60°, 120°), [120°, 180°), [180°, 240°), [240°, 300°), and [300°, 360°]. According to the hue value of the color represented by HSV values ​​of each pixel, the hue interval to which the color represented by HSV values ​​of each pixel belongs can be determined. For each hue interval, the total number of pixels belonging to the hue interval is determined, so as to filter out the hue interval with the largest total number of pixels. The colors represented by HSV of all pixels belonging to the hue interval with the largest total number of pixels are sorted in descending order of hue value, and the colors represented by HSV of the top N pixels are screened out. The colors represented by HSV of these pixels are converted into colors represented by RGB and then the average is calculated as the main color of the lighting effect color picking area. Since the display area captured by the camera device is darker than the actual display area, in order to approach the real visual effect, the main color is enhanced, and the incremental value of any one component of R, G, B preset by the user or the incremental value of any multiple components is added to the value of the same component of the main color to obtain the target color of the lighting effect color picking area. The N can be set as needed by a technician in this field, for example 10.

[0054] Step S1400: construct a lighting effect control signal according to the target color corresponding to each lighting effect color picking area, and output the lighting effect control signal to the ambient light device to control the ambient light device to emit light according to the lighting effect control signal.

[0055] The control device can predetermine the mapping relationship between each lighting effect color picking area and the corresponding light unit in the atmosphere light device. Specifically, since the atmosphere light device and each lighting effect color picking area are arranged circumferentially around the display, all light units in the atmosphere light device can be divided into light units of the total number of lighting effect color picking areas, so that each light unit and each lighting effect color picking area have a positional equivalence relationship, and thus a one-to-one mapping relationship is established between each light unit and each lighting effect color picking area based on this equivalence relationship. Therefore, the control device can encapsulate the lighting effect control signal based on the signal format of the WS2811 chip according to the target color corresponding to each lighting effect color picking area and the mapping relationship between each lighting effect color picking area and the corresponding light unit in the atmosphere light device, so as to control each light unit of the atmosphere light device to emit light corresponding to the color of its mapped associated lighting effect color picking area.

[0056] The control device is connected to the atmosphere light device and outputs the lighting effect control signal to the atmosphere light device, so that the lighting effect control circuit of the atmosphere light device can control the light-emitting element in the atmosphere light device to emit light accordingly and present the corresponding lighting effect according to the lighting effect control signal. The light-emitting element can be a light-emitting element capable of emitting monochromatic light or three light-emitting elements capable of emitting RGB (red, green, blue) three colors of light.

[0057] It can be known from the typical embodiments of the present application that the technical solution of the present application has many advantages, including but not limited to the following aspects:

[0058] This application first extracts image frames from the video stream generated by the camera device shooting the display, and determines the display area where the display emits light and the color picking point information of each color picking point set around the display area for each image frame, to ensure real-time and accurate capture of the display content of the display, and determine the color picking points on the edge of the corresponding display area, laying the foundation for subsequent lighting effect color picking.

[0059] Next, the display area is divided into multiple sub-areas based on the color picking point information, and the multiple sub-areas arranged along the circumference of the display area are respectively used as lighting effect color picking areas, and the target color corresponding to each lighting effect color picking area is sampled, so as to construct lighting effect control information and output it to the atmosphere lighting device, control the atmosphere lighting device to emit light accordingly, and realize the conversion of the target color corresponding to the display content to the corresponding atmosphere lighting effect, extending the atmosphere lighting effect to the environment outside the display, and being able to dynamically adapt to the changes in the display content in real time to present matching atmosphere lighting effects, creating a natural and beautiful viewing atmosphere, and significantly improving the user's visual experience and satisfaction.

[0060] In a further embodiment, step S1300, color sampling is performed on each lighting effect color picking area to determine a target color corresponding to each lighting effect color picking area, comprises the following steps:

[0061] Step S1310: for each lighting effect color picking area, sampling is performed on every other pixel in the lighting effect color picking area, and an average color is determined according to the total number of sampled pixels and the colors of the pixels;

[0062] The alternate pixel sampling is to sample the color of a pixel in the light effect color selection area each time, and the color is usually expressed as an RGB value, and there is one pixel between the pixels corresponding to two consecutive samples. The colors of all the sampled pixels are added together and divided by the total number of these pixels to obtain the average color.

[0063] Step S1320: perform color enhancement on the average color to obtain a target color of the lighting effect color picking area.

[0064] Since the display area captured by the camera device is darker in color than the actual display area, in order to approach the real visual effect, the average color is enhanced. In one embodiment, the average color represented by the RGB value is converted to an average color represented by the HSV value, and the values ​​of the H and S components are kept unchanged, and the value of the V component is added with a preset incremental value to increase the brightness. After that, the color represented by the corresponding HSV value is converted to a color represented by an RGB value as the target color of the lighting effect color picking area. The preset incremental value can be set by a technician in this field as needed, or it can be customized by the user. In other embodiments, the preset incremental values ​​of the H and / or S components can also be added to optimize the personalized visual effect, and the technicians in this field can flexibly implement it.

[0065] In this embodiment, each lighting effect color picking area is sampled at alternate pixel points, and the corresponding average color is obtained and then the color is enhanced to obtain the target color. On the one hand, since the colors of the pixels separated by the sampling intervals can reduce the amount of calculation, while still being able to obtain enough colors to represent the entire lighting effect color picking area. On the other hand, it can make up for the error between the camera shooting and the actual imaging, so that the subsequent lighting effect display effect is better.

[0066] In a further embodiment, step S1100, determining a display area in an image frame that belongs to an imaging range of the display, and color picking point information of each color picking point set around the display area, comprises the following steps:

[0067] Step S1110: using a preset image segmentation model to perform target detection and instance segmentation on the image frame, and determining the display area in the image frame and its edge point information to be completed, as well as the anchor frame area of ​​the display area in the frame;

[0068] The image segmentation model is pre-trained to a convergence state, and learns to determine the display area and its edge point information that belong to the imaging range of the display in the input image frame, as well as the anchor frame area of ​​the display area in the frame. The image segmentation model is recommended to be a YOLO series model such as YOLOV8, or other models that are also suitable for target detection and instance segmentation, such as U-Net, DeepLab series, BiSeNet, etc., and those skilled in the art can select and implement as needed.

[0069] The image segmentation model is used to take the image frame as input, and the image frame is subjected to target detection and instance segmentation. The image segmentation model extracts the image semantic features corresponding to the image frame by performing convolution operation on the image frame, including the color, texture, edge, contour, deep complex and abstract semantic information of the image frame, etc., and generates a corresponding multi-scale feature map. The display area in the image frame belonging to the imaging range of the display is used as the target detection object, and multiple candidate frame areas are predicted on the feature map. The overlapping candidate frame areas are filtered by non-maximum suppression to obtain the anchor frame area of ​​the target detection object in the frame. In addition, the image segmentation model classifies each pixel in the image frame, constructs a pixel-level mask of the display area, thereby segmenting the display area, and performs edge detection on the display area to determine the coordinate information of each edge point surrounding the edge of the display area to form edge point information. The coordinate information includes the corresponding horizontal coordinate and vertical coordinate of the corresponding edge point in the anchor frame area.

[0070] Step S1120: after linearly completing the edge point information to be completed, performing noise reduction processing on the corresponding edge contour point information to obtain edge framework point information;

[0071] Specifically, for the completion, for the coordinate information of each two adjacent edge points in the edge point information to be completed, the edge point with a relatively smaller horizontal coordinate and / or vertical coordinate among the two edge points is taken as the starting point, and the other edge point is taken as the ending point, and the Euclidean distance, the absolute value of the horizontal coordinate difference, and the absolute value of the vertical coordinate difference of the two edge points are calculated. When the Euclidean distance is greater than or equal to the preset spacing, the Euclidean distance is divided by the preset spacing to obtain a corresponding arithmetic result, and the arithmetic result minus 1 is the total number of edge points required to be supplemented between the two edge points. Then, a series of parameters uniformly distributed between 0 and 1 (excluding 0 and 1) are generated according to the arithmetic result. For each parameter, the parameter is multiplied by the absolute value of the horizontal coordinate difference and the absolute value of the vertical coordinate difference, and then the parameters are added to the horizontal coordinate and the vertical coordinate of the starting point respectively, to obtain the coordinate information of a single edge point required to be supplemented between the starting point and the ending point. When the Euclidean distance is less than the preset spacing, it means that no additional edge point needs to be supplemented between the two adjacent edge points. The preset spacing can be set by those skilled in the art as needed. For ease of understanding, an exemplary example is given, the coordinate information of the starting point is (x1: 0, y1: 20), the coordinate information of the end point is (x2: 50, y2: 40), the preset spacing is 5, the Euclidean distance between the two edge points is about 53.86, the absolute value of the horizontal coordinate difference is 50, and the absolute value of the vertical coordinate difference is 20. The Euclidean distance is divided by the preset spacing and rounded down to get the corresponding arithmetic result of 10. The total number of edge points to be added between the two edge points is 9, and the parameters are uniformly distributed between 0 and 1 (excluding 0 and 1). They are 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, and 0.9 respectively. The coordinate information of the edge points that need to be filled in are (x:5, y:22), (x:10, y:24), (x:15, y:26), (x:20, y:28), (x:25, y:30), (x:30, y:32), (x:35, y:34), (x:40, y:36), and (x:45, y:38).

[0072] Furthermore, the Ramer-Douglas-Peucker algorithm (full name Douglas Peucker algorithm, abbreviated as RDP algorithm) is used to perform noise reduction processing on the edge contour point information, remove the coordinate information of redundant edge points in the edge contour point information, retain the coordinate information of multiple edge points that constitute the edge contour of the display imaging and do not affect its true shape, and obtain the edge framework point information.

[0073] Step S1130, determining each color point region and its regional contour point information in the anchor frame region according to the preset regional positioning features and the edge frame point information;

[0074] The regional positioning feature includes the position distribution information of the color picking point area where each color picking point is located in the anchor frame area, and the position distribution information includes the corresponding horizontal coordinate range and vertical coordinate range of the corresponding color picking point area distribution in the anchor frame area. Those skilled in the art can set the regional positioning feature according to prior knowledge or experimental data, or can implement it according to the relevant disclosure in the subsequent embodiments.

[0075] The width of the anchor frame area is the maximum horizontal coordinate in the edge contour point information minus the minimum horizontal coordinate, and the height of the anchor frame area is the maximum vertical coordinate in the edge contour point information minus the minimum vertical coordinate.

[0076] In one embodiment, if the image frame is captured by a single fisheye camera, 8 color picking points are set according to the display area of ​​the complete imaging of the surround display, corresponding to the four corner points, two long side midpoints, and two wide side midpoints at the edge of the display, respectively, and the position distribution information corresponding to the color picking point area where each color picking point is located in the anchor frame area is preset, please refer to Figure 3 They are {horizontal axis range: [x min , 1 / 8*x max ], vertical coordinate range: [1 / 3*y max ,y max ]}(P1), {horizontal axis range: [1 / 32*x max ,7 / 32*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max ]}(P2), {horizontal axis range: [1 / 8*x max , 1 / 4*x max ], vertical coordinate range: [2 / 3*y max ,y max ]}(P3), {horizontal axis range: [3 / 8*x max , 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}(P4), {horizontal axis range: [5 / 8*x max , 3 / 4*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}(P5), {horizontal axis range: [21 / 32*x max , 31 / 32*x max ], vertical coordinate range: [3 / 12*y max , 11 / 12*y max ]}(P6), {horizontal axis range: [7 / 8*x max, x max ], vertical coordinate range: [1 / 6*y max ,y max ]} (P7), {horizontal axis range: [3 / 8*x max , 5 / 8*x max ], vertical coordinate range: [y min , 1 / 6*y max ]}(P8), all the location distribution information is collected to form the regional positioning feature. max 、x min ,y min ,y max They are the minimum horizontal coordinate, maximum horizontal coordinate, minimum vertical coordinate, and maximum vertical coordinate in the edge contour point information.

[0077] If the image frame is taken by the left fisheye camera of the camera device of the dual fisheye camera, 5 color picking points are set according to the display area of ​​the left half of the image surrounding the display, corresponding to two corner points on the left edge of the display, a midpoint of the upper long side, a midpoint of the left wide side, and a point located to the left of the midpoint of the lower long side and at a preset distance from the point, and the position distribution information corresponding to the color picking point area where each color picking point is located in the anchor frame area is preset, please refer to Figure 4 They are {horizontal axis range: [3 / 4*x max， x max ], vertical coordinate range: [y min , 1 / 6*y max ]}(P1), {horizontal axis range: [x min , 1 / 8*x max ], vertical axis range: [1 / 2y min , 2 / 3*y max ]}(P2), {horizontal axis range: [1 / 16*x max， 9 / 16*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max ]}(P3), {horizontal axis range: [3 / 8*x max， 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}(P4), {horizontal axis range: [3 / 4*x max， x max ], vertical coordinate range: [5 / 6*y max ,y max ]}(P5), all the location distribution information is collected to form the regional positioning feature. max 、x min ,y min ,ymax They are the minimum horizontal coordinate, maximum horizontal coordinate, minimum vertical coordinate, and maximum vertical coordinate in the edge contour point information.

[0078] If the image frame is taken by the right fisheye camera of the dual fisheye camera camera device, 5 color picking points are set according to the display area of ​​the right half of the image surrounding the display, corresponding to two corner points on the right edge of the display, a midpoint of the upper long side, a midpoint of the right wide side, and a point to the right of the midpoint of the lower long side and at a preset distance from the point, and the position distribution information corresponding to the color picking point area where each color picking point is located in the anchor frame area is preset, please refer to Figure 5 They are {horizontal axis range: [x min , 1 / 4*x max ], vertical coordinate range: [y min , 1 / 6*y max ]}(P1), {horizontal axis range: [x min , 1 / 4*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}(P2), {horizontal axis range: [3 / 8*x max， 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}(P3), {horizontal axis range: [7 / 16*x max， 15 / 16*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max ]}(P4), {horizontal axis range: [7 / 8*x max， x max ], vertical coordinate range: [1 / 2*y max , 5 / 6*y max ]}(P5), all the location distribution information is collected to form the regional positioning feature. max 、x min ,y min ,y max They are the minimum horizontal coordinate, maximum horizontal coordinate, minimum vertical coordinate, and maximum vertical coordinate in the edge contour point information.

[0079] It is not difficult to understand that, according to the position distribution information of each color picking point area in the regional positioning feature, each color picking point area can be determined in the anchor frame area, and then, according to the horizontal coordinate range and vertical coordinate range corresponding to each color picking point area, the coordinate information of each edge point in the edge frame point information is traversed to determine the color picking point area to which each edge point belongs, and then for each color picking point area, all edge points belonging to the color picking point area are respectively used as single regional contour points, and the coordinate information of these regional contour points are collected to constitute the regional contour point information of the color picking point area.

[0080] Step S1140: According to the preset regional fixed-point features and the contour point information of each region, the coordinate information of the color picking points in each color picking point region is determined to constitute the color picking point information.

[0081] The regional fixed-point features include locating the coordinate distribution patterns of the color picking points in each color picking point area. Those skilled in the art may set the regional fixed-point features based on prior knowledge or experimental data, or may implement them according to the relevant disclosures in some subsequent embodiments.

[0082] If the image frame is captured by a camera device using a single fisheye camera, for {abscissa range: [3 / 8*x max , 5 / 8*x max ], vertical coordinate range: [y min , 1 / 6*y max ]} color picking point area, determine the area contour point information of the color picking point area, the horizontal coordinate is the value obtained by multiplying the width of the anchor frame area by the first preset ratio and adding the minimum horizontal coordinate in the edge contour information, and the vertical coordinate is the minimum vertical coordinate in the area contour point information, and the corresponding coordinate information of the area contour point is used as the coordinate information of the color picking point of the color picking point area. The first preset ratio can be flexibly set by technical personnel in the field according to the display size and the shooting angle of the single-lens camera, for example, 50%; for {horizontal coordinate range: [3 / 8*x max , 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,y max ]} color picking point area, determine the area contour information of the color picking point area, the horizontal coordinate is the value obtained by multiplying the width of the anchor frame area by the first preset ratio and adding the minimum horizontal coordinate in the edge contour information, and the vertical coordinate is the area contour point corresponding to the maximum vertical coordinate in the area contour point information, and the coordinate information of the corresponding area contour point is used as the coordinate information of the color picking point of the color picking point area; for {horizontal coordinate range: [x min , 1 / 8*x max ], vertical coordinate range: [1 / 3*y max ,y max]} color picking point area, in one embodiment, the coordinate information of the area contour point with the smallest horizontal coordinate in the area contour information of the color picking point area is determined, or the coordinate information of the area contour point with the largest vertical coordinate among the N area contour points with smaller horizontal coordinates in the area contour information is determined as the coordinate information of the color picking point of the color picking point area. The N can be set as needed by those skilled in the art, for example, 5; for {horizontal coordinate range: [7 / 8*x max , x max ], vertical coordinate range: [1 / 6*y max ,y max ]} color picking point area, in one embodiment, the coordinate information of the area contour point with the largest horizontal coordinate in the area contour information of the color picking point area is determined, or the coordinate information of the area contour point with the largest vertical coordinate among the N area contour points with larger horizontal coordinates in the area contour information is determined as the coordinate information of the color picking point of the color picking point area; for {horizontal coordinate range: [1 / 8*x max , 1 / 4*x max ], vertical coordinate range: [2 / 3*y max ,y max ]} color point area, {horizontal axis range: [5 / 8*x max , 3 / 4*x max ], vertical coordinate range: [5 / 6*y max ,y max ]} color picking point area, first according to the edge contour of the display imaging clockwise around, the coordinate information of each edge point in the edge frame point information is sorted accordingly to obtain an edge point sequence, from the edge point sequence, the vector formed by each edge point in the edge point sequence and its next edge point is determined in sequence, and each two vectors obtained are adjacent in position, thus, the arc cosine value between each two vectors is calculated to be the angle between the two vectors, and then for each color picking point area, the coordinate information of the intersection between the two adjacent vectors with the largest angle is determined from the regional edge point position information of the color picking point area, as the color picking point position information of the color picking point area; for {horizontal coordinate range: [1 / 32*x max ,7 / 32*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max ]} color point area, {horizontal axis range: [21 / 32*x max , 31 / 32*x max ], vertical coordinate range: [3 / 12*y max , 11 / 12*y max]} color picking point area, for each color picking point area, determine the coordinate information of the area contour points with the same distance as two color picking points adjacent to the color picking point area as the coordinate information of the color picking points in the color picking point area.

[0083] If the image frame is taken by the left fisheye camera of the dual fisheye camera camera device, for {abscissa range: [3 / 4*x max , x max ], vertical coordinate range: [y min , 1 / 6*y max ]} color picking point area, determine the area contour point information of the color picking point area, the horizontal coordinate is the value obtained by multiplying the width of the anchor frame area by the second preset ratio and adding the minimum horizontal coordinate in the edge contour information, and the vertical coordinate is the minimum vertical coordinate in the area contour point information, and the corresponding coordinate information of the area contour point is used as the coordinate information of the color picking point of the color picking point area. The second preset ratio can be flexibly set by technical personnel in the field according to the display size and the shooting angle of the single-lens camera, for example, 68%; for {horizontal coordinate range: [3 / 4*x max , x max ], vertical coordinate range: [5 / 6*y max ,y max ]} color picking point area, determine the area contour information of the color picking point area, the horizontal coordinate is the value obtained by multiplying the width of the anchor frame area by the second preset ratio and adding the minimum horizontal coordinate in the edge contour information, and the vertical coordinate is the area contour point corresponding to the maximum vertical coordinate in the area contour point information, and the coordinate information of the corresponding area contour point is used as the coordinate information of the color picking point of the color picking point area; for {horizontal coordinate range: [x min , 1 / 8*x max ], vertical axis range: [1 / 2y min , 2 / 3*y max ]} color picking point area, in one embodiment, the coordinate information of the area contour point with the smallest horizontal coordinate in the area contour information of the color picking point area is determined, or the coordinate information of the area contour point with the largest vertical coordinate among the N area contour points with smaller horizontal coordinates in the area contour information is determined as the coordinate information of the color picking point of the color picking point area. The N can be set as needed by those skilled in the art, for example, 5; for {horizontal coordinate range: [3 / 8*x max , 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,y max]} color picking point area, first according to the clockwise edge contour of the display imaging, the coordinate information of each edge point in the edge frame point information is sorted accordingly to obtain an edge point sequence, from the edge point sequence, the vector formed by each edge point in the edge point sequence and its next edge point is determined in sequence, and each two vectors obtained are adjacent in position, thus, the arc cosine value between each two vectors is calculated to be the angle between the two vectors, and then for the color picking point area, the coordinate information of the intersection between the two adjacent vectors with the largest angle is determined from the regional edge point position information of the color picking point area, as the color picking point position information of the color picking point area; for {horizontal coordinate range: [1 / 16*x max , 9 / 16*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max ]} color picking point area, determine the coordinate information of the area contour point whose distance ratio of two color picking points adjacent to the color picking point area belongs to the first preset value range, as the coordinate information of the color picking point in the color picking point area. The first preset value range can be flexibly set by those skilled in the art according to the size of the display. For example, the first preset value range corresponding to a 65-inch display can be [0.52-0.55].

[0084] If the image frame is taken by the right fisheye camera of the dual fisheye camera camera device, for {horizontal coordinate range: [x min , 1 / 4*x max ], vertical coordinate range: [y min , 1 / 6*y max ]} color picking point area, determine the area contour point information of the color picking point area, the horizontal coordinate is the value obtained by multiplying the width of the anchor frame area by the s third preset ratio plus the minimum horizontal coordinate in the edge contour information, and the vertical coordinate is the minimum vertical coordinate in the area contour point information, and the corresponding area contour point coordinate information is used as the coordinate information of the color picking point of the color picking point area. The third preset ratio can be flexibly set by technical personnel in the field according to the display size and the shooting angle of the single-lens camera, such as 32%; for {horizontal coordinate range: [x min , 1 / 4*x max ], vertical coordinate range: [5 / 6*y max ,y max ]} color picking point area, determine the area contour information of the color picking point area, the horizontal coordinate is the value obtained by multiplying the width of the anchor frame area by the third preset ratio and adding the minimum horizontal coordinate in the edge contour information, and the vertical coordinate is the area contour point corresponding to the maximum vertical coordinate in the area contour point information, and the coordinate information of the corresponding area contour point is used as the coordinate information of the color picking point of the color picking point area; for {horizontal coordinate range: [7 / 8*x max, x max ], vertical coordinate range: [1 / 2*y max , 5 / 6*y max ]} color picking point area, in one embodiment, the coordinate information of the area contour point with the largest horizontal coordinate in the area contour information of the color picking point area is determined, or the coordinate information of the area contour point with the largest vertical coordinate among the N area contour points with larger horizontal coordinates in the area contour information is determined as the coordinate information of the color picking point in the color picking point area. The N can be set as needed by those skilled in the art, for example, 5; for {horizontal coordinate range: [3 / 8*x max , 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,y max ]} color picking point area, first according to the clockwise edge contour of the display imaging, the coordinate information of each edge point in the edge frame point information is sorted accordingly to obtain an edge point sequence, from the edge point sequence, the vector formed by each edge point in the edge point sequence and its next edge point is determined in sequence, and each two vectors obtained are adjacent in position, thus, the arc cosine value between each two vectors is calculated to be the angle between the two vectors, and then for the color picking point area, the coordinate information of the intersection between the two adjacent vectors with the largest angle is determined from the regional edge point position information of the color picking point area, as the color picking point position information of the color picking point area; for {horizontal coordinate range: [1 / 16*x max , 9 / 16*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max ]} color picking point area, determine the coordinate information of the area contour point whose distance ratio of two color picking points adjacent to the color picking point area belongs to the second preset value range, as the coordinate information of the color picking point in the color picking point area. The second preset value range can be flexibly set by those skilled in the art according to the size of the display. For example, the first preset value range corresponding to a 65-inch display can be [0.45-0.48].

[0085] In the present embodiment, an image segmentation model is used to perform target detection and instance segmentation on an image frame, and the display area in the image frame and its edge point information to be completed, as well as the anchor frame area of ​​the display area in the frame are determined. A linear interpolation algorithm is used to complete the edge point information to be completed, and the edge contour point information is obtained. According to preset regional positioning features and the edge contour point information, each color picking point area in the anchor frame area and its regional contour point information are determined. According to preset regional fixed-point features and each regional contour point information, the coordinate information of the color picking points in each color picking point area is determined to constitute the color picking point information. Based on each edge point of the complete edge contour of the display area, the accuracy and reliability of the coordinate information of each color picking point fixed from each color picking point area can be ensured.

[0086] In a further embodiment, step S1140, determining the coordinate information of the color picking points in each color picking point area to constitute the color picking point information according to the preset regional fixed point features and the contour point information of each area, includes any one or more of the following steps:

[0087] Step S1141, for each color picking point area including a color picking point belonging to the midpoint of the long side of the display, according to the position of the color picking point area in the anchor frame area, determine the coordinate information of the color picking point in the color picking point area from the area contour point information of the color picking point area;

[0088] If the image frame is captured by a camera device using a single fisheye camera, and the color point area is located directly above the anchor frame area ({horizontal coordinate range: [3 / 8*x max , 5 / 8*x max ], vertical coordinate range: [y min , 1 / 6*y max ]}), from the regional contour point information of the color picking point area, determine the value obtained by multiplying the width of the anchor frame area by the first preset ratio and adding the minimum horizontal coordinate in the edge contour information, and the vertical coordinate is the regional contour point corresponding to the minimum vertical coordinate in the regional contour point information, as the color picking point of the color picking point area, and use the coordinate information of the regional contour point as the coordinate information of the color picking point of the color picking point area. The first preset ratio can be flexibly set by technical personnel in the field according to the display size and the shooting angle of the monocular camera, for example, 50%.

[0089] If the image frame is captured by a camera device using a single fisheye camera, and the color point area is located directly below the anchor frame area ({horizontal coordinate range: [3 / 8*x max , 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,y max]}), from the area contour point information of the color picking point area, determine the area contour point with the horizontal coordinate being the value obtained by multiplying the width of the anchor frame area by the first preset ratio and adding the value obtained by adding the minimum horizontal coordinate in the edge contour information, and the vertical coordinate being the area contour point corresponding to the maximum vertical coordinate in the area contour point information, as the color picking point of the color picking point area, and use the coordinate information of the area contour point as the coordinate information of the color picking point of the color picking point area.

[0090] If the image frame is captured by the left fisheye camera of the camera device using dual fisheye cameras, and the color point area is located at the upper right corner of the anchor frame area ({horizontal coordinate range: [3 / 4*x max， x max ], vertical coordinate range: [y min , 1 / 6*y max ]}), from the regional contour information of the color picking point area, determine the value obtained by multiplying the width of the anchor frame area by the second preset ratio and adding the minimum horizontal coordinate in the edge contour information, and the vertical coordinate is the regional contour point corresponding to the minimum vertical coordinate in the regional contour point information, as the color picking point of the color picking point area, and use the coordinate information of the regional contour point as the coordinate information of the color picking point of the color picking point area. The second preset ratio can be flexibly set by technical personnel in the field according to the display size and the shooting angle of the left fisheye camera, for example, 68%.

[0091] If the image frame is captured by the left fisheye camera of the camera device using dual fisheye cameras, and the color point area is located at the lower right corner of the anchor frame area ({horizontal coordinate range: [3 / 4*x max， x max ], vertical coordinate range: [5 / 6*y max ,y max ]}), from the area contour information of the color picking point area, determine the area contour point with the horizontal coordinate being the value obtained by multiplying the width of the anchor frame area by the second preset ratio and adding the value obtained by adding the minimum horizontal coordinate in the edge contour information, and the vertical coordinate being the area contour point corresponding to the maximum vertical coordinate in the area contour point information, as the color picking point of the color picking point area, and use the coordinate information of the area contour point as the coordinate information of the color picking point of the color picking point area.

[0092] If the image frame is captured by the right fisheye camera of the camera device using dual fisheye cameras, and the color point area is located at the upper left corner of the anchor frame area ({horizontal coordinate range: [x min , 1 / 4*x max ], vertical coordinate range: [y min , 1 / 6*y max]}), from the regional contour information of the color picking point area, determine the value obtained by multiplying the width of the anchor frame area by the third preset ratio and adding the minimum horizontal coordinate in the edge contour information, and the vertical coordinate is the regional contour point corresponding to the minimum vertical coordinate in the regional contour point information, as the color picking point of the color picking point area, and use the coordinate information of the regional contour point as the coordinate information of the color picking point of the color picking point area. The third preset ratio can be flexibly set by technical personnel in the field according to the display size and the shooting angle of the right fisheye camera, for example, 32%.

[0093] If the image frame is captured by the right fisheye camera of the camera device using dual fisheye cameras, and the color point area is located at the lower left corner of the anchor frame area ({horizontal coordinate range: [x min , 1 / 4*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}), from the area contour information of the color picking point area, determine the area contour point with the horizontal coordinate being the value obtained by multiplying the width of the anchor frame area by the third preset ratio and adding the value obtained by adding the minimum horizontal coordinate in the edge contour information, and the vertical coordinate being the area contour point corresponding to the maximum vertical coordinate in the area contour point information, as the color picking point of the color picking point area, and use the coordinate information of the area contour point as the coordinate information of the color picking point of the color picking point area.

[0094] Step S1142: for each color picking point area including the color picking points belonging to the corner points of the display, determine the vector formed by every two adjacent edge points and the angle between every two adjacent vectors according to the edge frame point information, and determine the coordinate information of the intersection between two adjacent vectors with the largest angle from the regional edge point position information of the color picking point area as the color picking point position information of the color picking point area;

[0095] If the image frame is captured by a camera device using a single fisheye camera, each color point area including the color points belonging to the corner points of the display is respectively in the anchor frame area {horizontal coordinate range: [x min , 1 / 8*x max ], vertical coordinate range: [1 / 3*y max ,y max ]}, {horizontal axis range: [1 / 8*x max , 1 / 4*x max ], vertical coordinate range: [2 / 3*y max ,y max ]}, {horizontal axis range: [5 / 8*x max , 3 / 4*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}, {horizontal axis range: [7 / 8*xmax , x max ], vertical coordinate range: [1 / 6*y max ,y max ]}, firstly, according to the edge contour of the imaging of the display in a clockwise direction, the coordinate information of each edge point in the edge frame point information is sorted accordingly to obtain an edge point sequence, and from the edge point sequence, the vectors formed by each edge point in the edge point sequence and its next edge point are determined in order, and each two vectors obtained are adjacent in position, thereby calculating the arc cosine value between each two vectors, which is the angle between the two vectors, and then for each color picking point area, the coordinate information of the intersection between the two adjacent vectors with the largest angle is determined from the regional edge point position information of the color picking point area, as the color picking point position information of the color picking point area.

[0096] If the image frame is captured by the left fisheye camera of the camera device using dual fisheye cameras, each color point area including the color points belonging to the corner points of the display is respectively in the anchor frame area {horizontal coordinate range: [x min , 1 / 8*x max ], vertical axis range: [1 / 2y min , 2 / 3*y max ]}, {horizontal axis range: [3 / 8*x max， 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}, firstly, according to the edge contour of the imaging of the display in a clockwise direction, the coordinate information of each edge point in the edge frame point information is sorted accordingly to obtain an edge point sequence, and from the edge point sequence, the vectors formed by each edge point in the edge point sequence and its next edge point are determined in order, and each two vectors obtained are adjacent in position, thereby calculating the arc cosine value between each two vectors, which is the angle between the two vectors, and then for each color picking point area, the coordinate information of the intersection between the two adjacent vectors with the largest angle is determined from the regional edge point position information of the color picking point area, as the color picking point position information of the color picking point area.

[0097] If the image frame is captured by the right fisheye camera of the camera device using dual fisheye cameras, each color point area including the color points belonging to the corner points of the display is respectively in the anchor frame area {abscissa range: [3 / 8*x max， 5 / 8*x max ], vertical coordinate range: [5 / 6*y max ,y max ]}, {horizontal axis range: [7 / 8*x max， x max ], vertical coordinate range: [1 / 2*ymax , 5 / 6*y max ]}, firstly, according to the edge contour of the imaging of the display in a clockwise direction, the coordinate information of each edge point in the edge frame point information is sorted accordingly to obtain an edge point sequence, and from the edge point sequence, the vectors formed by each edge point in the edge point sequence and its next edge point are determined in order, and each two vectors obtained are adjacent in position, thereby calculating the arc cosine value between each two vectors, which is the angle between the two vectors, and then for each color picking point area, the coordinate information of the intersection between the two adjacent vectors with the largest angle is determined from the regional edge point position information of the color picking point area, as the color picking point position information of the color picking point area.

[0098] Step S1143, for each color picking point area that includes a color picking point belonging to the midpoint of the wide side of the display, based on the coordinate information of two color picking points adjacent to the color picking point area, determine from the area contour point information of the color picking point area the coordinate information of the area contour point whose distance ratio with the two color picking points meets a preset condition, as the coordinate information of the color picking point in the color picking point area.

[0099] If the image frame is captured by a camera device using a single fisheye camera, each color point area including the color point belonging to the midpoint of the wide side of the display is respectively in the anchor frame area {abscissa range: [1 / 32*x max ,7 / 32*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max ]}, {horizontal axis range: [21 / 32*x max , 31 / 32*x max ], vertical coordinate range: [3 / 12*y max , 11 / 12*y max ]}, for each color picking point area, determine the coordinate information of the area contour points with the same distance as two color picking points adjacent to the color picking point area from the area contour point information of the color picking point area, as the coordinate information of the color picking points in the color picking point area.

[0100] If the image frame is captured by the left fisheye camera of the camera device using dual fisheye cameras, each color point area including the color point belonging to the midpoint of the wide side of the display is respectively in the anchor frame area {abscissa range: [1 / 16*x max， 9 / 16*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max]}, from the regional contour point information of the color picking point area, determine the coordinate information of the regional contour points whose distance ratio of two color picking points adjacent to the color picking point area belongs to the first preset value range, as the coordinate information of the color picking points in the color picking point area. The first preset value range can be flexibly set by those skilled in the art according to the size of the display. For example, the first preset value range corresponding to a 65-inch display can be [0.52-0.55].

[0101] If the image frame is captured by the right fisheye camera of the camera device using dual fisheye cameras, each color point area including the color point belonging to the midpoint of the wide side of the display is respectively in the anchor frame area {abscissa range: [7 / 16*x max， 15 / 16*x max ], vertical coordinate range: [5 / 12*y max , 11 / 12*y max ]}, from the regional contour point information of the color picking point area, determine the coordinate information of the regional contour points whose distance ratio of two color picking points adjacent to the color picking point area belongs to the second preset value range, as the coordinate information of the color picking points in the color picking point area. The second preset value range can be flexibly set by those skilled in the art according to the size of the display. For example, the second preset value range corresponding to a 65-inch display can be [0.45-0.48].

[0102] In this embodiment, the implementation process of locating the color picking points in each color picking point area is disclosed, ensuring that each color picking point corresponds to the midpoint of the long side, the midpoint of the wide side, and the corner point of the edge of the display.

[0103] In a further embodiment, step S1300, color sampling is performed on each lighting effect color picking area to determine a target color corresponding to each lighting effect color picking area, comprises the following steps:

[0104] Step S1301: for each lighting effect color picking area, determine the core hue interval belonging to the lighting effect color picking area, and the colors of multiple pixels whose corresponding hue values ​​meet preset conditions, and calculate the corresponding average color;

[0105] Taking a single lighting effect color picking area as an example, the lighting effect color picking area is sampled at alternate pixel points to determine the color of each corresponding pixel point, which is usually represented by an RGB value, and the color represented by the RGB value of each pixel point is converted into a corresponding color represented by an HSV value. Obtain a plurality of preset hue intervals, which can be divided into [0°, 360°] as required by a technician in this field, for example, divided into 6 hue intervals, namely [0°, 60°), [60°, 120°), [120°, 180°), [180°, 240°), [240°, 300°), [300°, 360°]. According to the hue value of the color represented by the HSV value of each pixel point, the hue interval to which the color represented by the HSV value of each pixel point belongs can be determined. For each hue interval, the total number of pixels belonging to the hue interval is determined, so as to select the hue interval with the largest total number of pixels as the core hue area. The colors represented by HSV of all pixels belonging to the core hue interval are sorted in descending order of hue value, and the colors represented by HSV of the top N pixels are selected. The colors represented by HSV of these pixels are converted to colors represented by RGB and then the average is calculated as the average color of the lighting effect color selection area. The N can be set as needed by those skilled in the art, for example, 10.

[0106] Step S1302: perform color enhancement on the average color to obtain a target color of the lighting effect color picking area.

[0107] Since the display effect captured by the camera equipment is darker in color than the actual display effect, in order to approach the real visual effect, the average color is enhanced, and the incremental value of any one component of R, G, B preset by the user or the incremental values ​​of any multiple components are added to the value of the same component of the average color to obtain the target color of the lighting effect color picking area.

[0108] In this embodiment, each lighting effect color picking area is sampled at alternate pixel points, and the average color corresponding to the core hue interval of the lighting effect color picking area is obtained and then the color is enhanced to obtain the target color. On the one hand, since the color of the pixels separated by the sampling interval can reduce the amount of calculation, while still being able to obtain enough colors to represent the entire lighting effect color picking area, on the other hand, it can make up for the error between the camera shooting and the actual imaging, so that the subsequent lighting effect display effect is better.

[0109] In a further embodiment, step S1143, for each color picking point area including a color picking point belonging to the midpoint of the wide side of the display, according to the coordinate information of two color picking points adjacent to the color picking point area, from the area contour point information of the color picking point area, the coordinate information of the area contour point whose distance ratio with the two color picking points meets the preset conditions is determined as the coordinate information of the color picking point in the color picking point area, including the following steps:

[0110] Step S11431, when the image frame belongs to a single-eye shooting type, determining coordinate information of area contour points that are adjacent to the color picking point area and have the same distance as the two color picking points, as coordinate information of the color picking points in the color picking point area;

[0111] It can be understood that different camera identification codes can be preset for a camera device using a single fisheye camera, a left fisheye camera of a camera device using a dual fisheye camera, and a right fisheye camera of a camera device using a dual fisheye camera. Thus, when it is identified that the camera identification code of the camera device from which the image frame comes is the camera identification code of a single fisheye camera, it is determined that the image frame belongs to a single-eye shooting type. At this time, according to the coordinate information of the color picking points in the two color picking point areas adjacent to the color picking point area, from the coordinate information of the area contour points of the color picking point area, the area contour points with the same Euclidean distance corresponding to the two color picking points are determined as the color picking points in the color picking point area, and the coordinate information of the area contour points is used as the coordinate information of the color picking points in the color picking point area.

[0112] Step S11432: when the image frame belongs to the left-side shooting type of both eyes, coordinate information of the area contour points whose distance ratio between two color picking points adjacent to the color picking point area belongs to the first preset value range is determined as the coordinate information of the color picking points in the color picking point area;

[0113] When it is identified that the camera identification code of the camera device of the image frame source is the camera identification code of the left eye camera, it is determined that the image frame belongs to the left-side shooting type of both eyes. At this time, according to the coordinate information of the color picking points in the two color picking point areas adjacent to the color picking point area, each area contour point is arranged in the order of the horizontal coordinate from large to small from the coordinate information of the area contour points of the color picking point area, and each area contour point is traversed in this order, and the Euclidean distance between the area contour point and the color picking point on its left is divided by the Euler distance between the area contour point and the color picking point on its right to obtain the distance ratio. When the distance ratio belongs to the first preset value range, the area contour point is used as the color picking point in the color picking point area, and the coordinate information of the area contour point is used as the coordinate information of the color picking point in the color picking point area, and the traversal is stopped; when the distance ratio does not belong to the first preset value range, the next area contour point is traversed.

[0114] The first preset value range can be flexibly set by those skilled in the art according to the size of the display. For example, the first preset value range corresponding to a 65-inch display may be [0.52-0.55].

[0115] Step S11433, when the image frame belongs to the binocular right side shooting type, determine the coordinate information of the area contour point whose distance ratio between two color picking points adjacent to the color picking point area belongs to the second preset value range as the coordinate information of the color picking points in the color picking point area.

[0116] When it is identified that the camera identification code of the camera device of the image frame source is the camera identification code of the right fisheye camera, it is determined that the image frame belongs to the binocular right side shooting type. At this time, according to the coordinate information of the color picking points in the two color picking point areas adjacent to the color picking point area, each area contour point is arranged in the order of the horizontal coordinate from large to small from the coordinate information of the area contour points of the color picking point area, and each area contour point is traversed in this order, and the Euclidean distance between the area contour point and the color picking point on its left is divided by the Euler distance between the area contour point and the color picking point on its right to obtain the distance ratio. When the distance ratio belongs to the second preset value range, the area contour point is used as the color picking point in the color picking point area, and the coordinate information of the area contour point is used as the coordinate information of the color picking point in the color picking point area, and the traversal is stopped; when the distance ratio does not belong to the second preset value range, the next area contour point is traversed.

[0117] The second preset value range can be flexibly set by those skilled in the art according to the size of the display. For example, the second preset value range corresponding to a 65-inch display may be [0.45-0.48].

[0118] In this embodiment, for image frames belonging to monocular shooting type, binocular left side shooting type, and binocular right side shooting type, respectively, a method for determining the coordinate information of the color picking points in the corresponding color picking point areas is revealed, which can accurately locate the color picking points according to different shooting angles.

[0119] In other embodiments, an image classification model pre-trained to a convergent state may be used to determine whether the image frame belongs to a binocular right-side shooting type, a binocular left-side shooting type, or a monocular shooting type. Those skilled in the art may flexibly implement the image classification model based on the disclosure herein.

[0120] In a further embodiment, before step S1110, performing target detection and instance segmentation on the image frame using a preset image segmentation model to determine the display area in the image frame and its edge point information to be completed, as well as the anchor frame area of ​​the display area in the frame, the method includes:

[0121] Step S1000, obtaining a preset training set, training the image segmentation model to a convergence state, so that it can learn to determine the display area and its edge point information belonging to the imaging range of the display, and the anchor frame area of ​​the display area in the frame from the input image, the training set includes multiple training samples and their supervisory labels, the training samples are images containing the display area, the supervisory labels mark the target edge detection information corresponding to the display area in the corresponding training samples, the training samples in the training set are obtained by photographing the display with a camera device using a single fisheye camera, and / or include photographing the display with a left fisheye camera of a camera device using a dual fisheye camera and a right fisheye camera of a camera device using a dual fisheye camera, respectively.

[0122] In the recommended embodiment, multiple images of the display produced by a camera device using a single fisheye camera in different environments are collected in advance, multiple images of the display produced by the left fisheye camera of a camera device using dual fisheye cameras in different environments are collected, and multiple images of the display produced by the right fisheye camera of the camera device in different environments are collected. It is not difficult to understand that each collected image contains a display area corresponding to the imaging range of the display, for example, the display area in the image produced by the single fisheye camera is the complete imaging of the display, the display area in the image produced by the left fisheye camera is the imaging of the left half of the display, and the display area in the image produced by the right fisheye camera is the imaging of the right half of the display. The different environments can be any one or any multiple environments such as different times, different objects placed next to the display, the display placed on different objects, the display attached to and hung on different objects, and the display in different spaces, so as to enhance the generalization ability, reliability, and accuracy of the image segmentation model.

[0123] Each image is used as a single training sample. Further, labelme and labelimg marking tools are used to mark each training sample. Taking a single training sample as an example, the display area in the image of the training sample is used as the target detection object. The labelimg marking tool is used to frame the specific position and size of the target detection object in the image. In addition, the labelme marking tool is used to set corresponding edge points along the edge contour of the target detection object in the image. The edge points are connected to each other to form an edge contour that just encloses the target detection object. After the labeling is completed, the labelme and labelimg marking tools generate a corresponding labeling file containing the target edge detection information of the display area as the supervisory label of the training sample. The target edge detection information includes the coordinate information of the center pixel of the display area in the corresponding image (also the coordinate information of the center pixel of the anchor frame of the display area in the frame in the image), the width and height of the display area in the image (also the width and height of the anchor frame in the image), and the coordinate information of each edge point of the edge contour of the display area in the image.

[0124] The image segmentation model is recommended to be a YOLO series model such as YOLOV8, or other models that are also suitable for target detection and instance segmentation, such as U-Net, DeepLab series, BiSeNet, etc. Those skilled in the art can select and implement them as needed.

[0125] A single training sample in the training set and its supervision label are called, and the image segmentation model uses the training sample as input to perform target detection and instance segmentation on the training sample. The image segmentation model extracts the image semantic features corresponding to the training sample by performing convolution operations on the training sample, including the color, texture, edge, contour, deep complex and abstract semantic information of the training sample, and generates a corresponding multi-scale feature map. The display area in the training sample is used as the target detection object to predict multiple candidate box areas on the feature map, and the overlapping candidate box areas are filtered by non-maximum suppression to predict the predicted anchor box area of ​​the target detection object in the box. In addition, the image segmentation model classifies each pixel in the training sample and constructs a pixel-level mask of the display area to segment the predicted display area, and performs edge detection on the predicted display area to determine the coordinate information of each edge point surrounding the edge of the predicted display area to form the predicted edge point information. A cross entropy loss function is used based on the supervisory labels of the training samples to calculate the loss values ​​corresponding to the cross entropy losses of the predicted anchor frame area, the predicted display area, and the predicted edge point information. When the loss values ​​are all lower than a preset threshold, it indicates that the image segmentation model has been trained to a convergence state, thereby terminating the training of the image segmentation model. When the loss value is greater than or equal to the preset threshold, it indicates that the image segmentation model has not converged, and then the image segmentation model is gradient updated according to the loss value. Usually, the weight parameters of each link in the image segmentation model are corrected by back propagation to make the image segmentation model further approach convergence. Then, other training samples in the training set and their supervisory labels are continuously called to perform iterative training on the image segmentation model until the image segmentation model is trained to a convergence state.

[0126] In this embodiment, the training process of the image segmentation model is disclosed to ensure the accuracy and reliability of the image segmentation model in identifying the display area and its edge point information, and can accordingly process images produced by photographing the display using a camera device using a single fisheye camera, a left fisheye camera of a camera device using a dual fisheye camera, and a right fisheye camera of a camera device using a dual fisheye camera.

[0127] In a further embodiment, step S1200, partitioning the display area according to the color picking point information to obtain a plurality of subdivided areas, and using the plurality of subdivided areas arranged along the circumference of the display area as lighting effect color picking areas, comprises the following steps:

[0128] Step S1210, using a fisheye correction algorithm to perform image correction on each subdivided area;

[0129] The fisheye correction algorithm of the OpenCV library can be used to correct each subdivided area, thereby realizing image correction of the display area in disguise. Of course, technical personnel in this field can also use other open source fisheye correction algorithms, such as a deep learning model that has been pre-trained to convergence, which learns through training to correct images captured by fisheye imaging.

[0130] Step S1220: For each corrected subdivided area arranged along the circumference of the display area, a central area of ​​the subdivided area is cut out as a lighting effect color selection area.

[0131] For each corrected subdivided area arranged along the circumference of the display area, considering that the imaging of the central part of each subdivided area is highly close to the imaging taken from a conventional viewing angle, that is, the error is minimal, the central part of each subdivided area is cut out as the lighting effect color selection area. The central part area can be set as needed by those skilled in the art, for example, the central part area accounts for 85% of the subdivided area, and the center points of the two areas are the same.

[0132] In this embodiment, by correcting the segmented image and then cropping out the central area to obtain the lighting effect color picking area, visual distortion can be corrected to ensure the color accuracy in the lighting effect color picking area, laying the foundation for subsequent precise color picking and display of lighting effects, and ensuring the user's viewing experience.

[0133] See also Figure 6 , an atmosphere lighting effect control device provided to meet one of the purposes of the present application is a functional embodiment of the atmosphere lighting effect control method of the present application. On the other hand, the device is an atmosphere lighting effect control device provided to meet one of the purposes of the present application, including an image processing module 1100, a color determination module 1200, a color sampling module 1300 and a lighting effect playback module 1400, wherein the image processing module 1100 is used to obtain an image frame in a video stream generated by a camera device shooting a display, determine a display area in the image frame that belongs to the imaging range of the display, and various color determination modules set around the display area. Color point information of a color point; a color determination module 1200, used to partition the display area according to the color point information to obtain a plurality of subdivided areas, and use the plurality of refined areas arranged circumferentially along the display area as lighting effect color picking areas; a color sampling module 1300, used to perform color sampling on each lighting effect color picking area, and determine a target color corresponding to each lighting effect color picking area; a lighting effect playback module 1400, used to construct a lighting effect control signal according to the target color corresponding to each lighting effect color picking area, and output the lighting effect control signal to the atmosphere light device to control the atmosphere light device to emit light according to the lighting effect control signal.

[0134] In a further embodiment, the color sampling module 1300 includes: an average color determination submodule, which is used to sample every other pixel of each lighting effect color sampling area, and determine the average color based on the total number of sampled pixels and the colors of the pixels; and a target color determination submodule, which is used to enhance the average color to obtain the target color of the lighting effect color sampling area.

[0135] In a further embodiment, the image processing module 1100 includes: an image segmentation submodule, which is used to perform target detection and instance segmentation on the image frame using a preset image segmentation model, and determine the display area in the image frame and its edge point information to be completed, as well as the anchor frame area of ​​the display area in the frame; an edge point completion submodule, which is used to linearly complete the edge point information to be completed, and then perform noise reduction processing on the corresponding edge contour point information to obtain edge frame point information; a region and its information determination submodule, which is used to determine each color picking point region and its region contour point information in the anchor frame region according to preset region positioning features and the edge frame point information; a color picking point information determination submodule, which is used to determine the coordinate information of the color picking points in each color picking point region according to preset region fixed point features and the region contour point information to constitute the color picking point information.

[0136] In a further embodiment, the color picking point information determination submodule includes any one or more of the following: a first color picking point submodule, for each color picking point area including a color picking point belonging to the midpoint of the long side of the display, according to the position of the color picking point area in the anchor frame area, and determining the coordinate information of the color picking point in the color picking point area from the area contour point information of the color picking point area; a second color picking point submodule, for each color picking point area including a color picking point belonging to a corner point of the display, according to the edge frame point information, determining the vector formed by every two adjacent edge points, and the coordinate information of every two adjacent edge points. The angle between the vectors is determined from the regional edge point information of the color picking point area, and the coordinate information of the intersection point between the two adjacent vectors with the largest angle is determined as the color picking point position information of the color picking point area; the third color picking point sub-module is used to, for each color picking point area including the color picking point belonging to the midpoint of the wide side of the display, determine the coordinate information of the regional contour point whose distance ratio with the two color picking points meets the preset conditions from the regional contour point information of the color picking point area according to the coordinate information of two color picking points adjacent to the color picking point area, as the coordinate information of the color picking point in the color picking point area.

[0137] In a further embodiment, the color sampling module 1300 includes: a first average color determination submodule, which is used to determine, for each lighting effect color picking area, the core hue interval belonging to the lighting effect color picking area, and the colors of multiple pixels whose corresponding hue values ​​meet preset conditions, and obtain the corresponding average color; a second target color determination submodule, which is used to enhance the average color to obtain the target color of the lighting effect color picking area.

[0138] In a further embodiment, the third color picking point submodule includes: a first fixed-point submodule, which is used to determine the coordinate information of the area contour points with the same distance as two color picking points adjacent to the color picking point area when the image frame belongs to a monocular shooting type, as the coordinate information of the color picking points in the color picking point area; a second fixed-point submodule, which is used to determine the coordinate information of the area contour points with a ratio of the distance between two color picking points adjacent to the color picking point area belonging to a first preset value range when the image frame belongs to a binocular left-side shooting type, as the coordinate information of the color picking points in the color picking point area; a third fixed-point submodule, which is used to determine the coordinate information of the area contour points with a ratio of the distance between two color picking points adjacent to the color picking point area belonging to a second preset value range when the image frame belongs to a binocular right-side shooting type, as the coordinate information of the color picking points in the color picking point area.

[0139] In a further embodiment, the image segmentation submodule includes: a model training submodule, which is used to obtain a preset training set and train the image segmentation model to a convergent state so that it can learn to determine the display area and its edge point information belonging to the imaging range of the display, as well as the anchor frame area of ​​the display area in the frame from the input image. The training set includes multiple training samples and their supervisory labels. The training samples are images containing display areas. The supervisory labels mark the target edge detection information corresponding to the display areas in the corresponding training samples. The training samples in the training set are obtained by photographing the display using a camera device with a single fisheye camera, and / or include photographing the display using a left fisheye camera of a camera device with a dual fisheye camera and a right fisheye camera of a camera device with a dual fisheye camera, respectively.

[0140] In a further embodiment, the color determination module 1200 includes: an image correction submodule, which is used to use a fisheye correction algorithm to perform image correction on each subdivided area; and an area cropping submodule, which is used to crop the central part of each corrected subdivided area arranged along the circumference of the display area as a lighting effect color picking area.

[0141] In order to solve the above technical problems, the present application also provides a computer device. Figure 7As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. Among them, the computer-readable storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement an atmosphere lighting effect control method. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute the atmosphere lighting effect control method of the present application. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art can understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In this embodiment, the processor is used to execute Figure 6 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the atmosphere lighting effect control device of this application, and the server can call the program code and data of the server to execute the functions of all submodules.

[0143] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the atmosphere lighting effect control method of any embodiment of the present application.

[0144] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0145] To sum up, the present application realizes an atmosphere lighting effect that is synchronously matched with the display content, and extends the atmosphere lighting effect to the environment outside the display. It can dynamically adapt to the changes in the display content in real time to present a matching atmosphere lighting effect, creating a natural and beautiful viewing atmosphere, and significantly improving the user's visual experience and satisfaction.

[0146] It will be understood by those skilled in the art that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, changed, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, changed, rearranged, decomposed, combined, or deleted. Furthermore, the steps, measures, and schemes in the various operations, methods, and processes in the prior art that are open source and in this application may also be alternated, changed, rearranged, decomposed, combined, or deleted.

[0147] The above description is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for controlling atmosphere lighting effects, characterized in that: The steps include: Acquire an image frame in a video stream generated by a camera device shooting a display, determine a display area in the image frame that belongs to an imaging range of the display, and color picking point information of each color picking point set around the display area; Partitioning the display area according to the color picking point information to obtain a plurality of subdivided areas, and using the plurality of subdivided areas arranged along the circumference of the display area as lighting effect color picking areas; Perform color sampling on each lighting effect color picking area to determine the target color corresponding to each lighting effect color picking area; Constructing a lighting effect control signal according to the target color corresponding to each lighting effect color picking area, and outputting the lighting effect control signal to the ambient light device to control the ambient light device to emit light according to the lighting effect control signal; The step of determining the display area in the image frame that belongs to the imaging range of the display and the color picking point information of each color picking point set around the display area includes: Using a preset image segmentation model to perform target detection and instance segmentation on the image frame, determine the display area in the image frame and its edge point information to be completed, as well as the anchor frame area of ​​the display area in the frame; After linearly completing the edge point information to be completed, performing noise reduction processing on the corresponding edge contour point information to obtain edge framework point information; Determine each color picking point area and its area contour point information in the anchor frame area according to the preset area positioning features and the edge frame point information; According to the preset regional fixed-point features and the contour point information of each region, the coordinate information of the color picking point in each color picking point region is determined to constitute the color picking point information.

2. The atmosphere lighting effect control method according to claim 1, characterized in that: Performing color sampling on each lighting effect color picking area to determine a target color corresponding to each lighting effect color picking area includes the following steps: For each lighting effect color picking area, sampling is performed on every other pixel in the lighting effect color picking area, and the average color is determined according to the total number of sampled pixels and the colors of the pixels; The average color is enhanced to obtain a target color of the lighting effect color picking area.

3. The ambient lighting effect control method according to claim 1, characterized in that: According to the preset regional fixed-point features and the contour point information of each region, the coordinate information of the color picking point in each color picking point region is determined to form the color picking point information, including any one or more of the following steps: For each color picking point region including a color picking point belonging to the midpoint of the long side of the display, according to the position of the color picking point region in the anchor frame region, the coordinate information of the color picking point in the color picking point region is determined from the region contour point information of the color picking point region; For each color picking point area including the color picking points belonging to the corner points of the display, the vector formed by every two adjacent edge points and the angle between every two adjacent vectors are determined according to the edge frame point information, and the coordinate information of the intersection between the two adjacent vectors with the largest angle is determined from the regional edge point position information of the color picking point area as the color picking point position information of the color picking point area; For each color picking point area that includes a color picking point that belongs to the midpoint of the wide side of the display, based on the coordinate information of two color picking points adjacent to the color picking point area, the coordinate information of the area contour point whose distance ratio with the two color picking points meets the preset conditions is determined from the area contour point information of the color picking point area as the coordinate information of the color picking point in the color picking point area.

4. The ambient lighting effect control method according to claim 1, characterized in that: Performing color sampling on each lighting effect color picking area to determine a target color corresponding to each lighting effect color picking area includes the following steps: For each lighting effect color picking area, determine the core hue interval belonging to the lighting effect color picking area, and the colors of multiple pixels whose corresponding hue values ​​meet preset conditions, and calculate the corresponding average color; The average color is enhanced to obtain a target color of the lighting effect color picking area.

5. The ambient lighting effect control method according to claim 3, characterized in that: For each color picking point area including a color picking point belonging to the midpoint of the wide side of the display, according to the coordinate information of two color picking points adjacent to the color picking point area, from the area contour point information of the color picking point area, the coordinate information of the area contour point whose distance ratio to the two color picking points meets the preset conditions is determined as the coordinate information of the color picking points in the color picking point area, including the following steps: When the image frame belongs to a single-eye shooting type, coordinate information of area contour points having the same distance from two color picking points adjacent to the color picking point area is determined as coordinate information of the color picking points in the color picking point area; When the image frame belongs to the left-side shooting type of both eyes, coordinate information of a region contour point whose distance ratio between two color picking points adjacent to the color picking point region belongs to a first preset value range is determined as the coordinate information of the color picking point in the color picking point region; When the image frame belongs to the binocular right side shooting type, the coordinate information of the area contour points whose distance ratio between two color picking points adjacent to the color picking point area belongs to the second preset value range is determined as the coordinate information of the color picking points in the color picking point area.

6. The atmosphere lighting effect control method according to claim 1, characterized in that: Before performing target detection and instance segmentation on the image frame using a preset image segmentation model to determine the display area in the image frame and its edge point information to be completed, as well as the anchor frame area of ​​the display area in the frame, the method includes: A preset training set is obtained, and an image segmentation model is trained to a convergence state so that the model can learn to determine the display area and its edge point information belonging to the imaging range of the display, as well as the anchor frame area of ​​the display area in the frame from the input image. The training set includes multiple training samples and supervisory labels thereof. The training sample is an image including a display area. The supervisory label marks target edge detection information corresponding to the display area in the corresponding training sample. The training samples in the training set are obtained by photographing the display using a camera device with a single fisheye camera, and / or include photographing the display using a left fisheye camera of a camera device with a dual fisheye camera and a right fisheye camera of a camera device with a dual fisheye camera, respectively.

7. The ambient lighting effect control method according to claim 1, wherein a plurality of subdivided areas arranged along the circumference of the display area are used as lighting effect color selection areas, characterized in that: The steps include: Use fisheye correction algorithm to correct the image of each subdivided area; For each corrected subdivided area arranged along the circumference of the display area, a central portion of the subdivided area is cut out as a lighting effect color selection area.

8. An ambient lighting effect control device, characterized in that: include: An image processing module is used to obtain an image frame in a video stream generated by a camera device shooting a display, determine a display area in the image frame that belongs to an imaging range of the display, and color picking point information of each color picking point set around the display area; A color picking determination module, used for partitioning the display area according to the color picking point information to obtain a plurality of subdivided areas, and using the plurality of subdivided areas arranged along the circumference of the display area as lighting effect color picking areas; A color sampling module is used to perform color sampling on each lighting effect color sampling area to determine the target color corresponding to each lighting effect color sampling area; A lighting effect playback module, used to construct a lighting effect control signal according to the target color corresponding to each lighting effect color picking area, and output the lighting effect control signal to the atmosphere light device to control the atmosphere light device to emit light according to the lighting effect control signal; The step of determining the display area in the image frame that belongs to the imaging range of the display and the color picking point information of each color picking point set around the display area includes: Using a preset image segmentation model to perform target detection and instance segmentation on the image frame, determine the display area in the image frame and its edge point information to be completed, as well as the anchor frame area of ​​the display area in the frame; After linearly completing the edge point information to be completed, performing noise reduction processing on the corresponding edge contour point information to obtain edge framework point information; Determine each color picking point area and its area contour point information in the anchor frame area according to the preset area positioning features and the edge frame point information; According to the preset regional fixed-point features and the contour point information of each region, the coordinate information of the color picking point in each color picking point region is determined to constitute the color picking point information.

9. A computer device comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

Images