Ambient lighting device and color matching method therefor

Abstract

The present application relates to an ambient lighting device and a color matching method therefor. The color matching method for an ambient lighting device provided by the present application comprises: detecting each content object in a target image, so as to obtain corresponding boundary information of each content object; on the basis of the boundary information of the content objects, determining a light-emitting unit set in an ambient lighting device corresponding to each content object; on the basis of image content in the target image corresponding to the boundary information of each content object, correspondingly determining the image color value of a dominant hue of each content object; and, on the basis of the image color value of the dominant hue of each content object, determining the light-emitting color value of each light-emitting unit in the light-emitting unit set corresponding to each content object.

Description

Ambient light device and color matching method thereof Technical Field

[0001] The present application relates to the field of lighting control, and in particular to an atmosphere lighting device and a color matching method thereof. Background Art

[0002] The lighting effects projected by the ambient lighting equipment in the prior art cannot accurately and effectively express the distribution of content objects in the environmental reference image through color, resulting in the lighting effects created by the ambient lighting equipment having a low degree of simulation of the light atmosphere of the environmental reference image. The rendered light atmosphere is not realistic and accurate enough, making it difficult to create an immersive effect.

[0003] In addition, for ambient light devices in the form of border lights, mechanical partition mapping of the original image according to traditional technology cannot effectively express the lighting atmosphere of the environmental reference image, and the technology needs to be updated. Summary of the Invention

[0004] The purpose of this application is to provide a color matching method for an atmosphere lighting device.

[0005] The color matching method for an atmosphere light device provided in the present application includes: detecting each content object in a target image to obtain boundary information corresponding to each content object; determining a light-emitting unit set corresponding to each content object in the atmosphere light device based on the boundary information of the content object; determining an image color value of the main hue of each content object based on the image content corresponding to the boundary information of each content object in the target image; and determining a light-emitting color value of each light-emitting unit in the light-emitting unit set corresponding to the content object based on the image color value of the main hue of each content object.

[0006] The atmosphere light device provided in the present application includes a central processing unit and a memory, and the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the aforementioned method.

[0007] Compared to the existing technology, this application has many advantages. First, it improves the degree of simulation of the lighting atmosphere of the target image by the lighting effect, making the lighting atmosphere rendered by the atmosphere lighting device more realistic and accurate. The simulation of the lighting atmosphere of the target image by the atmosphere lighting device is more delicate, realistic, and soft, and the corresponding lighting effect created by the atmosphere lighting device is more refined. In addition, under the rendering of the lighting atmosphere of the atmosphere lighting device, the picture atmosphere of the desktop image can be effectively extended to the physical space, thereby enhancing the immersion of the terminal device user. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] FIG1 is a schematic diagram showing the principle of the electrical structure of an atmosphere light device in an embodiment of the present application;

[0009] Figures 2, 3, and 4 are schematic diagrams of display frames of the ambient light device in the embodiments of the present application, wherein the ambient light of Figure 2 is arranged as a curtain light, Figure 3 is arranged as a border light, and Figure 4 is arranged as a spliced ​​lamp.

[0010] FIG5 is a flow chart of a method for matching the color of an ambient light device in an embodiment of the present application;

[0011] FIG6 is an exemplary target image of the present application;

[0012] FIG7 is a schematic diagram showing the correspondence between various content objects in FIG6 and various sets of light-emitting units in the border light;

[0013] FIG8 is an exemplary graphical user interface showing an interface canvas and related function setting buttons for adding an identification location;

[0014] FIG9 is a flow chart of a method for matching lighting effects of an ambient light device according to an embodiment of the present application;

[0015] FIG10 is a flow chart of a method for mapping and matching colors of an ambient light device in an embodiment of the present application;

[0016] FIG11 is a flow chart of a method for controlling lighting effect playback of an ambient light device in an embodiment of the present application. DETAILED DESCRIPTION

[0017] Please refer to Figure 1. It can be seen from the principle schematic diagram of the electrical structure of an atmosphere light device provided by an embodiment of the present application that the atmosphere light device includes a controller 1, an atmosphere light 2, and an image acquisition interface. The atmosphere light 2 is electrically connected to the controller 1 so as to accept the control of the computer program running in the controller 1 and work together to realize light effect playback.

[0018] The controller 1 generally includes a control chip, a communication component, and a bus connector. In some embodiments, the controller 1 may also be configured with a power adapter, a control panel, a display screen, etc. as needed.

[0019] The power adapter is mainly used to convert AC power into DC power to power the entire atmosphere light device. The control chip can be implemented using various embedded chips, such as Bluetooth SoC (System on Chip), WiFi SoC, MCU (Micro Controller Unit), DSP (Digital Signal Processing), and other types of chips. The control chip usually includes a central processing unit and a memory. The memory and the central processing unit are used to store and execute program instructions respectively to achieve corresponding functions. The above various types of control chips can have their own communication components, or they can be configured with additional communication components as needed. The communication component can be used to communicate with external devices. For example, it can communicate with terminal devices such as personal computers or various smart phones, so that after the user issues various configuration instructions through their terminal device, the control chip of the controller 1 can receive the configuration instructions through the communication component and complete the basic configuration to control the operation of the atmosphere light. In addition, the controller 1 can also obtain the interface image of the terminal device or the real-time preview image captured by the camera through the communication component. The bus connector is mainly used to connect the power supply to the atmosphere lamp 2 connected to the bus and provide lighting effect playback instructions, so the corresponding pins of the power bus and the signal bus are provided. Therefore, when the atmosphere lamp 2 needs to be connected to the controller 1, it can be connected to the bus connector through the corresponding connector of the atmosphere lamp. The control panel usually provides one or more buttons for implementing switch control of the controller 1, selecting various preset lighting effect control methods, etc. The display screen can be used to display various control information so as to cooperate with the buttons in the control panel to support the implementation of human-computer interaction functions. In some embodiments, the control panel and the display screen can be integrated into the same touch screen.

[0020] Referring to FIG. 2 , the ambient light in FIG. 2 is arranged in the form of a curtain light. The ambient light 2 includes multiple light strips 21 connected to a bus. Each light strip 21 includes multiple serially connected lamp beads 210, each lamp bead 210 serving as a light-emitting unit. Typically, each light strip 21 has the same number of lamp beads 210, arranged at equal intervals. When used as a curtain light, the ambient light 2 typically deploys its light strips 21 according to the layout shown in FIG. 2 , thereby arranging all the lamp beads in all light strips 21 in an array, forming a lamp bead matrix structure. Because all the lamp beads can produce a frame effect when emitting light in concert, the surface of the entire lamp bead matrix structure forms a display frame 4. When lighting effects are displayed, a specific pattern effect can be formed within the display frame 4. A static lighting effect results when a single pattern is displayed statically, while a dynamic lighting effect results when the patterns are switched sequentially.

[0021] Each light strip 21 can be composed of multiple lamp beads 210 connected in series, each lamp bead 210 being a light-emitting unit. The operating current of each lamp bead 210 in the same light strip 21 is transmitted by the same set of cables connected to the bus. The lamp beads 210 in the same light strip 21 can be electrically connected in parallel. In one embodiment, the light strips 21 in the same lamp bead matrix structure can be arranged at equal intervals along the bus direction, and the number and position of the lamp beads 210 in each light strip 21 are also arranged correspondingly. In this way, when the entire display frame 4 is viewed from a distance, its luminous effect acts like a screen, forming a visual pattern effect to the human eye.

[0022] Similarly, referring to Figure 3, the ambient lighting in Figure 3 is arranged around the terminal device's display, forming a border light pattern. The border light can be composed of a single or multiple light strips connected to a bus. The light strips and the lamps within these strips share the same structure and communication mechanism as curtain lights. When arranged as a border light pattern, although all the lamps are arranged around the display, the overall pattern can be considered to be a display frame 4 constructed based on a lamp matrix structure. However, the lamps are not located in the center of the display frame 4, but only on the four edges. When the lighting effects are activated, a certain ambient light effect is diffused within and outside the display frame 4.

[0023] The controller 1 of the atmosphere light device is used to realize the operation control of the entire atmosphere light device and is responsible for the communication inside and outside the entire atmosphere light device. Among them, the controller 1 is also responsible for driving the image acquisition interface to collect the environmental reference image frame by frame through the image acquisition interface. The environmental reference image can be the interface image of the terminal device or the real scene image of the physical space. Then, according to each frame of the environmental reference image, the lighting effect playback instruction of the corresponding frame is generated, and the curtain light is controlled to play the lighting effect of the corresponding frame through the lighting effect playback instruction.

[0024] Each lamp bead 210 of the light strip 21 of the ambient light 2 is also equipped with a corresponding control chip. This control chip can be selected according to the above disclosure, or other more economical control chips can be selected. Its main function is to extract the luminous color value corresponding to the lamp bead 210 from the lighting effect playback instruction, and control the light-emitting element in the lamp bead 210 to emit the corresponding color light. The light-emitting element can be an LED lamp.

[0025] Figure 4 further reveals another form of the atmosphere lamp device of the present application, which is essentially a spliced ​​lamp. The atmosphere lamp 2 in the spliced ​​lamp is composed of one or more lamp blocks 22. Inside the lamp block 22 of the spliced ​​lamp, there are multiple light-emitting units (not shown) that are standardized and set at different positions of the lamp block. Each light-emitting unit can also be provided with a corresponding light-emitting control chip to parse the corresponding control data and generate a corresponding light-emitting control signal. The light-emitting control signal is used to control the light-emitting element in the corresponding light-emitting unit to emit light according to a specific light-emitting color value. The lamp block as a whole can also be provided with an independent control chip as a control unit to control the light emission of all the light-emitting units therein. This independent control unit can transmit the corresponding timing control data to the control chip of each light-emitting unit to achieve the purpose of centralized control. Of course, the entire lamp block can also be directly controlled by a single control chip to achieve the purpose of corresponding lighting effect playback. This is mainly designed flexibly according to the capabilities of the control chip used in the lamp block and its light-emitting unit, and does not affect the embodiment of the creative spirit of the present application. According to these principles, for a lamp block, it is possible to control all its light-emitting units to emit light simultaneously, or to control each light-emitting unit at a specific granularity. The finer the control granularity, the more delicate the generated lighting effect.

[0026] Light blocks 1 of different shapes can be connected to each other. For example, a square light block can be connected to any of the outer edges of a hexagonal light block. It is easy to understand that by combining light blocks of different shapes, a richer array pattern can be constructed. When it is necessary to control each light block to play a corresponding lighting effect, the light units of each light block are coordinated and controlled to emit light, thereby presenting a display frame 4 showing the corresponding lighting effect.

[0027] The image acquisition interface can be either a hardware interface or a software interface implemented in controller 1. If it's a hardware interface, the image acquisition interface can be implemented as a camera, with controller 1 loading the corresponding driver to drive the camera. When the camera is aimed at a target screen, such as the terminal device's display desktop, or at a physical space environment, images are captured at a certain frame rate to capture the interface image. If it's a software interface, the image acquisition interface can be an image acquisition program implemented on controller 1 using the graphics infrastructure technology provided by the terminal device's operating system. Controller 1 connects to the terminal device via various cables, such as HDMI or Type-C cables, and with the support of this graphics infrastructure, it can continuously capture the terminal device's interface image. Of course, if a wireless screen projection protocol has been pre-established between controller 1 and the terminal device, controller 1 can also acquire the terminal device's interface image via wireless communication. The graphics infrastructure technology of the operating system varies depending on the operating system type. For example, the Windows operating system provides a corresponding technology, namely, Microsoft Direct X Graphics Infrastructure (DXGI), which can implement this function.

[0028] It can be seen that when the image acquisition interface is responsible for collecting environmental reference images, the specific environment in which the images are collected can be flexibly set by the user. For example, when the image acquisition interface is a camera, the user can point the camera at the computer's graphical user interface to shoot to collect the corresponding interface image as the target image for playing lighting effects, so that the atmosphere lamp 2 can generate corresponding lighting effects according to the interface image; the user can also point the camera at a physical space environment, such as an outdoor environment, and record the real scene image as the environmental reference image, so that the atmosphere lamp 2 can play the corresponding lighting effects according to the real scene.

[0029] The atmosphere light device of the present application, when it is powered on, the control chip of the controller can call and execute the computer program from the memory, and through the default initialization process of the computer program, power on and initialize the atmosphere light, and complete the drive configuration of the atmosphere light and other hardware devices.

[0030] In one embodiment, when the controller starts the atmosphere light, it can first send a self-test instruction to the atmosphere light to drive each light strip or each lamp bead in the light block of the atmosphere light to return its position information in the light strip or light block. Each lamp bead is provided with a corresponding control chip for data communication with the control chip in the controller, so that the characteristic information of the lamp bead itself can be serially connected with the characteristic information of other lamp beads in accordance with the serial communication protocol to realize the representation of its own position information. The serial communication protocol executed between the controller and the lamp bead can be any one of IIC (Inter-Integrated Circuit, integrated circuit bus), SPI (serial peripheral interface, serial peripheral interface), and UART (Universal Asynchronous Receiver-Transmitter, universal asynchronous receiver-transmitter). After the controller obtains the result data returned by the self-test of each lamp bead from the bus, it parses it and determines the position of each lamp bead in the display frame 4 presented by the entire atmosphere light according to the order of the characteristic information of each lamp bead in the result data. In this way, each lamp bead can be regarded as a light-emitting unit, which can be understood as a basic pixel. When constructing the lighting effect playback instruction, the controller can set the corresponding light-emitting color value for each basic pixel according to the position information of each lamp bead and actual needs.

[0031] After completing initialization, the controller can continuously acquire the environmental reference image as the target image through the image acquisition interface, and perform color extraction on the target image to determine the luminous color value of each luminous unit in its display frame. To this end, the method of the present application can be used to identify each content object in the target image, and then determine the main color tone based on the image content of each content object. The corresponding image color value is determined based on the main color tone of each content object, and then the luminous color value of each luminous unit in the mapping area of ​​the content object in the display frame is generated based on the image color value.

[0032] In some embodiments, the controller 2 of the present application can be implemented in an independent computer device, as long as the computer device is equipped with the corresponding hardware of the controller 2, and the corresponding business logic of the controller 2, including the business logic executed by the method of the present application, is implemented as a computer program installed in the computer device for operation. When the controller 2 is implemented in a computer device, various resources inherent to the computer device, such as a camera, can be shared to read the real-time preview image captured by the camera as an environmental reference image, or the business logic can be simplified, for example, the interface image can be read as an environmental reference image through the computer's graphics open library (OpenGL), etc., which can save the overall implementation cost. The computer device referred to here can be any terminal device for use by the user, such as a smart phone, a personal computer, a laptop computer, a tablet computer, etc.

[0033] Referring to FIG. 5 , in one embodiment, the color matching method of the ambient light device of the present application is mainly implemented on the controller side of the ambient light device and is executed by the control chip of the controller, including:

[0034] Step S5100: Detect each content object in the target image and obtain boundary information corresponding to each content object;

[0035] The target image is used as a reference for the color distribution of the lighting effects played by the ambient light device, so it is essentially an environmental reference image. The source of the environmental reference image can be obtained from a real camera shot, called from the cache or video memory, read directly from an image file, or received via wired or wireless transmission. For example, the environmental reference image can be a real-life image captured by a camera or an interface image of a terminal device. The interface image can be captured by a camera, or read from the terminal device or captured and transmitted.

[0036] Target images typically contain a variety of content objects, such as objects and patterns. Each content object may have a unique shape. Furthermore, even the same content object may appear differently in different target images due to factors such as the angle at which the content object is displayed and the surrounding lighting. Therefore, to determine the boundaries of each content object in each target image, boundary detection must be performed to obtain boundary information for each content object.

[0037] In one embodiment, considering that the atmosphere lighting device plays corresponding lighting effects based on multiple consecutive target images, each target image may come from the same image stream, and two adjacent target images may be two image frames of the same video content, and the image content between these two image frames changes little. In this case, the frame difference information between the current target image and the previous target image can be calculated based on the current target image. When the change amplitude presented in the frame difference information is less than a preset threshold, the boundary information corresponding to the previous target image can be used. Otherwise, the boundary information of each content object in the current target image is re-determined.

[0038] There are many ways to determine the boundary information of each content object in the target image, for example:

[0039] In some embodiments, based on traditional boundary detection algorithms, such as Sobel, Prewitt, Roberts, Canny, Marr-Hildreth, etc., the target image can be binarized to obtain its corresponding grayscale image, and then the selected algorithm can be applied to determine the boundary information corresponding to the outline of each content object.

[0040] In other embodiments, image semantic segmentation can be performed based on a pre-trained deep learning model, including a traditional non-prompt image segmentation model or a prompt image segmentation model with image interaction processing capabilities, according to the deep semantics of the target image to obtain corresponding boundary information.

[0041] In some further enhanced embodiments, before determining the boundary information of each content object, target detection can be performed on the target image with the help of a target detection model. After detecting one or more targets as the regional images of the content objects, the regional images of each content object are successively input into the deep learning model to implement image semantic segmentation, and the boundary information of each content object is successively obtained.

[0042] The boundary information obtained from the target image can be pre-defined by an algorithm or by leveraging the capabilities provided by the technical architecture of a deep learning model to be expressed in various data formats that are convenient for parsing and calling. For example, it can be expressed as an image in the form of a hard mask or a soft mask to obtain a corresponding regional mask or full image mask. The soft mask represents the probability that a pixel in the target image belongs to the foreground using any value in the numerical interval [0, 1]. The hard mask can indicate whether a pixel in the target image belongs to the background or foreground using 0 or 1. It is not difficult to understand that the boundary information of the content object in the target image not only defines the outline of the content object, but also defines the content area of ​​the content object in the corresponding image. By extracting the set of pixels within the closed area defined by the outline of the boundary information, the image content of the content object can be obtained.

[0043] In some embodiments, for each content object, its corresponding region mask can be determined based on its region image. A region mask simply represents the boundary information of a single content object's region image. Therefore, to facilitate alignment with the target image, each content object's region mask can be further converted into a full-image mask based on the target image's dimensions.

[0044] The full-image mask can be independently set corresponding to the boundary information of each content object relative to the target image, or it can be set to adapt to the simultaneous display of the boundary information of multiple content objects, so that the boundary information of multiple content objects relative to the target image can be simultaneously displayed in such a full-image mask, which can be flexibly implemented as needed.

[0045] When using a full-image mask containing multiple content objects to determine the boundary information of each content object, a connected domain calculation can be performed using the full-image mask to identify foreground or background pixels within the same connected domain. The image content belonging to the same connected domain is then converted to the image content of the content object corresponding to that connected domain. As can be seen, in this step, since the boundary information of each content object in the target image has been detected, the content region corresponding to each content object can be quickly extracted based on this boundary information. All pixels within the content region are extracted to form the image content, and a mapping relationship is established between the content object and its image content for rapid access.

[0046] Considering that content objects with a relatively small area in the target image have little impact on the visual effects of the lighting effects, and sometimes may even cause an abrupt effect in the lighting effects due to the excessive difference between their color tones and the color tones of other surrounding content objects, in order to solve this problem, in some embodiments, the proportion of the total number of pixels of the content object relative to the total number of pixels in the entire target image can be calculated based on the boundary information of the content object, and the boundary information of the content object whose proportion is lower than a preset threshold is merged with the boundary information of any content object in its surroundings, so that the boundary information of the two content objects is merged into the same boundary information, and the two content objects are treated as the same content object.

[0047] Step S5200: Determine the light unit set corresponding to each content object in the atmosphere light device according to the boundary information of the content object;

[0048] As one of the functions of the atmosphere light device of the present application, it is to project the main color of each content object in the target image into the corresponding area of ​​the display frame presented by the atmosphere light device, so as to achieve the effect of simulating the color distribution of the target image in the atmosphere light device. Therefore, there is a corresponding relationship between the size specifications of the display frame and the target image, and the atmosphere light device usually expresses the position information of each light-emitting unit relative to the reference coordinate system through layout configuration information, which actually defines the position information of each light-emitting unit on the display frame. Based on this, it is not difficult to understand that with the help of this correspondence between the display frame and the target image, the mapping area of ​​the content object in the display frame of the atmosphere light device can be determined according to the boundary information of each content object. For example, the mapping areas 401, 402, 403 shown in Figure 2, or the mapping area 40 shown in Figure 3, etc., and then determine the various light-emitting units covered by this mapping area, and define these light-emitting units as a light-emitting unit set. This light-emitting unit set is the light-emitting unit set mapped to the target content object. In this way, each content object in the target image can obtain a corresponding luminous unit set based on its boundary information, so that a one-to-one mapping relationship is established between the content object and the luminous unit set. In fact, a one-to-one mapping relationship is established between the content area corresponding to the image content of each content object and each luminous unit set.

[0049] Although FIG4 does not provide a diagram of the mapping area, it is not difficult to understand that given a boundary information, a mapping area can also be obtained in the display frame of the spliced ​​lamp in FIG4 , and then it can be determined that the various light-emitting units within the range covered by the mapping area constitute a light-emitting unit set. For the same mapping area, it is allowed to cover light-emitting units on different light blocks across multiple light blocks.

[0050] In some embodiments, considering that the content object located in the middle of the target image cannot strictly correspond to the light-emitting units in the border light state, in this case, the correspondence between the boundary information of the content object and the light-emitting units can be adjusted, and the boundary information can be projected to the adjacent side of the border light. According to the projection correspondence, the light-emitting unit set corresponding to the boundary information of each content object is determined. Figure 6 is an exemplary target image of the present application, and Figure 7 is a schematic diagram of the segmentation result obtained by performing image semantic segmentation on the target image in Figure 6, that is, the mapping relationship between each content object and the light-emitting unit set in the atmosphere light. In Figure 7, ABCDE respectively represent the image content of each content object, which also shows the mapping relationship between each image content and each light-emitting unit set in the atmosphere light in the border light state. It can be seen that various forms of atmosphere lights can establish a corresponding mapping relationship between each content object and the corresponding light-emitting unit set.

[0051] In a more specific embodiment, in order to facilitate the calling of each light-emitting unit set, each light-emitting unit in each light-emitting unit set can be stored in the same array, and then a mapping relationship is established between the array and the corresponding content object.

[0052] Step S5300: Determine the image color value of the main hue of each content object according to the image content corresponding to the boundary information of each content object in the target image;

[0053] As the basis for achieving lighting effects, it is necessary to determine the primary color tone of each content object based on its image content and use a corresponding image color value to represent it. In this regard, if the corresponding image content has been extracted based on the boundary information of the content object in step S5100 and the corresponding mapping relationship has been established, the image content of each content object can be directly called according to the corresponding mapping relationship to determine the image color value of the primary color tone of each content object. Otherwise, the image content of each content object can also be determined in this step first, for example:

[0054] In some embodiments, based on the boundary information of each content object, each pixel within the contour range defined by the boundary information can be extracted from the target image to obtain the corresponding image content, and then the image color value of the corresponding main color tone can be determined based on the color value of each pixel of this image content.

[0055] In other embodiments, when boundary information is obtained using a deep learning model, pre-implemented functionality of the deep learning model can be utilized to directly obtain an image set corresponding to the target image segmented by the deep learning model using the mask. This image set contains the image content corresponding to each content object. In this case, the image content corresponding to each content object in the image set can be directly retrieved, and the image color value of the corresponding dominant hue can be determined using the color values ​​of each pixel therein.

[0056] Of course, in some embodiments, with the help of the output function pre-implemented by the deep learning model, while the deep learning model implements image semantic segmentation of the target image or regional image to obtain the regional mask of the content object, the corresponding image content of the content object can also be obtained.

[0057] The target image is generally represented as bitmap data to facilitate reading the color value of each pixel therein. The representation of its color value can have different representation methods depending on the format of the target image. For example, different formats such as RGB and YUV have different ways of representing color values. Those skilled in the art can adapt to these different formats and perform corresponding representation processing, which will not be elaborated here.

[0058] In some embodiments, when determining the image color value of the primary hue of each content object based on its image content, pixels in the image content with color values ​​below a preset threshold can be first filtered out, and then the image color value of the primary hue can be determined based on the remaining valid pixels. The preset threshold can be used to measure the brightness of the pixel. For example, for RGB format, the preset threshold corresponding to the three primary colors is RGB (10, 10, 10), which is closer to black. When a pixel in the image content is below this threshold, it can be filtered out. In this way, the last remaining valid pixels are used to determine the image color value of the corresponding primary hue, reducing the impact of black on the primary hue, so that the determined primary hue can maintain a high brightness, ensuring that the lighting efficiency is presented at a high brightness.

[0059] In some embodiments, when determining the image color value of the dominant hue of each content object based on its image content, a simple arithmetic average can be performed on all pixels in the image content, or all valid pixels as previously described, to obtain a corresponding mean value, which can be used as the image color value corresponding to the dominant hue. This approach can save computational effort and offer computational cost advantages for embedded control chips.

[0060] In other embodiments, when determining the image color value of the dominant hue of each content object based on its image content, a central region of a predetermined area can be first determined in the image content. The image color value corresponding to the central hue can be determined using all pixels or all valid pixels in the central region, and the image color value corresponding to the peripheral hue can be determined using all pixels or all valid pixels outside the central region. The specific determination method can be to calculate the arithmetic average of each. Then, the central hue and the peripheral hue are matched according to a preset ratio. For example, the weight ratio of the central hue to the peripheral hue is set to 6:4. The arithmetic average between the image color value of the central hue and the image color value of the peripheral hue is calculated according to this ratio and used as the image color value of the dominant hue of the content object. Determining the image color value of the dominant hue in this manner can amplify the effect of the hue of the central region of the content object, conform to the visual habits of the human eye, and better maintain the correspondence between the lighting effect and the target image in terms of color distribution perception.

[0061] Step S5400: Determine the luminous color value of each luminous unit in the luminous unit set corresponding to each content object according to the image color value of the main hue of the content object.

[0062] A mapping relationship has been pre-established between the content object's light-emitting unit set and the image content. Once the image color value of the primary hue of each content object is determined, it can be used to set the luminous color value of each light-emitting unit in the corresponding light-emitting unit set for each content object. Simply set the image color value of the primary hue of each content object as the luminous color value of each light-emitting unit in the corresponding light-emitting unit set for that content object. In this way, the luminous color value of each light-emitting unit in the corresponding light-emitting unit set can be set for each content object, thereby setting the luminous color value of all light-emitting units in the ambient lighting device, forming the light-emitting unit control data for a frame of lighting effects corresponding to the target image.

[0063] To adapt to lighting effects, the control data corresponding to each light-emitting unit within the entire display frame of the ambient light device, whose target image is the ambient light, is packaged into a corresponding lighting effect playback command and transmitted to the ambient light. The control chips within the ambient light then parse the lighting effect playback command according to pre-set business logic, extracting the control data. Based on the correspondence between the control data and the light-emitting unit, the corresponding light-emitting unit is controlled to emit light of the corresponding color value according to the corresponding control data. The coordinated light emitted by all the light-emitting units in the ambient light presents a lighting effect corresponding to the color distribution of the target image.

[0064] It can be seen from the above embodiments that compared with the prior art, the present application has many advantages, including but not limited to:

[0065] First, the present application detects the boundary information of each content object from the target image, and determines the corresponding association of the light-emitting unit set of each content object in the atmosphere lighting device and the image color value of the main color tone of the image content in the target image based on the boundary information of each content object, so that each light-emitting unit in the display frame presented by the atmosphere lighting device can be partitioned to display the lighting effect according to each content object of the target image, so that the lighting effect of each light-emitting unit in each partition can maintain a corresponding relationship with the main color tone of the content object corresponding to the partition, so that the entire display frame of the atmosphere lighting device can more accurately correspond to the main color of each content object and present the corresponding color layout, and also enable the atmosphere lighting device to relatively accurately reproduce the distribution of content objects in the target image through the light color emitted by the light-emitting unit, thereby improving the simulation degree of the lighting effect to the light atmosphere of the target image, and making the light atmosphere rendered by the atmosphere lighting device more realistic and accurate.

[0066] Secondly, the present application uses the boundary information of the content object as the basis for partition mapping between the light-emitting unit set and the content object, rather than using a regular rectangular area as the basis for partition mapping, so that the correspondence between the content object and the light-emitting unit it covers is more accurate, and the main color tone of the image content of the content object near its boundary will not be disturbed by the main color tone of other content objects. The boundaries of each content object are relatively clearer, the color layout projection relationship is more accurate, and the simulation of the light atmosphere of the target image by the atmosphere lighting device is more delicate, realistic and soft, and the corresponding lighting effect created by the atmosphere lighting device is also more refined.

[0067] In addition, the ambient light device implemented in accordance with the present application creates a more realistic, accurate, and refined light atmosphere by contrasting with the target image. Therefore, when the ambient light device uses the desktop image of the terminal device as the target image and plays corresponding lighting effects against the target image, the ambient light device can effectively extend the desktop image's image atmosphere into the physical space under the rendering of the light atmosphere, thereby enhancing the immersive feeling of the user of the terminal device. Based on any embodiment of the method of the present application, detecting each content object in the target image and obtaining the boundary information corresponding to each content object includes:

[0068] Step S5111: call a target image, perform target detection on the target image using a target detection model, and obtain window position information corresponding to at least one content object;

[0069] When playing lighting effects, the atmosphere light device usually plays multiple frames of target images. These target images can be individual image frames in an image frame sequence. Therefore, according to certain rules, each image frame is called from the image frame sequence as the target image in succession, and the atmosphere light can be controlled to play the corresponding lighting effect according to each target image. Each image frame in the image frame sequence can be obtained by decoding the video stream. For example, after the controller receives the video recorded by the camera, it can store each image frame of the video in the image frame sequence and call them in an orderly manner as the target image. Similarly, after the controller receives the interface image of the terminal device captured at a certain frame rate in the form of a video stream, it can also decode the video stream to obtain the corresponding image frames, store them in the image frame sequence, and call them in an orderly manner as the target image.

[0070] To preliminarily identify the content objects in the target image, you can use a pre-trained target detection model, such as the Yolo series model, EfficientDet model, Faster R-CNN model, and other feasible models, to perform target detection on the target image and determine the window position information corresponding to each content object.

[0071] Step S5112: extracting a region image of a corresponding region from the target image according to the window position information of each content object;

[0072] The window position information output by the object detection model is typically represented by the coordinates of the window's two corners. This information can be used to locate the corresponding content object's region image within the target image. Based on the window position information determined by the object detection model for the target image, the corresponding region image for each window is extracted from the target image. Each region image contains the image content of a corresponding content object.

[0073] Step S5113: using the default image segmentation model, determine a region mask of the content object of each region image relative to the region image, wherein the region mask includes boundary information of the corresponding content object in the region image;

[0074] In this embodiment, the deep learning model used to detect boundary information can be a default image segmentation model that has been pre-trained to reach a convergence state, and is a non-prompt image segmentation model. For example, various models in the U-Net series and various models in the Deeplab series can be used. Such models can extract deep semantic information based on the image embedding data input therein, and further decode and determine the masks corresponding to each content object based on the deep semantic information. In this mask, the boundary information of each content object is represented in the form of a binary mask, and it also serves to distinguish between foreground and background. Therefore, based on the image semantics of the target image, semantic segmentation of each content object in the target image is achieved.

[0075] Since the regional images of each content object have been obtained in advance, each regional image can be encoded into corresponding image embedding data and input into the deep learning model respectively. The deep learning model can infer and detect the corresponding regional mask based on the image embedding data, thereby realizing the effective representation of the boundary information of the image content of the content object in the regional image.

[0076] Taking U-Net as an example, image embedding data is encoded as data corresponding to the three color channels (red, green, and blue). After the image embedding data is input into the U-Net model, it is downsampled by encoders at multiple scales in the encoding path to obtain semantic feature maps corresponding to each scale, effectively representing the deep semantics of the target image at that scale. The semantic feature map corresponding to the smallest scale is then input into the decoding path as the restored feature map. In the decoding path, for each scale, the restored feature map is input and referenced to the semantic feature map represented by the encoder corresponding to that scale to restore the corresponding decoded feature map. This decoded feature map is then used as the input to the decoder at the next higher scale, achieving layer-by-layer decoding. After obtaining the decoded feature map at the last scale, all decoded feature maps are fused to obtain the corresponding mask. When a mask is derived from a single region of the image, it is called a region mask. When a mask is derived from the entire target image, it is called a global mask, regardless of the number of content objects contained in the global mask. In the mask, a binary representation of 0 or 1 is used to indicate whether each pixel in the target image belongs to the foreground or background. The values ​​0 and 1 are used to determine two connected domains, where the set of edge pixels of the connected domain with a pixel value of 1 constitutes the designation of the boundary information of the image content of the content object belonging to the foreground.

[0077] Of course, in this application, boundary information also defines the entire actual image area occupied by the image content, and thus also defines the effective coverage area of ​​the content object's image content within the regional image. Connected domains belonging to the background within the regional image are not processed here. However, connected domains belonging to the background in multiple regional images can also be merged into the same content object as needed.

[0078] Step S5114: convert the region mask of each content object into a corresponding full-image mask, wherein the full-image mask represents the boundary information of the corresponding content object according to the size specification of the target image.

[0079] After determining the region masks for each of the multiple region images, a full-image mask can be further determined. The method for determining the full-image mask is relatively flexible. A dedicated full-image mask can be provided for each content object's image boundary information, or the same full-image mask can be used to represent the image boundary information for each content object. The following describes this in detail using different embodiments:

[0080] In one embodiment, a corresponding full-image mask is determined for the region mask of each content object. Specifically, a binary image with all pixel values ​​assigned to 0 can be first established according to the original image frame of the target image. Then, the region mask of the content object is used to cover the corresponding area in the binary image according to the corresponding area and position of the region image of the content object in the target image. If the region image is scaled in advance, resulting in the scale base of the region mask being inconsistent with the original image of the target image, the region mask can also be scaled according to the size specifications of the original image of the target image, and then the region mask can be replaced with the corresponding area of ​​the binary image. This can achieve the conversion of a single region mask into a single full-image mask. It is not difficult to understand that

[0081] In another embodiment, based on the previous embodiment, in the same manner, on the basis of the same binary image, the various area masks corresponding to the various content objects can be overlaid onto the binary image to obtain a single full-image mask. The full-image mask is compared with the size specifications of the original image of the target image and simultaneously represents the boundary information of each content object.

[0082] It is easy to understand that no matter whether the full image mask represents each content object or all content objects, the image content of each content object can be determined through the boundary information represented therein.

[0083] It should be noted that in the full-image mask, other areas of the image content that are not regarded as content objects generally belong to the background. This background can be obtained on the basis of excluding the pixels corresponding to the image content of each content object. The pixels belonging to the background in the full-image mask are all determined to be the same content object, and their image content is determined accordingly. The corresponding luminous unit set is determined according to its boundary information. Similarly, the luminous color value of each luminous unit in the luminous unit set can also be determined according to the luminous color value of the main color tone of the image content, ensuring that the entire target image establishes a reliable color mapping relationship with all the luminous units in the entire display frame of the atmosphere light.

[0084] In the above embodiment, the window position information of each content object in the target image is first predicted with the help of the target detection model, and the regional image of each content object is extracted based on this. On the basis of each regional image, image semantic segmentation is performed with the help of the image default segmentation model to obtain the regional mask of each regional image, and then the full image mask is converted according to the regional mask, so as to accurately extract the image content of each content object in the target image according to the boundary information represented in the full image mask, and determine the light-emitting unit set in the display frame of the atmosphere light, so as to achieve precise correspondence between the image content and the light-emitting unit set, ensure that the main color tone of the image content in the content object can be accurately projected to the corresponding light-emitting unit set for playback, so that the entire atmosphere light can correspond to the color distribution between each content object and render the corresponding light atmosphere effect. The target detection model and the image default segmentation model are both models that can be deployed in a lightweight manner, and their respective trained segmentation capabilities are also relatively focused. Therefore, they can be reliably deployed and maintain stable operation in ambient light devices based on embedded chips, while ensuring the accurate projection of the color distribution in the lighting effect, further improving the ability of the ambient light device to simulate the color distribution in the target image, making the lighting effect rendered by the ambient light device more realistic, more natural and harmonious. Compared with the existing technology, this undoubtedly improves the ability of the ambient light device to render the light atmosphere in an all-round way. Based on any embodiment of the method of the present application, each content object in the target image is detected, and the boundary information corresponding to each content object is obtained, including:

[0085] Step S5124: calling the target image and segmentation prompt information, wherein the segmentation prompt information is used to indicate the recognition rule of the content object in the target image;

[0086] Regarding the call of target images, as disclosed in the previous embodiments, individual target images can be called from an image frame sequence to play lighting effects, and this will not be further elaborated. Regarding the segmentation prompt information, its function is to guide the deep learning model to identify individual content objects from the target image using certain recognition rules. Therefore, the segmentation prompt information can be determined in advance. In one embodiment, the segmentation prompt information can express the recognition rules using natural language and can be stored in advance as a template for direct call in this step.

[0087] The recognition rules used in the segmentation hint information can be set flexibly. For example, the recognition rules can be expressed as instructing the corresponding hint-type segmentation model to identify the corresponding content object in the target image based on any one or more of the position information or area information specified relative to the target image, or the content type or size information specified based on the content object.

[0088] Step S5125: Using the image hint segmentation model to detect the full-image mask corresponding to each content object in the target image based on the recognition rule of the segmentation hint information, the full-image mask represents the boundary information of the corresponding content object according to the size specification of the target image.

[0089] In view of the rapid development of the image processing capabilities of the hint-based image segmentation model, this embodiment proposes a solution for determining the boundary information of each content object in the target image by means of the image hint-based segmentation model.

[0090] An exemplary image prompt segmentation model can be a Segment Anything Model (SAM). The SAM model as a whole consists of three major modules: an image encoder, a prompt encoder, and a mask decoder. The image encoder is used to map the image to be segmented into an image feature space, achieving a deep semantic representation of the input image, such as the target image of this application. The prompt encoder is responsible for mapping the input segmentation prompt information into the prompt feature space, achieving a deep semantic representation of the segmentation prompt information. The mask decoder has two functional meanings. First, it integrates the two deep semantic information output by the image encoder and the prompt encoder to obtain comprehensive semantic information, achieving a comprehensive representation of the target image and the segmentation prompt information. The final mask is then decoded from this comprehensive semantic information. Since this mask is determined corresponding to the entire image, it is a full-image mask. The boundary information of each content object is already represented in it. The sum of the boundary information of all content objects just fills the entire frame of the target image. Therefore, the full-image mask represents the boundary information corresponding to all content objects in the target image according to the size specifications of the target image.

[0091] In the above embodiments, segmentation prompt information is used, and with the help of the interactive capabilities of the image prompt segmentation model, the content objects in the target image are accurately identified according to the recognition rules specified in the segmentation prompt information. In addition to achieving the same technical advantages of the non-prompt image segmentation model in lighting effects, it can also open up certain interactive capabilities and enrich the recognition methods of content objects, so that the atmosphere lighting device can present the corresponding lighting atmosphere effect according to the color distribution of the content objects distinguished in a specified manner, which is more intelligent. Based on any embodiment of the method of the present application, before calling the target image and segmentation prompt information, it includes:

[0092] Step S5121: Determine an interface canvas according to the size of the target image, and display the interface canvas on a graphical user interface;

[0093] The controller can be equipped with a display and buttons to achieve built-in human-computer interaction capabilities. When communicating with a terminal device, it can also reuse the terminal device's human-computer interaction capabilities. Or, when the controller is implemented on a terminal device, the terminal device itself has human-computer interaction capabilities. With the support of this capability, in scenarios where semantic segmentation of the target image is performed using segmentation prompt information, the controller's human-computer interaction capabilities can be used to provide users with the ability to customize the recognition rules in the segmentation prompt information.

[0094] Based on this, as shown in Figure 8 , the target image can be sized uniformly, and an interface canvas can be set according to the corresponding dimensions. This interface canvas is then displayed in the graphical user interface (GUI), as shown in the upper box of Figure 8 . This interface canvas is consistent with the aspect ratio of the target image; however, when displayed in the GUI, it is scaled to suit the GUI's display size to facilitate user operation. Naturally, the location or area set on the interface canvas can also be associated with a corresponding scaling ratio, uniquely mapping it to the corresponding location or area in the target image.

[0095] Step S5122: Receive at least one recognition position specified based on the interface canvas, and determine position information of the recognition position relative to the target image;

[0096] As shown in Figure 8, the user can control the human-computer interaction capabilities opened by the controller and set one or more recognition locations in the interface canvas. The program process is responsible for mapping the location information set by the user in the interface canvas to the location information in the target image, thereby enabling the user to specify the location information of the content object to be identified by referring to the full frame of the target image.

[0097] As shown in Figure 8, the user can touch the "Recommend" button to submit the automatic recommendation instruction for the recognition position, and the program process will randomly recommend multiple recognition positions. Alternatively, the user can also touch the corresponding button to the right of each position text, such as "upper left corner" and "upper right corner", to start specifying the corresponding recognition position for each position text. When the recognition position corresponding to a position text is specified by the user, the status text in the button to the right of it will change from "unset" to "set", and a positioning mark such as a "+" sign will be used to prompt it in the interface canvas. Users can conveniently set each recognition position through the interface canvas, which is more efficient and quick.

[0098] It is not difficult to understand that the identification position can be a point, or any geometric object such as a closed curve or a smeared area. As long as it is represented according to the corresponding coordinate information, the position information corresponding to the identification position can be given.

[0099] Step S5123: construct the position information of the recognition position and the recognition instruction text expressed in natural language into a recognition rule, and store the recognition rule as segmentation prompt information. The recognition instruction text is used to instruct the image prompt segmentation model to determine the full-image mask corresponding to each content object in the target image.

[0100] This application provides corresponding identification instruction text in advance for identifying the content object in the target image based on the position information. After the user provides the specified position to determine the corresponding position information, the position information is spliced ​​with the identification instruction text to form an identification rule that can be understood by the prompt-type image segmentation model. This identification rule is stored as segmentation prompt information, which can be used to identify the boundary information of the content object for the target image.

[0101] Recognition instruction text can adapt to the capabilities of prompt-based image segmentation models in a natural language format and include representations of multiple tasks. For example, it can include a boundary determination task, which instructs the corresponding model to determine the boundary information of each content object in the target image; another example can include an image extraction task, which instructs the corresponding model to extract the image content corresponding to each content object from the target image based on the boundary information of each content object. In addition, recognition instruction text can also include other formatting tasks to further improve data processing efficiency.

[0102] Taking the representation of the boundary determination task as an example, the recognition instruction text is expressed in Chinese as "Based on the given target image, identify the boundary information of the content object in the following positions:", and the position information of each position can be connected to it to obtain the recognition rule.

[0103] For example, the image extraction task's representation might read, in Chinese, "Based on the boundary information of each object, extract the image content of each object from the target image." Generally speaking, the image extraction task's representation can be added after the boundary determination task's representation, but even if this order is not followed, the model still understands the task.

[0104] As can be seen from the above embodiments, the present application further opens up the service capability of image semantic segmentation of content objects to users through the interface canvas, allowing users to design their own recognition rules for identifying content objects. It is not difficult to understand that by specifying different recognition positions in the interface canvas, the deep learning model may adapt to the different positions and recognize different content objects, and the atmosphere light device may obtain different color distributions accordingly, thereby achieving flexible adjustment of the color distribution, enriching the human-computer interaction function, and comprehensively improving the user experience of the atmosphere light device. On the basis of any embodiment of the method of the present application, according to the boundary information of each content object, the corresponding light-emitting unit set of each content object in the atmosphere light device is determined, including:

[0105] Step S5210: Acquire layout configuration information of the atmosphere light device, where the layout configuration information is based on a preset reference coordinate system and describes position information of each light-emitting unit in the reference coordinate system;

[0106] In the atmosphere light device, the position information of each light-emitting unit in the atmosphere light is determined in advance through the layout configuration information. These position information are usually determined based on the same reference coordinate system. This reference coordinate system can be directly mapped to the coordinate system in the display frame formed by the atmosphere light. For ease of understanding, it can be understood as the reference coordinate system corresponding to the display frame. Accordingly, the layout configuration information of the atmosphere light device actually defines the position information of each light-emitting unit in the atmosphere light in the display frame formed by the atmosphere light based on this reference coordinate system.

[0107] Since the controller and each light-emitting unit communicate based on a serial communication protocol, and the product form of the light strip or light block in which each light-emitting unit is located is also known to the controller, the layout configuration information can be determined according to the business logic pre-set by the controller by sending self-test instructions to each light-emitting unit, obtaining the connection position information returned by each light-emitting unit, and combining it with the form of the ambient light. Of course, this layout configuration information can also be standardized and stored in the controller.

[0108] Step S5220: Determine a mapping area corresponding to the boundary information of each content object in the reference coordinate system based on a mapping relationship between the boundary information of each content object and the reference coordinate system;

[0109] As previously disclosed, a mapping relationship exists between the display frame of the ambient lighting device and the original image frame of the target image. This ensures a one-to-one correspondence between the area covered by the image content of each content object in the target image and the area covered by the display frame of the ambient lighting device. Based on this mapping relationship, the boundary information of each content object can be converted to boundary information in the display frame of the ambient lighting device, corresponding to the reference coordinate system, to obtain the corresponding mapping area in the display frame. Thus, each content object has its own corresponding mapping area in the display frame.

[0110] Step S5230: According to the mapping area corresponding to the content object, determine each light-emitting unit in the atmosphere light device whose position information belongs to the mapping area, and construct a light-emitting unit set corresponding to the content object.

[0111] After determining the mapping area corresponding to each content object, the individual light-emitting units covered by each mapping area can be determined. These light-emitting units are then grouped into a single light-emitting unit set, and a mapping relationship is established with the content object. This content object then determines its corresponding light-emitting unit set. Thus, each content object can determine its corresponding light-emitting unit set, and the corresponding mapping relationship data can be stored for future reference.

[0112] The above embodiments use the position information of each light-emitting unit in the layout configuration information and the corresponding reference coordinate system to determine the distribution of each light-emitting unit in the atmosphere light device in the display frame, and then map the boundary information of each content object to the display frame to determine the mapping area of ​​each content object, so that the image area covered by the image content of the content object is projected from the target image to the display frame, thereby accurately determining the corresponding light-emitting unit set of the content object in the display frame, so that the boundary of the light-emitting unit set in the display frame can correspond to the boundary of the image content of the content object in the target image, and realize the intelligent and precise partitioning of all light-emitting units in the display frame, facilitating the color projection according to the content object, ensuring that the corresponding color distribution can be presented in the display frame according to the position layout of the content object, and enriching the lighting effect display form of the atmosphere light device. On the basis of any embodiment of the method of the present application, according to the image color value of the main color tone of each content object, the light-emitting color value of each light-emitting unit in the light-emitting unit set corresponding to the content object is determined, including:

[0113] Step S5410: Acquire a lighting effect description template corresponding to the ambient light device, wherein the lighting effect description template includes color value attribute items corresponding to each light-emitting unit in the ambient light device;

[0114] In order to facilitate the generation of lighting effect playback instructions corresponding to each target image, a lighting effect description template is pre-stored in the controller of the atmosphere light device. The lighting effect description template can be generated by the controller according to a preset protocol format, corresponding to the sequential position represented by the position information of each light-emitting unit in the layout configuration information. Since the communication between the controller and each light-emitting unit is usually implemented based on a serial communication protocol, when it is necessary to control each light-emitting unit to play the lighting effect, it is only necessary to encapsulate the control data of each light-emitting unit according to the preset protocol format, and then encapsulate these control data in order according to the serial communication protocol to obtain the lighting effect playback instruction. Therefore, the lighting effect description template can represent the sequential position of each light-emitting unit in the atmosphere light according to the specification of the serial communication protocol, and standardize the various attribute items corresponding to the control of each light-emitting unit. When it is necessary to generate the corresponding lighting effect playback instruction later, it is only necessary to call this lighting effect description template and update the data in the attribute items corresponding to each light-emitting unit.

[0115] Each light-emitting unit's corresponding attribute items include a color attribute item. This color attribute item is used to set the color value of the light emitted by the corresponding light-emitting unit, hence the name "luminous color value." Setting the luminous color value is accomplished by assigning a value to the color attribute item.

[0116] Step S5420: Assign the color value attribute item of each light-emitting unit in the light-emitting unit set corresponding to each content object using the image color value corresponding to the main color tone of the content object, thereby setting the image color value as the light-emitting color value in the color value attribute item;

[0117] Since each content object can determine the image color value representing the corresponding main color tone of the content object based on the color value of the pixels in its image content as disclosed in the previous embodiments, for each content object, the image color value of its main color tone can be used to directly assign a color value attribute item to each light-emitting unit in the light-emitting unit set mapped to the content object, and the image color value can be set to the light-emitting color value in the color value attribute item.

[0118] In some embodiments, for each content object, when setting the luminous color value of each light-emitting unit in its light-emitting unit set, the image color value of the main color tone can also be set as the luminous color value of the color value attribute item corresponding to the light-emitting unit located at the center position of the mapping area corresponding to the light-emitting unit set according to the central diffusion principle. Then, along the radial order of the mapping area, with the image color value as the base value, the luminous color value in the color value attribute item of each light-emitting unit diffused along each radial direction is set to the luminous color value obtained by gradiently descending the color value of the base value. In this way, the lighting effect of the entire light-emitting unit set is shaped into an effect of fading in and out in color based on the center position, making the overall lighting effect softer.

[0119] Step S5430: converting a lighting effect description template that sets the light color value of each light-emitting unit in the atmosphere light device into a lighting effect playback instruction for controlling each light-emitting unit to collaboratively play the lighting effect corresponding to the target image.

[0120] After setting the luminous color value of each luminous unit in each luminous unit set corresponding to each content object in accordance with the above method, the data of the lighting effect description template is updated. Accordingly, all the data in the updated lighting effect description template is encapsulated according to the serial communication protocol, and the lighting effect description template can be converted into the lighting effect playback instruction corresponding to the target image.

[0121] After the controller encapsulates the lighting effect playback command for the target image, it can send it to the ambient light. Through the ambient light's connection topology, the individual light-emitting units encapsulated in the lighting effect playback command are ultimately sent to their destination. The light-emitting units then extract their own control data and control the internal light-emitting elements to emit light of the corresponding hue based on the luminous color value in the control data. Each light-emitting unit operates according to the same principle. As a result, all the light-emitting units in the entire display frame formed by the ambient light can coordinately play the lighting effects corresponding to the target image, achieving accurate projection of the color distribution of the image content of each content object in the target image within the display frame.

[0122] As can be seen from the above embodiments, with the help of the lighting effect description template and the mapping relationship between the content object and its lighting unit set, it is possible to quickly complete the assignment of luminous color values ​​for each lighting unit in the atmosphere light, and quickly generate lighting effect playback instructions corresponding to the target image. The generation is rapid, the assignment is accurate, and it is relatively efficient. Based on any embodiment of the method of the present application, before detecting each content object in the target image and obtaining the boundary information corresponding to each content object, it includes:

[0123] Step S4100: continuously collecting interface images from an external terminal device;

[0124] In the exemplary application scenario of this embodiment, the atmosphere lighting device is used to form an image stream based on the interface image of the terminal device, take each interface image in the image stream as the corresponding target image, and generate corresponding lighting effects based on the target image, thereby simulating the light atmosphere on the graphical user interface of the terminal device.

[0125] To this end, the controller can continuously capture interface images from an external terminal device through its image acquisition interface. For details, please refer to the aforementioned embodiments. Each frame of the interface image displayed on the terminal device's graphical user interface can be captured through camera recording, screen projection, or wired transmission. To facilitate subsequent processing, if the interface image received by the controller is not bitmap data, it can first be converted into bitmap data.

[0126] Considering that the interface image of the terminal device usually has a high resolution, that is, a huge total number of pixels, and the number of light-emitting units in the atmosphere lighting device, that is, the basic pixels, is far less than the total number of pixels, the controller can compress each interface image and compress it to a predetermined specification, which can not only ensure the playback of the lighting effect, but also save the system overhead of the controller.

[0127] Step S4200: Eliminate the black bands on the edges of the currently acquired interface image to obtain an image without black bands;

[0128] The controller continuously captures interface images from the terminal device and preprocesses each image obtained at each moment. Each image obtained at each moment is considered the current image. For this image, the controller first detects black pixels at the edge of the full frame, then identifies the rectangular area closest to the outer edge as the edge black band. The controller then crops these black bands from the current image to produce an image without them.

[0129] Eliminating the black edge band takes into account that the effective image content displayed in the terminal device usually occupies the middle of the interface, and the black edge band itself cannot effectively express the image color. Eliminating it can avoid the adverse effects of the black edge area on the lighting effect, highlight the color presentation of the effective image content, and ensure the quality of the played lighting effect.

[0130] Step S4300: increasing the color value of a target pixel belonging to a dark pixel in the image without black bands to obtain an enhanced image;

[0131] In order to avoid the presence of large black pixels in the interface image that affect the brightness of the played lighting effects, the color value of each pixel in the image can be detected on the basis of obtaining the image without black bands. For pixels with color values ​​lower than the preset threshold, they are determined to be dark light pixels. For dark light pixels, their color values ​​are increased according to the preset value or preset ratio, so that they become non-dark light pixels, and the image without black bands is also made into an enhanced image.

[0132] Step S4400: Add the enhanced image to an image frame sequence, and sequentially remove the enhanced image from the image frame sequence and use it as the target image.

[0133] After obtaining the enhanced image corresponding to the current interface image, it is added to the image frame sequence set in the controller's cache. From this image frame sequence, each image frame is dequeued in succession according to a preset dequeuing rule, usually a first-in-first-out rule. The dequeued enhanced image can then be used as the target image for the subsequent processing steps disclosed in the above embodiments of this application to achieve the display of lighting effects based on the target image.

[0134] According to the above embodiments, in the scenario where the atmosphere light device plays the corresponding lighting effect according to the interface image of the terminal device, the controller can adapt to the characteristics of the interface image and perform various targeted pre-processing on it, among which the problem of reduced lighting effect quality caused by black pixels is concentratedly processed, and the color of the interface image is effectively enhanced by eliminating the black bands on the edges and increasing the color value of the black pixels. Then, the color of the interface image is added to the image frame sequence and the corresponding lighting effects are played in sequence, thereby ensuring that the atmosphere light device can play the light atmosphere effect that simulates the color distribution between the various content objects in the interface image according to the interface image, comprehensively improving the quality of shaping the light atmosphere according to the interface image, realizing the effective extension of the light atmosphere of the interface image of the terminal device in the physical space, and enhancing the atmosphere immersion of the user of the terminal device through the corresponding lighting effects. On the basis of any embodiment of the method of the present application, according to the image content corresponding to the boundary information of each content object in the target image, the image color value of the main color tone of each content object is correspondingly determined, including:

[0135] Step S8100: Input the image content of each content object into a preset emotion classification model to infer and determine the emotion attribute conveyed by the image content;

[0136] This application prepares an emotion classification model, which can be composed of a feature extraction model based on a convolutional neural network followed by a classifier, wherein the convolutional neural network is used to extract the deep semantic information of the input image content, and then map it to the classification space corresponding to the classifier to obtain the classification probabilities corresponding to various preset emotion attributes, and determine the emotion attribute with the largest classification probability as the emotion attribute corresponding to the image content, thereby realizing emotion classification of the image content.

[0137] The emotion classification model of the present application is pre-trained using training data. In this training data, selected images containing content objects are used as training samples, and the emotion attributes corresponding to the emotions expressed by these images are set as supervisory labels for these training samples. Using a sufficient number of training sample and supervisory label pairs, the emotion classification model is trained a limited number of times until it reaches a convergence state, indicating that the emotion classification model is suitable for determining the emotion attributes corresponding to the content objects within a given image, and is ready for use in the inference phase. Accordingly, by inputting the image content of each content object into the emotion classification model, the corresponding emotion attributes of the content object can be obtained.

[0138] Step S8200: Determine the image color value corresponding to each content object based on the main color tone corresponding to the emotional attribute of each content object.

[0139] The classification system for the emotional attributes represented by the image content of content objects can be flexibly set by those skilled in the art. For example, in one embodiment, content objects in the musical instrument category can be set to represent the emotional attributes corresponding to joy, while content objects in the work category can be set to represent the emotional attributes corresponding to tension, and content objects in the social category can be set to represent the emotional attributes corresponding to relaxation, and so on. In the classification system, a primary color tone is determined for each emotional attribute to express the corresponding emotional atmosphere. For example, the primary color tone representing joy is represented by a fixed image color value corresponding to orange-red, the primary color tone representing tension is represented by a fixed image color value corresponding to yellow, and the primary color tone representing warmth is represented by a fixed image color value corresponding to lavender, etc. In this way, a mapping relationship is established between each emotional attribute and an image color value. Based on this mapping relationship, the primary color tone corresponding to each content object can be obtained based on the determined emotional attributes of each content object. In turn, the image color value corresponding to this primary color tone can be obtained, which can be used to set the luminous color value of the corresponding luminous unit set of the content object.

[0140] According to the above embodiments, it can be known that based on the image content of the content object, with the help of a pre-trained emotion classification model, the main color tone and image color value corresponding to each content object can be determined, which changes the traditional brute force method of simply mechanically determining the luminous color value of the corresponding luminous unit set based on the color value of the pixel in the image content. The lighting effect of the atmosphere lighting device is set based on the image color value obtained after translating the emotional attribute of the content object, so that the lighting effect of the atmosphere lighting device can better match the user's subjective feelings and better create an atmosphere effect. Based on any embodiment of the method of the present application, after determining the image color value corresponding to each content object according to the main color tone corresponding to the emotional attribute of each content object, it includes:

[0141] Step S8300: input the target image into the emotion classification model, and determine the emotion attribute conveyed by the target image by reasoning;

[0142] Although each content object can theoretically represent a corresponding emotional attribute, sometimes the target image composed of the image contents of several content objects may not necessarily express the same overall emotional feeling as the emotional attribute expressed by each content object. For example, the target image as a whole conveys a joyful emotion, but some of the content objects convey a depressed emotion, while multiple content objects convey a joyful emotion. In similar cases, based on step S8200, the image color values ​​corresponding to each content object can be further reconciled to obtain the final image color value, and then the luminous color value of each light-emitting unit in the corresponding light-emitting unit set can be set according to this re-determined image color value.

[0143] Based on this, the target image can be treated as an independent entity and input into the emotion classification model of the present application. Using the capabilities acquired during the training of the emotion classification model, the corresponding emotional attributes of the target image can be determined. To meet the needs of this embodiment, when training the emotion classification model, images composed of the image contents of multiple content objects can also be used as training samples. The corresponding emotional attributes can be provided as supervisory labels and input into the emotion classification model for enhanced training, enabling it to more accurately determine the emotional attributes of the target image composed of the image contents of multiple content objects.

[0144] Step S8400: Blending the image color values ​​corresponding to the various content objects according to the main color tone corresponding to the emotional attribute of the target image.

[0145] Similarly, based on the mapping relationship between emotional attributes, dominant tones, and image color values, the dominant tones corresponding to the target image can be determined based on the emotional attributes of the target image. This dominant tones' image color values ​​can then be used to adjust the image color values ​​corresponding to each content object in the target image. This can be achieved in a variety of ways, such as by taking the average or weighted average of the image color values ​​corresponding to each content object and the image color values ​​corresponding to the target image. Alternatively, the color system of the target image's image color values ​​can be used as the base tone, and the color system of the image color values ​​corresponding to each content object can be transformed.

[0146] According to the above embodiments, by using the image color values ​​corresponding to the emotional attributes of the target image to harmonize the image color values ​​corresponding to each content object, it is possible to ensure that the emotional value conveyed by each content object is more consistent with the emotional value conveyed by the target image as a whole, making the emotional semantics expressed by the lighting effect more accurate, and improving the lighting effect quality when the ambient lighting device plays the lighting effect according to the target image. Based on any embodiment of the method of the present application, harmonizing the image color values ​​corresponding to each content object according to the main color tone corresponding to the emotional attribute of the target image includes:

[0147] Step S8410: determining the target color system to which the target image belongs based on the main color tone corresponding to the emotional attribute of the target image;

[0148] According to the disclosure of the above embodiments, after the target image determines its emotional attributes, the image color value corresponding to its main color tone can be found. Then, the target color system to which it belongs can be determined based on the image color value. The correspondence between image color values ​​and color systems can be preset in advance. For example, image color values ​​corresponding to red, yellow, etc. can be classified as warm colors, and image color values ​​corresponding to gray, blue, etc. can be classified as cool colors, etc. It is not difficult to understand that by comparing whether the image color value falls within the color gamut corresponding to a certain color system, the target color system to which the image color value belongs can be determined. Once the target color system to which the target image belongs is determined, it can be used to reconcile the image color values ​​of the image content of each content object in the target image.

[0149] Step S8420: Detect whether the main color corresponding to the emotional attribute of each content object belongs to the target color system, and transform the main color of each content object that does not belong to the target color system to the target color system with reference to the same reference.

[0150] In order to ensure that the image color values ​​corresponding to each content object can comply with the target color system corresponding to the target image, in this embodiment, the main color tone of the content object, that is, its image color value, can be detected one by one to see whether it belongs to the color gamut range corresponding to the target color system, so as to determine whether the main color tone of the corresponding content object belongs to the target color system. That is, when the image color values ​​of one or more content objects do not belong to the target color system, these image color values ​​that do not belong to the target color system can be transformed with reference to the same preset benchmark. For example, based on a predetermined amount as a benchmark, the predetermined amount is superimposed on these image color values ​​to change each image color value to obtain a new image color value, which is used to match the color of each light-emitting unit in the corresponding light-emitting unit set.

[0151] According to the above embodiments, it can be seen that the target color system to which the main color corresponding to the emotional attribute of the target image belongs is used to detect whether the main color of each content object serves the target color system, and the image color value of the main color of the content object that does not belong to the target color system is adjusted to make the image color value of each content object obey the main color of the target image as much as possible, which can further maintain the ability of each content object to express the main color of the target image and ensure the quality of the lighting effect.

[0152] Please refer to Figure 9. In one embodiment, the present application provides a method for matching lighting effects for an ambient light device. The method is primarily implemented on the controller side of the ambient light device and is executed by the controller chip of the controller. In this embodiment, step S5100 shown in Figure 5 further includes: performing image semantic segmentation based on the target image to identify the content area where the content object is located and the image content in the content area. As shown in Figure 9, the method for matching lighting effects for an ambient light device in this embodiment includes:

[0153] Step S7100: performing image semantic segmentation based on the target image to identify the content area where the content object is located and the image content in the content area;

[0154] Therefore, for each target image, when it is necessary to determine the content area of ​​each content object, detection of the content area of ​​each content object must be performed to obtain the content area of ​​each content object.

[0155] In one embodiment, in order to obtain the regional images corresponding to each content object in the target image, in this application, image semantic segmentation can be performed based on the deep semantics of the target image based on a pre-trained deep learning model, including a traditional non-prompt image segmentation model or a prompt image segmentation model with image interaction processing capabilities, to obtain the corresponding content area.

[0156] In other embodiments, before determining the content area of ​​each content object, target detection can be performed on the target image with the help of a target detection model. After detecting one or more targets as the regional image of the content object, the regional images of each content object are successively input into the deep learning model to perform image semantic segmentation, and the content area of ​​each content object is successively obtained.

[0157] Step S7200: Determine the image color value corresponding to each content object according to the image content of each content object;

[0158] To achieve lighting effects, the primary color of each content object must be determined based on its image content, and represented by a corresponding image color value. To this end, the image color value of the primary color of each content object is determined based on its corresponding image content. There are various ways to determine the image color value of the primary color of a content object based on its image content. In some further embodiments, the emotional attributes of the target content object can be classified, the primary color corresponding to the emotional attribute can be determined, and a preset image color value for the primary color can be obtained as the image color value corresponding to the target content object.

[0159] Before determining the image color value in the above embodiment, the image content of each content object can be pre-processed to further improve the display effect of the determined image color value of the content object. For example, pixels in the image content with color values ​​below a preset threshold can be filtered out, and then the image color value of the main hue can be determined based on the remaining valid pixels. The preset threshold can be used to measure the brightness of the pixels.

[0160] Step S7300: Determine, according to the content area of ​​each content object, a light unit set to which each content object is mapped in all the light units of the atmosphere light device;

[0161] Step S7400: Control each light emitting unit set corresponding to each content object to emit light according to the image color value determined from the image content of each content object.

[0162] Compared to the prior art, this embodiment has several advantages. First, the present application performs semantic image segmentation on the target image to obtain the content area and area image of each content object therein. Based on the content area of ​​each content object, the present application determines the corresponding light unit set of each content object in the ambient light device. Corresponding to the image content of each content object, the present application determines the image color value corresponding to each content object. Utilizing the correspondence between the image color value associated with the content object and the light unit set, the present application controls the display frame presented by all light units of the ambient light device to coordinately play the lighting effect. Each light unit in the display frame can display the lighting effect in a partitioned manner according to each content object in the target image, so that the lighting effect of each light unit in each partition can maintain a corresponding relationship with the image color value of the content object corresponding to the partition. This allows the entire display frame of the ambient light device to more accurately present a corresponding color layout corresponding to the color of each content object. Furthermore, the ambient light device can relatively accurately reproduce the distribution of the content objects in the target image through the color of the light emitted by the light units, thereby improving the simulation of the lighting effect to the lighting atmosphere of the target image and making the lighting atmosphere rendered by the ambient light device more realistic and accurate. Secondly, the present application uses the content area of ​​the content object obtained through image semantic segmentation as the basis for partition mapping between the luminous unit set and the image color value of the content object. This content area is obtained according to the outline of the content object in the target image, rather than using a regular rectangular area as the basis for partition mapping. This makes the correspondence between the content object and the luminous unit it covers more accurate, and the color information of the image content near the boundary of the content object will not be interfered with by the color information of other content objects. The boundaries of each content object are relatively clearer, the color layout projection relationship is more accurate, and the simulation of the light atmosphere of the target image by the atmosphere lighting device is more delicate, realistic, and soft, and the lighting effect created by the atmosphere lighting device is more refined. In addition, the atmosphere lighting device implemented according to the present application has a more realistic, accurate, and refined light atmosphere created by the target image. Therefore, when the atmosphere lighting device uses the desktop image of the terminal device as the target image and plays the corresponding lighting effect according to the target image, the desktop image's picture atmosphere can be effectively extended to the physical space under the rendering of the light atmosphere of the atmosphere lighting device, thereby enhancing the immersion of the terminal device user.

[0163] Based on any embodiment of the method of the present application, performing image semantic segmentation based on the target image to identify the content area where the content object is located and the image content in the content area includes:

[0164] Step S77100: Segment the target image into a plurality of region images corresponding to the content objects;

[0165] Step S77200: Determine a region mask of the content object of each region image relative to the region image using a default image segmentation model, wherein the region mask represents a content region of the corresponding content object in the region image;

[0166] In this embodiment, the deep learning model used to detect the content area can be a default image segmentation model that has been pre-trained to reach a convergence state, which is a non-prompt image segmentation model, for example, various models of the U-Net series and various models of the Deeplab series.

[0167] Step S77300: Convert the region mask of each content object into a corresponding full-image mask, where the full-image mask represents the content region of the corresponding content object according to the size specification of the target image.

[0168] Based on the above embodiment, image semantic segmentation is performed based on the target image to identify the content area where the content object is located and the image content in the content area, including:

[0169] Step S77400: calling a target image and segmentation prompt information, wherein the segmentation prompt information includes coordinate information corresponding to a position where a content object may appear in the target image;

[0170] Regarding the calling of the target image, as disclosed in the previous embodiment, each target image can be called from the image frame sequence to play the lighting effect, which will not be elaborated here. Regarding the segmentation prompt information, its role is to guide the deep learning model to identify each content object from the target image with a certain recognition rule. Therefore, the segmentation prompt information can be determined in advance. In this embodiment, the recognition rule used in the segmentation prompt information is expressed as instructing the corresponding prompt-type segmentation model to identify the corresponding content object in the target image based on the coordinate information of the recognition position specified relative to the target image. The coordinate information of each recognition position can be determined in advance and then stored as a corresponding template so that the segmentation prompt information can be obtained by calling the template.

[0171] Step S77500: Using an image hint segmentation model, based on the segmentation hint information, a full-image mask corresponding to each content object in the target image is detected, wherein the full-image mask represents the content area of ​​the corresponding content object according to the size specification of the target image.

[0172] In view of the rapid development of the image processing capabilities of the hint-based image segmentation model, this embodiment proposes a solution for determining the content area of ​​each content object in the target image by means of the image hint-based segmentation model.

[0173] The sum of the content areas of all content objects perfectly fills the entire target image. Therefore, the full-image mask represents the content areas corresponding to all content objects in the target image, according to the target image's dimensions. It's easy to understand that, based on the coordinates of each identification location specified in the segmentation hint information, the image hint segmentation model can reference each corresponding identification location to determine the content area of ​​the content object that may exist at each identification location in the target image.

[0174] In the above embodiments, segmentation prompt information is utilized, and with the help of the interactive capabilities of the image prompt segmentation model, the content objects in the target image are accurately identified according to the recognition rules specified in the segmentation prompt information. In addition to achieving the same technical advantages of the non-prompt image segmentation model in lighting effects, it can also open up certain interactive capabilities and enrich the recognition methods of content objects, so that the atmosphere lighting equipment can present corresponding lighting atmosphere effects according to the color distribution of content objects distinguished in a specified manner, which is more intelligent.

[0175] Please refer to Figure 3. The ambient light in Figure 2 is arranged around the display of the terminal device and becomes a border light. The border light can be surrounded by a single or multiple light strips connected to the bus. Each light strip 21 includes a plurality of lamp beads 210 connected in series, and each lamp bead 210 serves as a light-emitting unit. By analyzing it and sorting the characteristic information of each lamp bead in the result data, the position of each lamp bead in the frame 4 where the entire ambient light is arranged can be determined. Therefore, each lamp bead can be regarded as a light-emitting unit and can be understood as a basic pixel. When constructing the lighting effect playback instruction, the subsequent controller can set the corresponding light-emitting color value for each basic pixel according to the position information of each lamp bead and actual needs.

[0176] Referring to FIG. 10 , in one embodiment, the color matching method for the ambient light device of the present application is mainly implemented on the controller side of the ambient light device and executed by the control chip of the controller, including:

[0177] Step S88100: Acquire frame shape information of the ambient light device, where the frame shape information represents each light-emitting unit in the ambient light device as basic pixels orderly arranged on each frame constituting the same frame;

[0178] The ambient light in the ambient light device of the present application will be enclosed in a specific frame shape during actual use, usually a rectangular frame. Of course, it can also be enclosed in frames of other shapes, such as a rectangular frame, a hexagonal frame, etc., and it is only necessary to lay out the various light strips of the ambient light according to the requirements of the specific frame shape according to actual needs. The frame thus laid out is composed of a corresponding plurality of frames, each of which has a plurality of light-emitting units belonging to the frame. The frame shown in Figure 2 is a rectangular frame, and accordingly, it is composed of four frames.

[0179] The light strips occupying all the frames within the frame can be a single strip or composed of multiple strips. As long as the controller can centrally identify the position of the light-emitting units in each strip within the entire frame, it can effectively control the light-emitting units in each position. The frames within the frame can be laid out on a flat surface, on a curved surface, or extend from one surface to another. The specific layout is flexible and user-defined, depending on the actual effect required.

[0180] The controller and the light strip communicate data with each other via a serial communication protocol. Therefore, the controller can store in advance or determine the layout configuration information of the atmosphere light device through detection. The layout configuration information usually includes the position information of each light strip and each light-emitting unit therein in the data communication link. The position information indicates the order of each light-emitting unit in the data communication link. When the controller sends instructions or data to the entire frame, it can encapsulate the corresponding instructions or data of each light-emitting unit in accordance with the serial communication protocol and the order of each light-emitting unit. When the instructions or data are sent to the atmosphere light, each light-emitting unit can call its own corresponding instructions or data in sequence. The same applies when each light-emitting unit reports its own position to the controller.

[0181] When the atmosphere light leaves the factory, or at least when the user is laying out the frame of the atmosphere light, the shape of the frame is already known in advance, that is, the length of each frame and the number of light-emitting units therein. Therefore, on the controller side, the layout configuration information can also be combined with the number of light-emitting units of each frame to constitute the frame shape information.

[0182] Therefore, the frame shape information actually includes not only a description of the number of light-emitting units that constitute each frame, but also a description of the position information of each light-emitting unit in the corresponding frame. Through the frame shape information, the various light-emitting units in the atmosphere light device are represented as basic pixels arranged in an orderly manner on each frame of the same frame. In this sense, the frame can be regarded as a display frame with a hollowed-out middle. The present application projects the lighting effect of the target image onto the frame, which is actually to project the lighting effect onto this display frame.

[0183] Since the light-emitting units on a frame are usually arranged at equal intervals, determining the number of light-emitting units in each frame border based on the frame shape information is equivalent to determining the length of each frame, and further determining the frame ratio. For a rectangular frame, this is the frame ratio. Based on this frame ratio, it can be easily scaled to correspond to the target image.

[0184] Step S88200: performing image semantic segmentation based on the target image to determine the image content and content area of ​​each content object in the target image;

[0185] In order to project the main color of the content object in the target image into the ambient light, it is necessary to first perform image semantic segmentation on the target image to determine the image content and content area of ​​the content object. The specific determination method can refer to the above-mentioned embodiment.

[0186] Step S88300: Using the frame as the outer frame of the target image, assigning the content areas of the respective content objects to the respective borders of the frame, so that each light-emitting unit of the frame corresponds to one of the content areas, thereby obtaining a light-emitting unit set mapped to each content area;

[0187] In order to facilitate the projection of the main colors of each content object in the target image onto the various borders of the frame, a one-to-one correspondence between the various sides of the target image and the various borders of the frame is usually established first, so that the frame can be regarded as the outer frame of the target image for subsequent projection. When the frame is a rectangular frame by default, it just corresponds to the four sides of the target image. In this case, the correspondence between the two is clear and no additional processing is required. In some embodiments, when the frame is other shapes such as a pentagon or a hexagon, a virtual frame can be constructed based on the frame shape information, and the virtual frame can be scaled to a size roughly equivalent to the target image, and the target image can be cropped accordingly, so that the target image is more convenient to achieve one-to-one correspondence with the frame constructed by the atmosphere light.

[0188] Since the image edges of the target image correspond one-to-one to the borders of the atmosphere light frame, the content areas of each content object in the target image can be projected onto the various borders of the frame according to certain preset rules, and a mapping relationship between the content object and the border and the light-emitting unit in the frame can be established.

[0189] For example, based on the proximity principle, each content area can be projected onto the light-emitting unit of the border closest to its geometric center, or based on the proportion distribution principle, all the light-emitting units in the frame can be distributed to each content area according to the ratio between the area of ​​each content area, and then determined based on the light-emitting units corresponding to each content area whether the corresponding border falls on one or more.

[0190] In one embodiment, the entire frame can be considered as a display frame, and multiple unit frames can be divided along the periphery of the display frame. Then, based on the overlap between the projections of the content area and each unit frame on the plane, the content area is mapped to the light-emitting units covered by the unit frame to which it belongs, thereby obtaining the light-emitting unit set corresponding to each content area. In this case, the light-emitting unit set of a content area may be distributed on two adjacent frames.

[0191] In another embodiment, all the light-emitting units of the entire frame can be regarded as a line. Similarly, according to the proportion distribution principle, the light-emitting units on the entire line are allocated to each content area according to the ratio between the area of ​​each content area, thereby determining the light-emitting unit set mapped to each content area. Similarly, the light-emitting unit set of a content area may also be distributed on two adjacent borders.

[0192] In summary, according to certain preset rules, the content area of ​​each content object in the target image can be mapped to one or more borders of the frame, corresponding to one or more light-emitting units. These covered light-emitting units constitute the light-emitting unit set mapped to the content area. Based on this mapping relationship, it is not difficult to understand that each light-emitting unit corresponds to only one content area of ​​the content object, and each content area generally corresponds to one or more light-emitting units.

[0193] Please refer to the examples in Figures 6 and 7. Figure 6 is an exemplary target image of the present application, and Figure 7 is a schematic diagram of the segmentation results obtained by performing image semantic segmentation on the target image in Figure 6, that is, the mapping relationship between each content object and the set of light-emitting units in the atmosphere light. In Figure 7, ABCDE respectively represent the image content of each content object, which also shows the mapping relationship between each image content and each set of light-emitting units in the atmosphere light in the border light state. It can be seen that various forms of atmosphere lights can establish corresponding mapping relationships between each content object and the corresponding set of light-emitting units.

[0194] Step S88400: Determine the main color tone of each content object according to the image content of each content object, and control the lighting unit set mapped to each content object to play a lighting effect according to the main color tone of each content object.

[0195] As the basis for achieving lighting effect projection, it is necessary to determine the main color tone of each content object according to the image content of each content object and use a corresponding image color value to represent it. The specific control method can refer to the embodiment described above.

[0196] Compared with the prior art, this embodiment has many advantages. First, the present application is directed to an ambient light device in the form of a border light. After obtaining its frame shape information, the present application detects the content area and image content of each content object from the target image, and projects the content area of ​​each content object into each frame of the frame defined by the frame shape information, so that each content object has its own set of light-emitting units located on one of the frames, so that all the light-emitting units in the frame are allocated to the corresponding use of each content object. On this basis, according to the main color tone of the image content of each content object, each light-emitting unit in the set of light-emitting units corresponding to the content object in the ambient light device is controlled to emit light of corresponding color, so that the entire frame of the ambient light device can display the lighting effect corresponding to the main color tone of each content object in the target image, and the main color tone of each content object is spread throughout the entire frame to render the light atmosphere. As a result, the ambient light device can more accurately correspond to the main color of each content object to shape the corresponding color atmosphere, thereby improving the simulation degree of the lighting effect to the light atmosphere of the target image. Secondly, the present application determines the set of light-emitting units mapped to the content object based on the projection relationship between the content area and the border of the content object, rather than using a regular rectangular area as the basis for partition mapping. This allows the color atmosphere of each content object in the target image to be effectively transferred to the ambient light device in the form of a border light, rather than mechanically intercepting the edge image information of the target image to determine the color of each light-emitting unit in the border light. Therefore, the ambient light device in the form of a border light can achieve an effect closer to the light atmosphere of the original image when simulating the light atmosphere of the target image. In addition, the ambient light device implemented according to the present application is more realistic and effective in creating a light atmosphere in accordance with the target image. Therefore, when the ambient light device uses the desktop image of the terminal device as the target image and plays the corresponding lighting effect in accordance with the target image, the ambient light device can effectively extend the picture atmosphere of the desktop image to the physical space under the rendering of the light atmosphere of the ambient light device, thereby enhancing the immersion of the user of the terminal device.

[0197] On the basis of the above embodiment, the frame is used as the outer frame of the target image, and the content area of ​​each content object is allocated to each border of the frame, so that each light-emitting unit of the frame corresponds to one of the content areas, and a light-emitting unit set mapped to each content area is obtained, including:

[0198] Step S88310: Determine the relationship between the position information of each light-emitting unit in the frame and each frame border of the frame according to the frame shape information;

[0199] As mentioned above, the frame shape information includes position information representing the position of each light-emitting unit in the atmosphere light, and the number of light-emitting units corresponding to each border of the frame. Based on this, the light-emitting units corresponding to each border of the frame can be determined in sequence according to the number of light-emitting units corresponding to each border, and the coverage correspondence relationship information between the border and the light-emitting unit can be established, and the affiliation between the position information of each light-emitting unit and its corresponding border can be clarified.

[0200] Step S88320: Calculate the distance between the content area of ​​each content object and each frame of the frame, determine the frame with the shortest distance as the target frame corresponding to the content area, and determine the projection size of the content area when projecting it onto the target frame;

[0201] The target image typically contains multiple content objects. Alternatively, the user can control the model to identify multiple content objects in the target image by setting the recognition position, preferably four or more. In this case, each frame within the frame can correspond to at least one content object.

[0202] In order to establish a mapping relationship between the light-emitting units in each frame of the frame and the content object, in this embodiment, for the case where the frame is used as the outer frame of the target image, so that the image edge of the target image and the frame frame establish a one-to-one correspondence, the content area of ​​each content object can be calculated to determine its distance to each frame of the frame. This is done by first determining the geometric center position of the content area, and then calculating the distance to each frame based on this geometric center position. Then, for each content object, the frame with the smallest distance is used as the target frame corresponding to the light-emitting unit set determined for that content object. Since each content area has a certain span, when this span is projected onto the target frame, a projected size is obtained. This projected size can, to a certain extent, reflect the aspect ratio of this content area relative to the content areas of other content objects in the entire target image.

[0203] Step S88330: Allocate the total number of light-emitting units of the target frame to each content area in the corresponding content areas according to the proportional relationship of the projection sizes of all content areas corresponding to each target frame, and obtain a light-emitting unit set mapped to each content area.

[0204] It's easy to understand that, since multiple content areas may be projected onto the same frame, all light-emitting units within the same frame may need to be allocated to multiple content areas. Accordingly, each frame within the frame has one or more corresponding content areas, and all light-emitting units within the same frame need to be allocated between the corresponding one or more content areas to establish a mapping relationship between content areas and light-emitting unit sets.

[0205] To this end, the target border is used as a unit to calculate the proportional relationship between the projected sizes of each content area projected onto it. Then, the total number of light-emitting units in the target border is allocated according to the proportion of each content area. For example, if the total number of light-emitting units in the same target border is n, and the ratio of the projected sizes of content area 1 and content area 2 is 6:4, then content area 1 will receive 0.6n light-emitting units as a light-emitting unit set, while content area 2 will receive 0.4n light-emitting units as a light-emitting unit set.

[0206] Similarly, when determining the correspondence between the order of the light-emitting unit sets of the same target border and the respective content areas, it can be determined based on the position of each content area projected onto the target border. For example, when content area 1 is closer to the left side of the target border relative to content area 2, content area 1 is mapped to the light-emitting unit set consisting of 0.6n light-emitting units on the left side of the target border, while content area 2 is mapped to the light-emitting unit set consisting of 0.4n light-emitting units on the right side of the target border.

[0207] When establishing the mapping relationship between the content area and the light-emitting unit set in the above manner, if there are individual borders that are not mapped with the content objects due to the small number of content objects, the background of the target image can be used as the content object, and its corresponding content area can be determined. All the light-emitting units of these borders can be used to form a light-emitting unit set that is mapped to the content area of ​​the background.

[0208] According to the above process, each content area is first projected onto the corresponding frame of the frame to obtain a projection size, and then the light-emitting units of the frame are allocated according to the proportional relationship between the projection sizes of all the content areas corresponding to each frame, so that the mapping relationship between the content area and the light-emitting unit set can roughly follow the constrained proportion of the main color tone of the image content in the content area to the overall color atmosphere presented by the target image, and in a relatively simple and efficient way, the color atmosphere of the target image can be effectively projected onto the atmosphere light device in the form of a frame light for display, which is convenient for implementation and can ensure efficient operation.

[0209] Referring to FIG. 11 , in one embodiment, the lighting effect playback control method of the ambient light device of the present application is implemented on the controller side of the ambient light device and executed by the control chip of the controller, including:

[0210] Step S9100: Acquire location information of the identification location based on the interface canvas to construct a segmentation recognition rule, wherein the interface canvas is used to represent a display frame composed of each light-emitting unit in the atmosphere light device as a basic pixel;

[0211] The controller can be equipped with a display screen and buttons to achieve built-in human-computer interaction capabilities. When it communicates with a terminal device, it can also reuse the terminal device's human-computer interaction capabilities. Alternatively, when the controller is implemented on a terminal device, the terminal device itself has human-computer interaction capabilities. With the support of this capability, by displaying an interface canvas in a graphical user interface, the position information corresponding to one or more user-specified recognition locations is obtained based on the interface canvas. The position information of these recognition locations is then constructed into segmentation recognition rules, which can be used to perform image semantic segmentation on the target image.

[0212] As shown in Figure 6, the interface canvas is located in the upper area of ​​the screen and is basically rectangular. The size of this rectangle can correspond to the display frame formed by the various light-emitting units of the atmosphere light device, or it can correspond to the size of the target image. In one embodiment, the size of the display frame of the atmosphere light device can be scaled to the screen frame to obtain the corresponding size as the size of the interface canvas. According to the size, an interface canvas is set, and the interface canvas is displayed in the graphical user interface, as shown in the box above Figure 6. After determining the size of the interface canvas, if it is necessary to establish a correspondence between the size of the target image and the size of the interface canvas, this can be achieved by scaling or cropping the target image accordingly, so that the size of the target image finally obtained has a scaling ratio relationship with the size of the interface canvas. In this way, the target image, the interface canvas, and the display frame of the atmosphere light device form a proportional relationship based on the same reference coordinate system with different scaling ratios, which facilitates the calculation of various position information.

[0213] Accordingly, the frame of the interface canvas corresponds to the frame of the target image. It is just that when the interface canvas is displayed in the graphical user interface, it is scaled accordingly to adapt to the display size of the graphical user interface to facilitate user operation. The position or area set on the interface canvas can also be associated with the corresponding scaling ratio and uniquely mapped to the target image to obtain the corresponding position and area. Similarly, the canvas of the interface canvas also corresponds to the display frame of the atmosphere light device. The position or area specified on the interface frame can be mapped to the display frame of the atmosphere light device according to the known scaling ratio relationship to obtain the corresponding mapping area, so that each light-emitting unit falling within the mapping area can be determined.

[0214] The recognition location set by the user in the interface canvas can be any of a variety of geometric forms. For example, you can set the location information represented by point coordinates by specifying individual points, set the location information represented by a set of pixel coordinates by circling, set the location information represented by a window by using a rectangular box, and so on. In short, the recognition location specified by the user can be any of a point, a line, or a surface. Accordingly, the recognition location can be represented as corresponding coordinate information, realizing the representation of the location information.

[0215] When the user completes the determination of one or more recognition locations in the interface canvas, the corresponding position information of each recognition location is generated. These position information are spliced, encapsulated or encoded together in a predetermined format to form a segmentation recognition rule, which can be used to assist in the implementation of image semantic segmentation.

[0216] Step S9200: determining a target image from the video stream, performing image semantic segmentation on the target image according to the segmentation and recognition rules, and determining image content and content areas of content objects corresponding to the respective recognition positions in the target image;

[0217] The target image is used as a reference for the color distribution of the lighting effects played by the ambient light device, so it is essentially an environmental reference image. The source of the environmental reference image can be obtained from a real camera shot, called from the cache or video memory, read directly from an image file, or received via wired or wireless transmission. For example, the environmental reference image can be a real-life image captured by a camera or an interface image of a terminal device. The interface image can be captured by a camera or read or screenshotted from the terminal device.

[0218] A video stream can be obtained by shooting with a camera, taking screenshots, or reading streaming media files. The video stream contains multiple image frames. You can call each image frame as the target image according to the timestamp of the image frame to perform the operations of this step and the following steps to play the lighting effect corresponding to each target image.

[0219] In one embodiment, considering that the atmosphere lighting device plays corresponding lighting effects based on multiple consecutive target images, each target image may come from the same video stream, and two adjacent target images may be two image frames of the same video content, and the image content change between these two image frames may be small, in this case, the frame difference information between the current target image and the previous target image can be calculated. When the change amplitude presented in the frame difference information is less than a preset threshold, the content area corresponding to the previous target image can be used. Otherwise, the content area of ​​each content object in the current target image is re-determined.

[0220] In order to obtain the regional images corresponding to each content object in the target image, in this application, a pre-trained deep learning model can be used. The deep learning model is preferably a prompt-type image segmentation model. The prompt-type image segmentation model can perform image semantic segmentation based on the deep semantics of the target image under the constraints of the segmentation recognition rules obtained after specifying the recognition position based on the interface canvas, and obtain the content areas corresponding to each content object in the target image.

[0221] Hint-based image segmentation models typically have two modes: automatic segmentation mode and non-automatic segmentation mode. In automatic segmentation mode, the model can combine the target detection network to perform target detection on the target image. After detecting one or more targets as the regional images of the content objects, the regional images of each content object are input into the image segmentation network within the model to perform image semantic segmentation. This can obtain the content region of each content object, and ultimately select the content object whose content region matches the various identification positions specified in the segmentation recognition rules as the target content object. In non-automatic segmentation mode, the model directly infers based on the target image and the respective recognition rules to obtain the various content objects and their content regions corresponding to the various recognition positions specified in the segmentation recognition rules.

[0222] In some embodiments, for each content object, the image segmentation model can first determine its region image and then determine its corresponding region mask. A region mask simply represents the content region of a single content object within its region image. Therefore, to facilitate alignment with the target image, the region mask for each content object can be further converted into a full-image mask based on the target image's dimensions. Background content other than the content object in the target image can be treated as an independent content object, and the content region of this content object is accordingly determined. This allows for rapid extraction of the content region and image content of each content object based on the mask, establishing a corresponding mapping relationship for rapid access.

[0223] Step S9300: Determine, based on the mapping relationship between the target image and the display frame, a set of light-emitting units within a corresponding mapping area of ​​each content object in the display frame;

[0224] Since the display frame establishes a size-specification mapping relationship with the target image through the interface canvas, and the ambient light device typically uses layout configuration information to indicate the position information of each light-emitting unit relative to the reference coordinate system, usually expressed as coordinate information, which actually defines the position information of each light-emitting unit on the display frame, it is not difficult to understand that, with the help of this correspondence between the display frame and the target image, the mapping area of ​​each content object in the display frame of the ambient light device can be determined based on the content area of ​​each content object. In this way, each content object in the target image can obtain a corresponding light-emitting unit set based on its content area, so that a one-to-one mapping relationship is established between the content object and the light-emitting unit set. In fact, a one-to-one mapping relationship is established between the content area corresponding to the image content of each content object and each light-emitting unit set.

[0225] Although FIG11 does not provide a diagram of the mapping area, it is not difficult to understand that, given a content area, a mapping area can also be obtained in the display frame of the spliced ​​lamp in FIG11 , and then it can be determined that the various light-emitting units within the range covered by the mapping area constitute a light-emitting unit set. For the same mapping area, it is allowed to cover light-emitting units on different light blocks across multiple light blocks.

[0226] Figure 7 is an exemplary target image of the present application, and Figure 8 is a schematic diagram of the segmentation results obtained by performing image semantic segmentation on the target image in Figure 7, that is, the mapping relationship between each content object and the light-emitting unit set in the atmosphere light. In Figure 8, ABCDE respectively represent the image content of each content object, which also shows the mapping relationship between each image content and the light-emitting unit set in the atmosphere light in the border light state. It can be seen that various forms of atmosphere lights can establish corresponding mapping relationships between each content object and the corresponding light-emitting unit set.

[0227] Step S9400: Control each light-emitting unit in the corresponding light-emitting unit set to play a corresponding lighting effect according to the main color tone of the image content of each content object.

[0228] Compared with the prior art, this embodiment has many advantages. First, the present application first obtains the user's desired recognition position by representing the interface canvas of the atmosphere light device, so as to mark the key recognition area, and then performs image semantic segmentation on the target image to obtain the content area and image content of each content object corresponding to the position information of the recognition position calibrated by the user. According to the mapping relationship between the target image and the display frame, the mapping area of ​​each content object in the display frame is determined, thereby determining the corresponding light-emitting unit set of each content object in the atmosphere light device. Then, according to the image content of each content object, the image color value corresponding to each content object is determined, and the main color tone reflected by the image content of each content object is used to control the relative brightness of each content object. The corresponding set of light-emitting units emits light, so that all the light-emitting units in the atmosphere light device can be divided into zones according to the various content objects of the target image to determine the display color, thereby coordinating the overall lighting effect, thereby making the lighting effect of each light-emitting unit in each mapping area able to maintain a corresponding relationship with the main color tone of the content object corresponding to the mapping area, so that the entire display frame of the atmosphere light device can more accurately correspond to the color of each content object and present a corresponding color layout, and also enable the atmosphere light device to relatively accurately reproduce the distribution of content objects in the target image through the color of the light emitted by the light-emitting unit, thereby improving the degree of simulation of the light atmosphere of the target image by the lighting effect, and making the light atmosphere rendered by the atmosphere light device more realistic and accurate. Secondly, the present application allows users to customize the possible identification positions of content objects in the target image to guide the image semantic segmentation process, which can further improve the accuracy of the image semantic segmentation results, so that the determined content objects basically meet the user's expectations, avoid the final lighting effect being too scattered due to too many content objects in the target image, make the lighting effect concise as a whole and highlight the key points, and make the overall atmosphere expressed by the entire lighting effect more accurate and concentrated. Again, the present application uses the content area of ​​the content object obtained through image semantic segmentation as the basis for partition mapping between the luminous unit set and the main color tone of the content object. This content area is obtained according to the outline of the content object in the target image, rather than using a regular rectangular area as the basis for partition mapping, so that the correspondence between the content object and the luminous unit it covers is more accurate, and the color information of the image content of the content object near its boundary will not be interfered with by the color information of other content objects. The boundaries of each content object are relatively clearer, and the color layout projection relationship is more accurate. The simulation of the light atmosphere of the target image by the atmosphere lighting device is more delicate, realistic and soft, and the corresponding lighting effect created by the atmosphere lighting device is also more refined.

[0229] In addition, the atmosphere lighting device implemented according to the present application is more realistic, accurate and refined in shaping the light atmosphere in accordance with the target image. Therefore, when the atmosphere lighting device takes the desktop image of the terminal device as the target image and plays the corresponding lighting effect in accordance with the target image, the light atmosphere of the desktop image can be effectively extended to the physical space under the rendering of the light atmosphere of the atmosphere lighting device, thereby enhancing the sense of immersion of the terminal device user.

[0230] Based on the above embodiment, performing image semantic segmentation on the target image according to the segmentation recognition rule may include:

[0231] Step S9211: Using an image-cued segmentation model in automatic segmentation mode, determine a full-image mask corresponding to each content object in the target image, and window position information corresponding to a regional image of each content object in the target image, wherein the full-image mask represents the content region of the corresponding content object according to the size specifications of the target image;

[0232] A hint-based image segmentation model, also known as an image hint segmentation model, has the ability to determine the mask of a target content object based on given constraints. The image revealing segmentation model in this embodiment consists of a target detection network and a hint-based image segmentation network. The target detection network is used to determine the window position information, category, and confidence level corresponding to the regional images of multiple content objects within the target image. The target boxes, along with the target image, can be input into the hint-based image segmentation network as segmentation hint information, controlling the segmentation network to perform image semantic segmentation on the corresponding content objects, thereby obtaining full-image masks representing the content regions where the image content of these content objects resides.

[0233] Step S9212: The image prompt segmentation model matches each position information in the segmentation recognition rule with the window position information of each content object, and selects each content object that achieves the match as the target content object;

[0234] The image prompt segmentation network obtains the full-image mask of the target image, which indicates the content area where each content object is located. However, some of these content objects may not meet the user's expectations. In order to determine the content object that the user expects to select, for the segmentation recognition rules provided by the user, after the image prompt segmentation network determines the full-image mask of each content object, the position information of each identification position provided by the user in the segmentation recognition rule can be compared with the window position information of each content object obtained through target detection, and according to certain predetermined rules, it is determined whether the window position information of each content object matches an identification position specified by the user. For example, when the window position information of a content object represents a window that partially or completely contains the range defined by one or more identification positions specified by the user, the content object can be regarded as a content object that matches the identification position and is determined to be a target content object that meets the user's expectations.

[0235] Step S9213: Determine the image content of the target content object and the content area corresponding to the image content in the target image based on the full image mask.

[0236] At this point, the content area of ​​each target content object can be determined in the full image mask based on the label of the target content object. Then, the content area is mapped to the target image, and the image content corresponding to the target content object is extracted from the target image to further determine the main color tone.

[0237] In the above embodiment, the image prompt segmentation model working under the automatic segmentation model first performs target detection and then performs image semantic segmentation to obtain a full-image mask. With the enhancement of the target detection network, the accuracy of content object recognition can be improved. Then, with the help of the recognition position in the segmentation recognition rule provided by the user, combined with the full-image mask, the content area and image content of each target content object that meets the user's expectations are determined in the target image. When the light-emitting unit set in the display frame of the atmosphere light is determined according to the content area, and the main color tone of the image content in the content area is projected onto the light-emitting unit set, the image content and the light-emitting unit set can be accurately corresponded, so that the entire atmosphere light can correspond to the color distribution between each content object and render a corresponding light atmosphere effect.

[0238] Based on the above embodiment, performing image semantic segmentation on the target image according to the segmentation recognition rule may include:

[0239] Step S9221: calling the target image and the segmentation and recognition rules;

[0240] Regarding the call of target images, as described in the previous examples, each target image can be called from a sequence of image frames to play lighting effects, so I won't go into detail here. Regarding the segmentation and recognition rules, their function is to guide the deep learning model to identify each target content object from the target image. Since both the target image and the segmentation and recognition rules are predetermined, they can be directly called.

[0241] Step S9222: Using the image-cued segmentation model in a non-automatic segmentation mode, based on the segmentation recognition rule, determine a full-image mask corresponding to each content object in the target image, wherein the full-image mask represents the content area of ​​the corresponding content object according to the size specification of the target image;

[0242] The image prompt segmentation model of the present application can work in a non-automatic segmentation mode. In this mode, even if the image prompt segmentation model is configured with a target detection network, it will not play a role. Instead, the image semantic segmentation of the target image is processed directly by the image prompt segmentation network in the image prompt segmentation model. Since the image prompt segmentation network needs the help of the segmentation prompt information to complete the image semantic segmentation, the important difference between this embodiment and the previous embodiment is that the segmentation recognition rules are input as segmentation prompt information into the prompt encoder of the image prompt segmentation network, and the target image is input into the image encoder, so that the image prompt segmentation network can directly determine the content area of ​​each target content object corresponding to each recognition position in the segmentation recognition rule, and represent it as a corresponding full-image mask. Since the architecture and principle of the image prompt segmentation network are the same as those of the previous embodiment, it can be a SAM model or its evolved version, which will not be repeated here.

[0243] Step S9223: Determine the image content of the target content object and the content area corresponding to the image content in the target image based on the full image mask.

[0244] Similar to the previous embodiment, after the full image mask is determined, the content area of ​​each target content object can be further determined in the target image based on the full image mask, and the image content in the content area is extracted to determine the main color tone, which will not be repeated here.

[0245] In the above embodiments, the segmentation recognition rules are directly used as segmentation prompt information, and the interactive capabilities of the image prompt segmentation model are used to accurately identify the content objects in the target image. This can further compress the network scale, have efficiency advantages at runtime, and is more suitable for deployment in scenarios where embedded chips are used as the main body of the method of this application. Based on any embodiment of the method of this application, the location information of the recognition position obtained based on the interface canvas is constructed as a segmentation recognition rule, including:

[0246] Step S9110: Acquire layout configuration information of the atmosphere light device, where the layout configuration information is based on a reference coordinate system and describes position information of each light-emitting unit in the reference coordinate system;

[0247] In an ambient light device, the position information of each light-emitting unit in the ambient light is pre-determined through layout configuration information. This position information is usually determined based on the same reference coordinate system. This reference coordinate system can be directly mapped to the coordinate system in the display frame formed by the ambient light. For ease of understanding, it can be understood as the reference coordinate system corresponding to the display frame. Based on this, the layout configuration information of the ambient light device actually defines the position information of each light-emitting unit in the display frame formed by the ambient light based on this reference coordinate system. Of course, this layout configuration information can also be standardized and stored in the controller, and can be directly called and retrieved when needed.

[0248] Step S9120: Determine the display frame of the atmosphere light device according to the layout configuration information, generate an interface canvas corresponding to the display frame, and display it in a graphical user interface;

[0249] After calling the layout configuration information, it is parsed accordingly and then converted into a data form that represents the position information of each light-emitting unit relative to the display frame based on the reference coordinate system to facilitate searching. Since the layout configuration information defines all the light-emitting units in the atmosphere light device, it is actually equivalent to defining the total number of pixels in the display frame of the atmosphere light device. Based on this, according to the preset conversion rules, the corresponding frame ratio can be determined, and then the corresponding display frame can be determined. For example, for an atmosphere light device that adopts the curtain light style, assuming that each light strip has 9 light strips and each light strip has 16 light units, the frame ratio can be determined to be 9:16, thereby constructing a 9:16 display frame. Corresponding to this ratio, an interface canvas is constructed with reference to the actual size of the graphical user interface. In this way, each light-emitting unit can be uniquely mapped to a point on the interface canvas. Specifying a position or area in the interface canvas can also map to one or more light-emitting units.

[0250] Step S9130: receiving at least one identified position specified based on the interface canvas, and determining coordinate information of the identified position relative to the reference coordinate system as position information;

[0251] The user can control the human-computer interaction capabilities opened by the controller and set one or more identification positions in the interface canvas. The program process is responsible for mapping the position information set by the user in the interface canvas to the coordinate information in the reference coordinate system, thereby enabling the user to specify the coordinate information of the position of the content object to be identified by referring to the display frame of the atmosphere light device.

[0252] Step S9140: construct the position information of each of the identified positions into a segmentation recognition rule.

[0253] After the user gives the specified location and determines the corresponding coordinate information, these coordinate information are constructed according to certain rules, usually according to the input parameter format requirements of the image prompt segmentation model responsible for implementing image semantic segmentation, so that it becomes a segmentation recognition rule, which can be used to identify the content area of ​​the content object for the target image.

[0254] As can be seen from the above embodiments, the present application constructs an interface canvas based on the layout configuration information of the atmosphere light device, so that a corresponding mapping relationship is established between the interface canvas and the display frame of the atmosphere light device. Through the interface canvas, the service capability of performing image semantic segmentation of content objects is further opened to the user, so that the user can specify the location where the content object may exist. It is not difficult to understand that by specifying different recognition locations in the interface canvas, the deep learning model may adapt to the different locations and recognize different content objects, and the atmosphere light device may obtain different color distributions accordingly, thereby achieving flexible adjustment of the color distribution, enriching the human-computer interaction function, and comprehensively improving the user experience of the atmosphere light device.

[0255] The present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by a processor, the processor executes the steps of the atmosphere light device color matching method described in any embodiment of the present application.

[0256] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the atmosphere light device color matching method described in any embodiment of the present application.

[0257] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. The aforementioned storage medium can be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0258] The above description is only part of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A color matching method for an atmosphere light device, comprising: Detecting each content object in the target image and obtaining boundary information corresponding to each content object; Determine, according to the boundary information of the content object, a light unit set corresponding to each content object in the atmosphere light device; Determining the image color value of the main hue of each content object according to the image content corresponding to the boundary information of each content object in the target image; According to the image color value of the main hue of each content object, the luminous color value of each luminous unit in the luminous unit set corresponding to the content object is determined.

2. The color matching method of the atmosphere light device according to claim 1, characterized in that: Detect each content object in the target image and obtain the boundary information corresponding to each content object, including: Calling a target image, performing target detection on the target image using a target detection model, and obtaining window position information corresponding to at least one content object; Extracting a region image of a corresponding region from the target image according to the window position information of each content object; Using a default image segmentation model to determine a region mask of a content object of each region image relative to the region image, wherein the region mask includes boundary information of the corresponding content object in the region image; The regional mask of each content object is converted into a corresponding full-image mask, wherein the full-image mask represents the boundary information of the corresponding content object according to the size specification of the target image.

3. The color matching method of the atmosphere light device according to claim 1, characterized in that: Detect each content object in the target image and obtain the boundary information corresponding to each content object, including: Calling a target image and segmentation prompt information, wherein the segmentation prompt information is used to indicate a recognition rule of a content object in the target image; An image hint segmentation model is used to detect a full-image mask corresponding to each content object in the target image based on the recognition rule of the segmentation hint information, and the full-image mask represents the boundary information of the corresponding content object according to the size specification of the target image.

4. The color matching method of the atmosphere light device according to claim 3, characterized in that: Before calling the target image and the segmentation prompt information, the method includes: Determine an interface canvas according to the size specification of the target image, and display the interface canvas on a graphical user interface; receiving at least one identification position specified based on the interface canvas, and determining position information of the identification position relative to the target image; The position information of each of the identified positions is stored as segmentation prompt information, which is used to instruct the image prompt segmentation model to determine the full-image mask corresponding to each content object in the target image.

5. The color matching method of the atmosphere light device according to claim 1, characterized in that: According to the boundary information of each content object, a light unit set corresponding to the content object in the atmosphere light device is determined, including: Acquire layout configuration information of the atmosphere light device, where the layout configuration information is based on a reference coordinate system and describes position information of each light-emitting unit in the reference coordinate system; Determine, according to a mapping relationship between boundary information of each content object and the reference coordinate system, a mapping area corresponding to the boundary information of each content object in the reference coordinate system; According to the mapping area corresponding to each content object, each light-emitting unit whose position information belongs to the mapping area in the atmosphere light device is determined to construct a light-emitting unit set corresponding to the content object.

6. The method for color matching of an atmosphere light device according to any one of claims 1 to 5, characterized in that: Determining the luminous color value of each luminous unit in the luminous unit set corresponding to the content object according to the image color value of the main color tone of each content object includes: Obtaining a lighting effect description template corresponding to the atmosphere light device, wherein the lighting effect description template includes color value attribute items corresponding to each light-emitting unit in the atmosphere light device; Assigning the color value attribute item of each light-emitting unit in the light-emitting unit set corresponding to the content object with the image color value corresponding to the main color tone of each content object, thereby setting the image color value as the light-emitting color value in the color value attribute item; The lighting effect description template that sets the light color value of each light-emitting unit in the atmosphere light device is converted into a lighting effect playing instruction for controlling the various light-emitting units to collaboratively play the lighting effect corresponding to the target image.

7. The method for color matching of an ambient light device according to any one of claims 1 to 5, characterized in that: Before detecting each content object in the target image and obtaining boundary information corresponding to each content object, the method includes: Continuously collect interface images from external terminal devices; Eliminate the black bands on the edges of the currently acquired interface image to obtain an image without black bands; Improving the color value of the target pixel belonging to the dark light pixel in the image without black band to obtain an enhanced image; The enhanced image is added to an image frame sequence, and the enhanced image is sequentially dequeued from the image frame sequence and used as the target image.

8. The method for color matching of an ambient light device according to any one of claims 1 to 5, characterized in that: Detect each content object in the target image and obtain the boundary information corresponding to each content object, including: Image semantic segmentation is performed based on the target image to identify the content area where the content object is located and the image content in the content area.

9. The color matching method of the atmosphere light device according to claim 8, characterized in that: Perform image semantic segmentation based on the target image to identify the content area where the content object is located and the image content in the content area, including: Segmenting the target image into a plurality of region images corresponding to the content objects; Using an image default segmentation model to determine a region mask of a content object of each region image relative to the region image, the region mask representing a content region of a corresponding content object in the region image; The region mask of each content object is converted into a corresponding full-image mask, wherein the full-image mask represents the content region of the corresponding content object according to the size specification of the target image.

10. The color matching method of the atmosphere light device according to claim 8, characterized in that: Perform image semantic segmentation based on the target image to identify the content area where the content object is located and the image content in the content area, including: Calling a target image and segmentation prompt information, wherein the segmentation prompt information includes coordinate information corresponding to a position where a content object may appear in the target image; An image hint segmentation model is used to detect a full-image mask corresponding to each content object in the target image based on the segmentation hint information, and the full-image mask represents a content area of ​​the corresponding content object according to the size specification of the target image.

11. The color matching method of atmosphere light equipment according to claim 1 or 8, characterized in that: The corresponding determination of the image color value of the main color tone of each content object includes: Inputting the image content of each content object into a preset emotion classification model, and inferring and determining the emotion attribute conveyed by the image content; According to the main color tone corresponding to the emotional attribute of each content object, the image color value corresponding to the content object is determined.

12. The method for color matching of an atmosphere light device according to claim 11, characterized in that: After determining the image color value corresponding to each content object according to the main color tone corresponding to the emotional attribute of each content object, the method includes: Inputting the target image into the emotion classification model to determine the emotion attribute conveyed by the target image by inference; The image color values ​​corresponding to the various content objects are blended according to the main color tone corresponding to the emotional attribute of the target image.

13. The color matching method of the atmosphere light device according to claim 12, characterized in that: According to the main color tone corresponding to the emotional attribute of the target image, the image color values ​​corresponding to the respective content objects are adjusted, including: Determine the target color system to which the target image belongs according to the main color tone corresponding to the emotional attribute of the target image; It is detected whether the main color tone corresponding to the emotional attribute of each content object belongs to the target color system, and the main color tone of each content object that does not belong to the target color system is transformed to the target color system with reference to the same reference.

14. The method for color matching of an atmosphere light device according to claim 8, characterized in that: The method comprises: Acquire frame shape information of the atmosphere light device, wherein the frame shape information represents each light-emitting unit in the atmosphere light device as basic pixels orderly arranged on each frame constituting the same frame; Performing image semantic segmentation based on the target image to determine the image content and content area of ​​each content object in the target image; Taking the frame as the outer frame of the target image, allocating the content area of ​​each content object to each border of the frame, so that each light-emitting unit of the frame corresponds to one of the content areas, and obtaining a light-emitting unit set mapped to each content area; The main color tone of each content object is determined according to the image content of each content object, and the light-emitting unit set mapped to each content object is controlled to play the lighting effect according to the main color tone of each content object.

15. The method for color matching of an atmosphere light device according to claim 14, characterized in that: The step of obtaining a set of light-emitting units mapped to each content area includes: Determine the relationship between the position information of each light-emitting unit in the frame and each frame of the frame according to the frame shape information; Calculating the distance from the content area of ​​each content object to each frame of the frame, determining the frame with the shortest distance as the target frame corresponding to the content area, and the projection size when the content area is projected onto the target frame; According to the proportion of the projection sizes of all content areas corresponding to each target frame, the total amount of light-emitting units of the target frame is allocated to each content area in the corresponding all content areas to obtain a light-emitting unit set mapped to each content area.

16. The method for color matching of an atmosphere light device according to claim 8, characterized in that: The method comprises: Acquiring location information of the identification location based on the interface canvas to construct a segmentation identification rule, wherein the interface canvas is used to represent a display frame composed of each light-emitting unit in the atmosphere light device as a basic pixel; Determine a target image from the video stream, perform image semantic segmentation on the target image according to the segmentation and recognition rule, and determine the image content and content area of ​​the content object corresponding to each of the recognition positions in the target image; Determine, according to the mapping relationship between the target image and the display frame, a set of light-emitting units in a corresponding mapping area of ​​each content object in the display frame; According to the main color tone of the image content of each content object, each light-emitting unit in the corresponding light-emitting unit set is controlled to play a corresponding lighting effect.

17. The method for color matching of an atmosphere light device according to claim 16, characterized in that: Performing image semantic segmentation on the target image according to the segmentation recognition rule includes: Using an image prompt segmentation model in an automatic segmentation mode, determining a full-image mask corresponding to each content object in the target image and window position information corresponding to a regional image of each content object in the target image, wherein the full-image mask represents a content region of the corresponding content object according to a size specification of the target image; The image prompt segmentation model matches each position information in the segmentation recognition rule with the window position information of each content object, and selects each content object that achieves the match as the target content object; The image content of the target content object and a content area corresponding to the image content are determined in the target image based on the full image mask.

18. The method for color matching of an atmosphere light device according to claim 16, characterized in that: Performing image semantic segmentation on the target image according to the segmentation recognition rule includes: Calling the target image and the segmentation recognition rule; Using an image prompt segmentation model in a non-automatic segmentation mode, based on the segmentation recognition rule, a full-image mask corresponding to each content object in the target image is determined, wherein the full-image mask represents a content area of ​​the corresponding content object according to the size specification of the target image; The image content of the target content object and a content area corresponding to the image content are determined in the target image based on the full image mask.

19. The method for color matching of an ambient light device according to any one of claims 14 to 16, characterized in that: The position information of the recognition position is obtained based on the interface canvas and constructed into a segmentation recognition rule, including: Acquire layout configuration information of the atmosphere light device, where the layout configuration information is based on a reference coordinate system and describes position information of each light-emitting unit in the reference coordinate system; Determine the display frame of the atmosphere light device according to the layout configuration information, generate an interface canvas corresponding to the display frame, and display it in a graphical user interface; receiving at least one identified position specified based on the interface canvas, and determining coordinate information of the identified position relative to the reference coordinate system as position information; The position information of each of the identified positions is constructed as a segmentation identification rule.

20. An ambient light device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 19.

Images