Lighting effect control method, system, device, electronic device and storage medium

Abstract

The present application discloses a lighting effect control method, system, device, electronic device and storage medium. The method comprises the following steps: obtaining an audio stream in a video stream to be tested, performing frame processing on the audio stream to obtain an audio frame sequence; sequentially obtaining one of the audio frames that has not been traversed in the audio frame sequence as a detection audio frame; performing audio recognition processing on the audio frames in the detection audio frame sequence to obtain a recognition result corresponding to the detection audio frame, and controlling the lighting effect of a lighting device according to the recognition result. By framing the audio stream and performing audio recognition processing in units of audio frames, the calculation is simple and convenient, and the lighting effect of the lighting device is controlled in real time based on the recognition result of the audio frame, thereby enriching the control method of the lighting effect.

Description

Technical Field

[0001] The present application relates to the technical field of lighting equipment control, and more specifically, to a lighting effect control method, system, device, electronic device and storage medium. Background Art

[0002] In the related art, the rhythm points of the audio are usually detected to control the lighting effects according to the changes of the audio rhythm points. However, in order to obtain the rhythm points of the audio, a large number of complex calculations are required, and only the change information such as the amplitude and spectrum of the audio is paid attention to, and the lighting effect control is single. Summary of the invention

[0003] In view of the above problems, the present invention proposes a lighting effect control method, system, device, electronic device and storage medium to improve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a lighting effect control method, the method comprising: obtaining an audio stream in a video stream to be tested; performing frame processing on the audio stream to obtain an audio frame sequence; sequentially obtaining one of the audio frames that has not been traversed in the audio frame sequence as a detection audio frame; performing audio recognition processing on the detection audio frame to obtain a recognition result corresponding to the detection audio frame; controlling the lighting effect of the lighting equipment according to the recognition result, and returning to execute the step of sequentially obtaining one of the audio frames that has not been traversed in the audio frame sequence as the detection audio frame.

[0005] Optionally, performing audio recognition processing on the detected audio frame to obtain a recognition result corresponding to the detected audio frame includes: determining the probability that the audio frame belongs to a preset category; wherein the number of preset categories is multiple; and determining the preset category with the highest probability as the music category corresponding to the audio frame. Controlling the lighting effect of the lighting device according to the recognition result includes: determining the control lighting effect corresponding to the music category, and controlling the lighting device to display the control lighting effect.

[0006] Optionally, controlling the lighting device to display the control lighting effect includes: if the probability of the music category is higher than a preset category threshold, controlling the lighting device to display the control lighting effect.

[0007] Optionally, determining the control lighting effect corresponding to the music category includes: if the music category belongs to other categories, not controlling the lighting device; otherwise, determining the control lighting effect corresponding to the music category.

[0008] Optionally, controlling the lighting device to display the control lighting effect includes: determining a current state of the lighting device; if the current state is a lighting effect display state, not controlling the lighting device; otherwise, controlling the lighting device to display the control lighting effect.

[0009] Optionally, controlling the lighting device to display the control lighting effect includes: controlling the lighting device to repeatedly display the control lighting effect within a preset filtering time; wherein the display time of a single control lighting effect is less than the preset filtering time.

[0010] Optionally, sequentially obtaining one of the audio frames that has not been traversed in the audio frame sequence as a detection audio frame includes: sequentially obtaining one of the audio frames that has not been traversed in the audio frame sequence as a detection audio frame according to a preset sampling interval; wherein the display time of a single control lighting effect is greater than the preset sampling interval.

[0011] Optionally, before obtaining the video stream to be tested, the method further includes: obtaining the original video stream; generating at least two video streams to be tested based on the original video stream; controlling a display device to display based on one of the video streams to be tested; obtaining an audio stream in the video stream to be tested; including: obtaining an audio stream in another of the video streams to be tested.

[0012] Optionally, the method further includes: in response to a control operation on the display device, controlling the display device to enter a display state corresponding to the control operation, and simultaneously controlling the lighting device to enter a lighting effect state corresponding to the control operation.

[0013] Optionally, the lighting device comprises one or more lighting units.

[0014] In a second aspect, the present application also provides a lighting effect control system, including: a control device and a lighting device connected to the control device, wherein the control device is used to execute the lighting effect control method as described in the first aspect above.

[0015] In a third aspect, the embodiment of the present application also provides a lighting effect control device, including: an audio stream acquisition module, an audio frame sequence acquisition module, a detection audio frame determination module, an identification module, and a lighting effect control module. Among them, the audio stream acquisition module is used to obtain the audio stream in the video stream to be tested; the audio frame sequence acquisition module is used to perform frame processing on the audio stream to obtain an audio frame sequence; the detection audio frame determination module is used to sequentially obtain one of the audio frames that has not been traversed in the audio frame sequence as a detection audio frame. The identification module is used to perform audio recognition processing on the detection audio frame to obtain a recognition result corresponding to the detection audio frame; the lighting effect control module is used to control the lighting effect of the lighting equipment according to the recognition result.

[0016] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: one or more processors; a memory; and one or more applications. The one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the lighting effect control method as described in the first aspect above.

[0017] In a fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the lighting effect control method as described in the first aspect above.

[0018] The technical solution provided by the present invention obtains the audio stream in the video stream to be tested, performs frame processing on the audio stream to obtain an audio frame sequence; sequentially obtains one of the audio frames that has not been traversed in the audio frame sequence as a detection audio frame; performs audio recognition processing on the detection audio frame to obtain a recognition result corresponding to the detection audio frame, and controls the lighting effect of the lighting equipment according to the recognition result. By framing the audio stream and performing audio recognition processing in units of audio frames, the calculation is simple and convenient, and the lighting effect of the lighting equipment is controlled in real time based on the recognition result of the audio frame, thereby enriching the control method of the lighting effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments and drawings obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Figure 1 A schematic diagram of an application environment involved in an embodiment of the present application is shown.

[0021] Figure 2 A schematic diagram showing another application environment involved in an embodiment of the present application is shown.

[0022] Figure 3 A flow chart of a lighting effect control method proposed in an embodiment of the present application is shown.

[0023] Figure 4 A schematic diagram of a time domain diagram of an audio stream in an embodiment of the present application is shown.

[0024] Figure 5 A schematic diagram of a process for generating a video stream to be tested proposed in an embodiment of the present application is shown.

[0025] Figure 6 A structural block diagram of a lighting effect control device proposed in an embodiment of the present application is shown.

[0026] Figure 7 A structural block diagram of an electronic device proposed in an embodiment of the present application is shown.

[0027] Figure 8A structural block diagram of a computer-readable storage medium proposed in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0029] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0030] In the related art, the rhythm points of the audio are usually detected to control the lighting effects according to the changes in the audio rhythm points. In order to obtain the rhythm points of the audio, it is necessary to calculate the time domain and frequency domain of the audio at the same time, and combine the time domain and frequency domain calculation results to obtain the rhythm points, which is computationally intensive and complex. In addition, when the audio energy fluctuation is small, the rhythm points cannot be accurately obtained, which affects the lighting effect control. Moreover, the lighting effect control through the rhythm points only focuses on the change information such as the amplitude and spectrum of the audio, and the lighting effect control is single.

[0031] In order to improve the above-mentioned problems, the inventors proposed the lighting effect control method, system, device, electronic device and storage medium provided in the present application. The method obtains the audio stream in the video stream to be tested, performs frame processing on the audio stream to obtain an audio frame sequence; sequentially obtains one of the audio frames that has not been traversed in the audio frame sequence as a detection audio frame; performs audio recognition processing on the detection audio frame to obtain a recognition result corresponding to the detection audio frame, and controls the lighting effect of the lighting equipment according to the recognition result. By framing the audio stream and performing audio recognition processing in units of audio frames, the calculation is simple and convenient, and the lighting effect of the lighting equipment is controlled in real time based on the recognition result of the audio frame, thereby enriching the control method of the lighting effect.

[0032] The following introduces the application environment of the lighting effect control method provided by the embodiment of the present invention.

[0033] See also Figure 1 The lighting effect control method provided by the present invention can be applied to a lighting effect control system A, which includes a control device B and a lighting device C. It should be noted that: Figure 1This is an exemplary application environment. The method provided in the embodiment of the present application can also be run in other application environments without limitation.

[0034] In some implementations, the control device B may be used to obtain the video stream to be tested. The control device B may include a hardware interface for obtaining the video stream to be tested.

[0035] In some implementations, the control device B may be connected to a device such as XBOX (a home video game console), PS (PlayStation, an electronic game console) through a hardware interface to obtain a video stream to be tested.

[0036] In order to make the control device B compatible with video streams to be tested in multiple formats, in an embodiment of the present application, the control device B may also include a video interface conversion module, which can be used to convert the format of the received video stream to be tested, for example, converting the video stream to be tested into I2S (Inter-IC Sound, integrated circuit built-in audio bus), MIPI (Mobile Industry Processor Interface, mobile industry processor interface), LVDS (Low Voltage Differential Signaling, low voltage differential signal), BT656, BT1120 and other formats required by the control device B. The specific settings can be made according to actual use needs, and the present application does not impose any restrictions on this.

[0037] In some embodiments, the control device B may further include a control unit, which may be used to process the video stream to be tested, such as performing frame processing on the audio stream to obtain an audio frame sequence, performing audio recognition processing on the audio frames in the audio frame sequence to obtain a recognition result corresponding to the audio frame, and controlling the lighting effect of the lighting device C according to the recognition result. Exemplarily, the control unit may be a SOC (System on Chip), a CPU (central processing unit), an MPU (Microcontroller Unit), etc., which may be specifically configured according to actual use needs, and the present application does not limit this.

[0038] Optionally, the control unit may be loaded with a pre-trained audio recognition model for performing audio recognition processing on the audio frames.

[0039] Optionally, the control unit may not set the audio recognition model. The control unit may send the audio frame to a server loaded with a pre-trained audio recognition model. The server performs audio recognition processing on the audio frame and sends the recognition result to control device B.

[0040] Among them, the lighting device C can be connected to the control device B by wired or wireless means. When wireless connection is adopted, the lighting device C can communicate with the control device B through the network. The Internet is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, dedicated network or virtual private network. In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML) and the like are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) and the like can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0041] In some embodiments, the lighting device C may be a strip light, a bar light, a lamp post, a flat lamp (containing a plurality of lamp beads), or a lamp arranged in a space.

[0042] In some embodiments, the lighting device C may include a control unit and a lighting unit, and the control unit may be used to control the lighting effect of the lighting unit according to the control device B.

[0043] In some embodiments, the lighting device C may include one or more lighting units, which may be combined accordingly according to actual usage requirements.

[0044] In some embodiments, each lighting unit may include one or more lamp beads, and the control unit may control the brightness, color, etc. of the lamp beads of the lighting unit, and may also control the number of lights on, thereby achieving different lighting effects of the lighting device.

[0045] In some implementations, the position and quantity of the lighting devices can be set according to actual use needs, and the lighting effects in length, height, and area can be controlled according to the position of the lighting devices.

[0046] See also Figure 2In some embodiments, the lighting control system A may further include a display device D, and the display device D is connected to the control device B via a wired or wireless method.

[0047] In some implementations, the display device D may be displayed according to the video stream to be tested, so that while displaying the video, a lighting effect corresponding to the video may be displayed through a lighting device to create an immersive atmosphere and enhance the user experience.

[0048] Optionally, the display device D may be, for example, a smart TV, a tablet computer, a smart speaker with a display screen, a mobile phone, etc., which is not limited in this application.

[0049] In some embodiments, the lighting control system A may further include a video splitting unit for splitting the original video stream into at least two video streams to be tested, so that one of the video streams to be tested can be displayed by the display device D, and the other video stream to be tested can be used by the control device B to control the lighting effect of the lighting device C. For example, the video splitting unit may be a video one-to-two interface chip, a video distributor chip, etc., which is not limited in the present application.

[0050] In the embodiments of the present application, the control device B may also execute the lighting control method involved in the following embodiments, which will be described below in conjunction with specific embodiments.

[0051] See also Figure 3 The present application provides a lighting effect control method in an embodiment, which can be applied to the above lighting effect control system. In the method provided in the embodiment of the present application, the execution subject of the method can be the control device in the above lighting effect control system. Specifically, the lighting effect control method provided in the embodiment of the present application includes: step S110 to step S140.

[0052] In step S110, the audio stream in the video stream to be tested is obtained.

[0053] In an embodiment of the present application, the video stream to be tested includes picture frames for display and audio streams for playback. When the picture frames and audio are played simultaneously, they constitute a sound picture seen by the user. The audio stream in this article includes but is not limited to background music. In order to control the lighting equipment according to the background music corresponding to the video, the control device needs to obtain the audio stream in the video stream to be tested. Specifically, the required audio stream can be obtained by decoding and extracting the video stream to be tested. Exemplarily, the audio stream can be obtained by using FFMpeg (Fast Forward Mpeg, Fast Forward Motion Picture Experts Group), Premiere (video editing tool), etc., and this application does not limit this.

[0054] In step S120, the audio stream is framed to obtain an audio frame sequence.

[0055] In an embodiment of the present application, the control device performs frame processing on the audio data stream to obtain a plurality of audio frames, and the plurality of audio frames constitute an audio frame sequence.

[0056] like Figure 4 As shown, Figure 4 A schematic diagram of a time domain diagram of an audio stream in an embodiment of the present application is shown, wherein the audio stream can be divided into multiple audio frames a by framing the audio stream. In the embodiment of the present application, wlen is the frame length, inc is the frame shift, and the next audio frame can be obtained by translating the frame shift inc, thereby dividing the audio stream into multiple audio frames to form an audio frame sequence.

[0057] Optionally, to avoid missing the content of the audio stream, the length of the frame shift inc can be set to be less than or equal to the frame length of a single audio frame. It is understandable that for an audio stream of the same length, the smaller the length of the frame shift, the more audio frames are obtained, so that the subtle changes of the audio frames can be more sensitively reflected, but the amount of calculation will also increase. The length of the frame shift can be set according to the actual needs of sensitivity and calculation, and this application does not impose any restrictions on this.

[0058] By dividing the audio stream into multiple audio frames and performing audio recognition processing on the audio frames, the calculation is simple and the amount of calculation can be effectively reduced.

[0059] In step S130 , one of the audio frames that has not been traversed in the audio frame sequence is sequentially obtained as a detection audio frame.

[0060] In an embodiment of the present application, the control device may sequentially acquire one of the untraversed audio frames in the audio frame sequence in chronological order as a detection audio frame, so as to control the current lighting effect of the lighting device according to the detection audio frame.

[0061] Exemplarily, if the audio frame sequence includes the first audio frame, the second audio frame, and the third audio frame. The time of the first audio frame is earlier than the time of the second audio frame, and the time of the second audio frame is earlier than the time of the third audio frame. The control device has controlled the lighting effect of the lighting device according to the first audio frame, and the audio frames that have not been traversed in the audio frame sequence are the second audio frame and the third audio frame. Since the time of the second audio frame is earlier than the time of the third audio frame, the second audio frame can be obtained as the detection audio frame.

[0062] In some implementations, one of the audio frames that has not been traversed in the audio frame sequence may be sequentially acquired at a preset sampling interval as the detection audio frame.

[0063] Optionally, the display time of a single control lighting effect is greater than a preset sampling interval, thereby reducing repeated triggering of the lighting device.

[0064] In step S140, audio recognition processing is performed on the detected audio frame to obtain a recognition result corresponding to the detected audio frame.

[0065] In an embodiment of the present application, the control device may perform audio recognition processing on the detected audio frame to obtain a recognition result corresponding to the detected audio frame. In other embodiments, the control device may also send the detected audio frame to a server, which performs audio recognition processing and returns a recognition result corresponding to the detected audio frame.

[0066] In some embodiments, a pre-trained neural network model may be used to perform audio recognition processing on the detection audio frame to obtain a recognition result corresponding to the detection audio frame. The recognition result may be, for example, a category corresponding to the detection audio frame, so that the lighting effect is controlled according to the corresponding category.

[0067] The following will describe step S140 in detail with a specific implementation method. Figure 6 As shown, step S140 may include the following steps.

[0068] (1) Determine the probability of detecting that the audio frame is a preset category.

[0069] There are multiple preset categories.

[0070] In an embodiment of the present application, audio features in the detected audio frame can be extracted by an audio extraction algorithm. The audio transformation algorithm can be, for example, FFT (fast Fourier transform) spectrum transformation, MFCC (Mel-scale Frequency Cepstral Coefficients) feature transformation, etc., which can be set according to actual use needs, and the present application does not limit this.

[0071] In an embodiment of the present application, a pre-trained neural network model may be used to perform audio recognition processing on the extracted audio features to determine the probability that the audio frame belongs to a preset category.

[0072] Exemplarily, the number of preset categories can be 20, and the last layer of the neural network model can use softmax (normalized exponential function) for normalization processing to obtain the probability of detecting the audio frame in each preset category, which can also be called confidence.

[0073] In some embodiments, the detection of audio frames as preset categories can also be determined by using deep learning convolutional classification algorithms such as Resnet (residual neural network), VGG (Visual Geometry Group) convolutional neural network, Mobilenet (lightweight convolutional network), etc., and this application does not impose any restrictions on this.

[0074] Exemplarily, the preset categories may include, but are not limited to: flowing water sound category, explosion sound category, wind sound category, rain sound category, etc., which can be set according to actual usage needs, and this application does not impose any restrictions on this.

[0075] In some implementations, other categories may be set in the preset categories. For example, the preset categories include: the first category, the second category, the third category and other categories. If the recognition result of the audio frame is that it does not belong to any of the first category, the second category and the third category, the category of the audio frame is classified as other categories.

[0076] (2) Determine the preset category with the highest probability as the music category corresponding to the detected audio frame.

[0077] In an embodiment of the present application, after obtaining the probability of the detected audio frame corresponding to each preset category through the above steps, the preset category with the highest probability is used as the music category corresponding to the detected audio frame.

[0078] In step S150, the lighting effect of the lighting device is controlled according to the recognition result, and the process returns to the step of sequentially acquiring one of the untraversed audio frames in the audio frame sequence as the detection audio frame.

[0079] In an embodiment of the present application, the control device controls the lighting effect of the lighting device according to the recognition result of the detected audio frame to present the lighting effect corresponding to the detected audio frame.

[0080] In an embodiment of the present application, the control device can control the lighting effect of the lighting device according to the music category.

[0081] In some embodiments, after determining the music category corresponding to the detected audio frame, further judgment can be made. If the probability of the music category corresponding to the detected audio frame is higher than a preset category threshold, the lighting effect of the lighting device is controlled according to the music category. Otherwise, even if the detected audio frame belongs to the music category, the lighting effect of the lighting device will not be controlled according to the music category, that is, the detected audio frame is invalid and needs to be discarded, so the lighting effect of the lighting device is not controlled.

[0082] It is understandable that the lower the preset category threshold is set, the higher the probability that the audio frame belongs to the corresponding music category, and the more sensitive the triggering control of the lighting device is. Therefore, the specific value of the preset category threshold can be set according to actual use needs, and this application does not limit this.

[0083] In some embodiments, each music category is pre-set with a corresponding control lighting effect. After determining the music category corresponding to the audio frame, the control lighting effect corresponding to the music category can be determined, and then the lighting device can be controlled to display the corresponding control lighting effect, so as to present the corresponding lighting effect according to the category of the audio frame, so as to build an immersive lighting atmosphere and enhance the user experience.

[0084] In some embodiments, if the music category belongs to other categories, the lighting device is not controlled, otherwise, the lighting effect corresponding to the music category is determined. That is, when the audio frame belongs to other categories, the audio frame needs to be discarded, that is, the lighting device is not controlled. Only when the music category does not belong to other categories, subsequent processing is performed to ensure accurate control of the lighting effect of the lighting device.

[0085] In some embodiments, when the lighting effect of the previous detection audio frame has not been fully displayed, if the control lighting effect corresponding to the current detection audio frame has been determined at this time, controlling the lighting device that is displaying the lighting effect according to the lighting effect of the current detection frame will cause the lighting effect of the previous detection audio frame to be incompletely displayed. In order to ensure the effect of the lighting effect display, before performing the lighting effect control, first determine the current state of the lighting device. If the current state of the lighting device is the lighting effect display state, that is, the lighting device is still displaying the lighting effect of the previous detection audio frame, then the lighting device is not controlled. Otherwise, the lighting device is controlled to display the control lighting effect corresponding to the current detection audio frame.

[0086] In some embodiments, after determining the control lighting effect for controlling the lighting device, the lighting device may be controlled to display the control lighting effect once or multiple times. That is, the time required to control a complete single control lighting effect is assumed to be 2 seconds, and the time required to control two complete control lighting effects is 4 seconds. Specifically, the preset number of times the lighting device is required to display the lighting effect may be pre-set. The number of preset times may be set according to actual use needs, and this application does not limit this.

[0087] In some implementations, after determining the control lighting effect for controlling the lighting device, the lighting device may be controlled to repeatedly execute the control lighting effect within a preset filtering time.

[0088] Optionally, the display time of a single control lighting effect is less than the preset filtering time, so that the complete control lighting effect is displayed at least once within the preset filtering time. For example, if the preset filtering time is 6s and the display time of a single control lighting effect is 2s, the corresponding control lighting effect is repeatedly displayed 3 times within the preset filtering time.

[0089] Optionally, the display time of a single control lighting effect can also be greater than the preset filtering time, so that only part of the control lighting effect is displayed within the preset filtering time. The specific settings can be adjusted according to actual use needs, and this application does not limit this.

[0090] In an embodiment of the present application, corresponding lighting effects can be set according to the corresponding music category. Lighting effects include but are not limited to: the position, quantity, brightness, color, flashing frequency, etc. of the lighting equipment that lights up, so as to combine lighting effects that match the music category according to actual needs. For example, when the music category is a blasting sound category, the corresponding lighting effect can be that the current color gradually changes for 1 second, changes to red, and flashes with a flashing cycle of 0.05 seconds. After flashing for 1 second, the color gradually changes from red to the original color for 1 second. It can be arbitrarily matched according to actual use needs, and this application does not limit this.

[0091] In some implementations, before step S110, the lighting effect control method of the embodiment of the present application further includes the following steps.

[0092] (1) Get the original video stream.

[0093] (2) generating at least two video streams to be tested based on the original video stream;

[0094] (3) Sending one of the video streams to be tested to a display device, so that the display device displays the one of the video streams to be tested.

[0095] The lighting effect control method of the embodiment of the present application can also be applied to the scenario of simultaneously performing video display and performing lighting effect control using video stream.

[0096] Specifically, the original video stream can be obtained through a video source, and the video source can be, for example, a TV program, a game screen, an online video, etc.

[0097] Furthermore, the original video stream is used to generate two video streams to be tested. Figure 6As shown, the control device realizes the access of the video stream through the hardware interface, and after obtaining the original video stream, it is divided into two to generate at least two video streams to be tested, one of which is used to control the display device to display specific image content, and the other is used to convert the video stream type that can be received by the multimedia processing chip through the video interface, and then the video stream to be tested is processed by the multimedia processing chip to obtain a video frame image sequence, so as to control the lighting effect of the lighting device to change. Among them, the hardware interface can be, for example, VGA (Video Graphics Array), DP (DisplayPort, display interface), HDMI (High Definition Multimedia Interface, high definition multimedia interface), DVI (Digital Visual Interface, digital video interface), etc.; the video interface can be, for example, MIPI (Mobile Industry Processor Interface, mobile industry processor interface), LVDS (Low-Voltage Differential Signaling, low voltage differential signal transmission interface), etc.; the multimedia processing chip can be, for example, a SoC (System on Chip, system-level chip) chip. Further, the looping out of the original video stream can be realized by calling a video capture card.

[0098] It can be understood that the two video streams to be tested have the same content, one of which is used for lighting effect control, and the other is used to control the display device for display, thereby ensuring the consistency of lighting effect control and video display.

[0099] In some embodiments, the control device can also control the lighting device in combination with the image frame and audio frame in the video stream. For example, if a monster appears in the image frame and the music category corresponding to the audio frame is a horror category, the flashing frequency and brightness change of the horror category lighting effect can be adjusted according to the number and distance of the monsters, thereby presenting a richer lighting effect control.

[0100] In some embodiments, in order to ensure an immersive lighting atmosphere, lighting effect control and video display need to be synchronized. Since the display device and the lighting device are controlled by two video streams to be tested, the display state of the display device may change after the user operates the display device. If the lighting device still performs lighting effect control, the video display and lighting effect control will be out of sync, affecting the user experience.

[0101] In order to solve the synchronization problem, in an embodiment of the present application, the lighting effect control method may further include: when the display device responds to the control operation and enters a display state corresponding to the control operation, controlling the lighting device to enter a lighting effect state corresponding to the control operation.

[0102] Optionally, the control operation may include but is not limited to: a pause operation, a shutdown operation, and a restart operation. The user may send a control instruction to the display device through the terminal device to perform a control operation on the display device.

[0103] Optionally, when the display device receives the control instruction, it can send a lighting effect control instruction corresponding to the control instruction to the control device, so that the control device controls the lighting device synchronously.

[0104] Optionally, when sending a control instruction, the user sends two control instructions at the same time, one control instruction is sent to the display device to control the display device, and the other control instruction is sent to the control device to perform corresponding lighting effect control on the control device.

[0105] In this embodiment, the audio stream in the video stream to be tested is obtained, and the audio stream is framed to obtain an audio frame sequence; one of the audio frames that has not been traversed in the audio frame sequence is sequentially obtained as a detection audio frame; audio recognition processing is performed on the detection audio frame to obtain a recognition result corresponding to the detection audio frame, and the lighting effect of the lighting equipment is controlled according to the recognition result. By framing the audio stream and performing audio recognition processing in units of audio frames, the calculation is simple and convenient, and the lighting effect of the lighting equipment is controlled in real time based on the recognition result of the audio frame, thereby enriching the control method of the lighting effect.

[0106] See also Figure 6 , which shows a lighting effect control device 200 provided by an embodiment of the present invention, the lighting effect control device 200 includes: an audio stream acquisition module 210, an audio frame sequence acquisition module 220, a detection audio frame determination module 230, an identification module 240, and a lighting effect control module 250.

[0107] The audio stream acquisition module 210 is used to acquire the audio stream in the video stream to be tested.

[0108] The audio frame sequence acquisition module 220 is used to perform frame processing on the audio stream to obtain an audio frame sequence.

[0109] The detected audio frame determination module 230 is used to sequentially obtain one of the untraversed audio frames in the audio frame sequence as a detected audio frame.

[0110] The recognition module 240 is used to perform audio recognition processing on the audio frames in the audio frame sequence to obtain recognition results corresponding to the audio frames.

[0111] The lighting effect control module 240 is used to control the lighting effect of the lighting device according to the recognition result.

[0112] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Any processing method described in the method embodiment can be implemented by the corresponding processing module in the device embodiment, and will not be repeated one by one in the device embodiment.

[0113] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or modules, which may be electrical, mechanical or other forms.

[0114] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0115] See also Figure 7 Based on the above-mentioned lighting effect control method, the embodiment of the present application also provides another electronic device 300 including a processor that can execute the above-mentioned lighting effect control method, and the electronic device 300 also includes one or more processors 310, a memory 320, and one or more application programs. Among them, the memory 320 stores a program that can execute the content of the above-mentioned embodiment, and the processor 310 can execute the program stored in the memory.

[0116] Among them, the processor 310 may include one or more cores for processing data and a message matrix unit. The processor 310 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions and processes data of the electronic device 300 by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory. Optionally, the processor 310 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 310 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor, but implemented separately through a communication chip.

[0117] The memory 320 may include a random access memory (RAM) or a read-only memory (ROM). The memory 420 may be used to store instructions, programs, codes, code sets or instruction sets. The memory may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the terminal during use.

[0118] Please refer to Figure 8 , which shows a structural block diagram of a computer-readable storage medium 400 provided in an embodiment of the present application. The computer-readable storage medium 400 stores a program code 410, and the program code 410 can be called by a processor to execute the method described in the above method embodiment.

[0119] The computer readable storage medium 400 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 400 includes a non-transitory computer-readable storage medium. The computer readable storage medium 400 has storage space for program codes that execute any of the method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code can be compressed, for example, in an appropriate form.

[0120] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the lighting effect control method described in the above various optional embodiments.

[0121] In summary, the present application provides a lighting effect control method, system, device, electronic device and storage medium, which obtains the audio stream in the video stream to be tested, performs frame processing on the audio stream to obtain an audio frame sequence; performs audio recognition processing on the audio frames in the audio frame sequence to obtain recognition results corresponding to the audio frames, and controls the lighting effect of the lighting equipment according to the recognition results. By framing the audio stream and performing audio recognition processing in units of audio frames, the calculation is simple and convenient, and the lighting effect of the lighting equipment is controlled in real time based on the recognition results of the audio frames, thereby enriching the control methods of the lighting effects.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A lighting effect control method, characterized in that: The method comprises: Get the audio stream in the video stream to be tested; The audio stream is subjected to frame processing to obtain an audio frame sequence; wherein the next audio frame is obtained by translating the frame shift, thereby dividing the audio stream into multiple audio frames to form an audio frame sequence; the length of the frame shift is less than or equal to the frame length of a single audio frame; Sequentially acquiring one of the audio frames that has not been traversed in the audio frame sequence as a detection audio frame according to a preset sampling interval; wherein the display time of a single control lighting effect is greater than the preset sampling interval; Determining the probability that the detected audio frame belongs to a preset category; wherein the number of the preset categories is multiple; Determine the preset category with the highest probability as the music category corresponding to the detected audio frame; Determine the control lighting effect corresponding to the music category; Determining the control parameter for controlling the lighting effect according to the image frame corresponding to the audio frame in the video stream; The control lighting device is controlled to repeatedly display the control lighting effect according to the control parameters within a preset filtering time, and returns to execute the step of sequentially acquiring one of the audio frames that has not been traversed in the audio frame sequence as a detection audio frame according to a preset sampling interval; wherein the display time of a single control lighting effect is less than the preset filtering time.

2. The method according to claim 1, characterized in that The controlling lighting device to display the controlling lighting effect includes: If the probability of the music category is higher than a preset category threshold, the lighting device is controlled to display a control lighting effect.

3. The method according to claim 1, characterized in that The determining of the control lighting effect corresponding to the music category includes: If the music category belongs to other categories, the lighting device is not controlled; Otherwise, determine the control lighting effect corresponding to the music category.

4. The method according to claim 1, characterized in that: The controlling lighting device to display the controlling lighting effect includes: Determine the current status of lighting equipment; If the current state is a lighting effect display state, the lighting device is not controlled; Otherwise, the lighting device is controlled to display the controlled lighting effect.

5. The method according to any one of claims 1 to 4, characterized in that: Before obtaining the video stream to be tested, the method further includes: Get the original video stream; Generate at least two channels of the video streams to be tested based on the original video stream; Sending one of the video streams to be tested to a display device, so that the display device displays the video stream to be tested; The step of obtaining the audio stream in the video stream to be tested comprises: Obtain an audio stream from another of the video streams to be tested.

6. The method according to claim 5, characterized in that The method further comprises: When the display device enters a display state corresponding to the control operation in response to the control operation, the lighting device is controlled to enter a lighting effect state corresponding to the control operation.

7. The method according to any one of claims 1 to 6, characterized in that: The lighting device comprises one or more lighting units.

8. A lighting effect control system, characterized in that: include: A control device and a lighting device connected to the control device; Wherein, the control device is used to execute the lighting effect control method as described in any one of claims 1 to 7.

9. A lighting effect control device, characterized in that: include: An audio stream acquisition module is used to acquire the audio stream in the video stream to be tested; An audio frame sequence acquisition module, used for performing frame processing on the audio stream to obtain an audio frame sequence; A detection audio frame determination module is used to sequentially obtain one of the audio frames that has not been traversed in the audio frame sequence as a detection audio frame according to a preset sampling interval; wherein the display time of a single control lighting effect is greater than the preset sampling interval; wherein the next audio frame is obtained by translating the frame shift, thereby dividing the audio stream into multiple audio frames to form an audio frame sequence; the length of the frame shift is less than or equal to the frame length of a single audio frame; an identification module, configured to determine the probability that the detected audio frame belongs to a preset category; wherein the number of the preset categories is multiple; determine the preset category with the highest probability as the music category corresponding to the detected audio frame; and determine the control parameter for controlling the lighting effect according to the image frame corresponding to the audio frame in the video stream; A lighting effect control module is used to determine the control lighting effect corresponding to the music category; control the lighting equipment to repeatedly display the control lighting effect according to the control parameters within a preset filtering time, and return to execute the step of sequentially obtaining one of the audio frames that has not been traversed in the audio frame sequence according to a preset sampling interval as a detection audio frame; wherein the single display time of the control lighting effect is less than the preset filtering time.

10. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more applications are configured to execute the lighting effect control method as described in any one of claims 1-7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the lighting effect control method according to any one of claims 1 to 7.

Images