Lyric display method and system

Through a dynamic recognition system combining camera and sensor, the lyrics and lighting effects of KTV users are identified and adjusted, which solves the problem of poor interactive experience caused by the static effects of traditional KTV systems, and enhances the user's sense of participation and immersion.

CN119025704APending Publication Date: 2024-11-26BEIJING THUNDERSTONE TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411143102.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The traditional KTV system is static in lyrics and lighting effects, and cannot be dynamically adjusted according to the user's real-time behavior, resulting in poor user interaction experience and immersion.

Method used

Acquisition of images and microphone arrays or infrared sensors through the camera to obtain auxiliary positioning information, use the target recognition model to identify and locate the singing target, the action recognition algorithm recognizes action categories and intensity, and dynamically adjusts the lyrics and lighting effects in combination with the song effect mapping table.

Benefits of technology

Real-time feedback on user actions is achieved, enhances user's sense of participation and immersion, and makes the KTV experience more diverse and vivid.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025704A_ABST
    Figure CN119025704A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a lyric display method and system, and the method comprises the steps: obtaining an image collected by a camera at a current moment and a target effect mapping table corresponding to a current song; performing target identification on the image by adopting a target identification model to obtain at least one identification target and corresponding position information; positioning a singing target in the at least one identification target according to the auxiliary positioning information; wherein the auxiliary positioning information is acquired by a microphone array or an infrared sensor; performing action recognition on the singing target to obtain an action category of the singing target; obtaining action intensity according to the position information of the singing target at the previous moment and the position information of the singing target at the current moment; and determining a lyric display effect and a light display effect at the current moment according to the action category, the action intensity and the target effect mapping table, and performing corresponding lyric display and light display effect control. And the singing experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of multimedia, and in particular to a lyrics display method and system. Background Art

[0002] With the continuous development of the entertainment industry, KTV has become one of the important ways for people to relax and entertain. However, the lyrics display and lighting effects of traditional KTV systems are mostly static and preset, and cannot be dynamically adjusted according to the real-time behavior of users. This static form of expression limits the user's interactive experience and immersion to a certain extent.

[0003] There is currently no effective solution to the above problems in the prior art. Summary of the invention

[0004] To solve the above problems, the present invention provides a lyric display method and system, which dynamically adjusts the lyric display effects and lighting effects according to the user's actions and singing status in real time, so that each action will trigger corresponding feedback, enhance user participation, and solve the problems of poor user interaction experience and immersion in the prior art.

[0005] To achieve the above-mentioned purpose, the present invention provides a lyric display method, comprising: obtaining an image captured by a camera at the current moment and a target effect mapping table corresponding to the current song; using a target recognition model to perform target recognition on the image to obtain at least one recognized target and corresponding position information; locating a singing target in at least one recognized target according to auxiliary positioning information; wherein the auxiliary positioning information is collected by a microphone array or an infrared sensor; performing action recognition on the singing target to obtain an action category of the singing target; obtaining action intensity according to the position information of the singing target at the previous moment and the position information at the current moment; determining the lyrics display effect and lighting display effect at the current moment according to the action category, action intensity and target effect mapping table, and performing corresponding lyrics display and lighting display effect control.

[0006] Further optionally, locating the singing target in at least one recognition target based on the auxiliary positioning information includes: determining an initial sound source area based on the auxiliary positioning information; converting the coordinates in the initial sound source area to the coordinate system of the camera to obtain a converted sound source area; respectively calculating the distance between the converted sound source area and each recognition target, and taking the recognition target with the smallest distance as the singing target.

[0007] Further optionally, the action intensity is obtained according to the position information of the singing target at the previous moment and the position information at the current moment, including: calculating the key point displacement difference according to the key point coordinates in the position information at the previous moment and the key point coordinates in the position information at the current moment; calculating the key point speed and the key point acceleration according to the time difference between the previous moment and the current moment and the displacement difference; calculating the intensity value according to the key point displacement difference, the key point speed difference and the key point acceleration difference; determining the action intensity of the singing target according to the intensity value and a preset intensity level threshold.

[0008] Further optionally, the performing action recognition on the singing target to obtain the action category of the singing target includes: using a skeleton key point recognition algorithm to perform posture recognition on the singing target to obtain posture feature information; using an optical flow method to perform motion trend recognition on the singing target to obtain motion feature information; performing feature splicing on the posture feature information and the motion feature information to obtain category feature information; inputting the category feature information into a preset classification model to obtain the action category corresponding to the singing target.

[0009] Further optionally, before obtaining the target effect mapping table corresponding to the current song, it includes: defining lyrics display effects and lighting display effects corresponding to different action categories and action intensity combinations to obtain at least one effect mapping table; obtaining the target effect mapping table corresponding to the current song includes: determining the corresponding effect mapping table as the current effect mapping table according to the song type of the current song, and modifying the lyrics display effects and / or lighting display effects in the current effect mapping table according to user modification information to obtain the target effect mapping table.

[0010] On the other hand, the present invention also provides a lyrics display system, including: a data acquisition module, used to obtain the image collected by the camera at the current moment and the target effect mapping table corresponding to the current song; a target recognition module, used to use the target recognition model to perform target recognition on the image, and obtain at least one recognized target and corresponding position information; a target positioning module, used to locate the singing target in at least one recognized target according to the auxiliary positioning information; wherein the auxiliary positioning information is collected by a microphone array or an infrared sensor; a category recognition module, used to perform action recognition on the singing target, and obtain the action category of the singing target; an intensity recognition module, used to obtain the action intensity according to the position information of the singing target at the previous moment and the position information at the current moment; an effect display module, used to determine the lyrics display effect and the lighting display effect at the current moment according to the action category, action intensity and the target effect mapping table, and perform corresponding lyrics display and lighting display effect control.

[0011] Further optionally, the target positioning module includes: an auxiliary positioning submodule, used to determine the initial sound source area based on the auxiliary positioning information; a conversion submodule, used to convert the coordinates in the initial sound source area into the coordinate system of the camera to obtain a converted sound source area; a matching submodule, used to respectively calculate the distance between the converted sound source area and each recognized target, and take the recognized target with the smallest distance as the singing target.

[0012] Further optionally, the intensity identification module includes: a displacement difference calculation submodule, used to calculate the key point displacement difference based on the key point coordinates in the position information of the previous moment and the key point coordinates in the position information of the current moment; a speed calculation submodule, used to calculate the key point speed and key point acceleration based on the time difference between the previous moment and the current moment and the displacement difference; an intensity value calculation submodule, used to calculate the intensity value based on the key point displacement difference, key point speed difference and key point acceleration difference; an action intensity determination submodule, used to determine the action intensity of the singing target based on the intensity value and a preset intensity level threshold.

[0013] Further optionally, the category identification module includes: a posture recognition submodule, which is used to use a skeletal key point recognition algorithm to perform posture recognition on the singing target to obtain posture feature information; a motion trend recognition submodule, which is used to use an optical flow method to perform motion trend recognition on the singing target to obtain motion feature information; a feature splicing submodule, which is used to perform feature splicing on the posture feature information and the motion feature information to obtain category feature information; and an action category recognition submodule, which is used to input the category feature information into a preset classification model to obtain the action category corresponding to the singing target.

[0014] Further optionally, the system also includes: a definition module, used to define lyrics display effects and lighting display effects corresponding to different action categories and action intensity combinations, and obtain at least one effect mapping table; the data acquisition module includes: a selection submodule, used to determine the corresponding effect mapping table as the current effect mapping table according to the song type of the current song, and modify the lyrics display effects and / or lighting display effects in the current effect mapping table according to user modification information to obtain a target effect mapping table.

[0015] The above technical solution has the following beneficial effects: dynamically adjusting the lyrics display effect and lighting effect according to the user's real-time actions and singing status, so that each action of the user will trigger corresponding feedback, enhancing the user's sense of participation and immersion; by capturing and identifying the user's action type, the user can add more visual and auditory elements to the singing process, making the KTV experience more diverse and vivid. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a flow chart of a lyrics display method provided by an embodiment of the present invention;

[0018] Figure 2 is a flow chart of a singing target positioning method provided by an embodiment of the present invention;

[0019] Figure 3 is a flow chart of a method for determining action intensity provided by an embodiment of the present invention;

[0020] Figure 4 is a flow chart of an action category recognition method provided by an embodiment of the present invention;

[0021] Figure 5 is a structural schematic diagram of a lyrics display system provided by an embodiment of the present invention;

[0022] Figure 6 is a schematic diagram of the structure of a target positioning module provided by an embodiment of the present invention;

[0023] Figure 7 is a schematic diagram of the structure of a strength recognition module provided by an embodiment of the present invention;

[0024] Figure 8 It is a structural diagram of a category identification module provided in an embodiment of the present invention.

[0025] Figure markings: 100-data acquisition module; 1001-selection submodule; 200-target recognition module; 300-target positioning module; 3001-auxiliary positioning submodule; 3002-conversion submodule; 3003-matching submodule; 400-category recognition module; 4001-posture recognition submodule; 4002-motion trend recognition submodule; 4003-feature splicing submodule; 4004-action category recognition submodule; 500-intensity recognition module; 5001-displacement difference calculation submodule; 5002-speed calculation submodule; 5003-intensity value calculation submodule; 5004-action intensity determination submodule; 600-effect display module; 700-definition module. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] In order to solve the problem of low user immersion during KTV singing in the prior art, an embodiment of the present invention provides a lyrics display method. Figure 1 is a flowchart of a method for displaying lyrics provided by an embodiment of the present invention. Figure 1 As shown, the method includes:

[0028] S1. Obtain the image captured by the camera at the current moment and the target effect mapping table corresponding to the current song.

[0029] A camera is installed in the KTV box to capture the video of the user singing. The camera can be an ordinary webcam or a depth camera (such as Intel RealSense, Microsoft Kinect, etc.) to obtain richer depth information.

[0030] Use a camera to continuously capture the movement of the target and obtain a series of image frames. These frames are recorded at a fixed time interval (such as 30 frames per second, that is, the frame rate is 30FPS), and the frame images collected at each moment need to be processed.

[0031] The captured video stream is processed by the image preprocessing module, including image noise reduction, background subtraction, and key area extraction, to ensure the image quality for subsequent use.

[0032] Image denoising: Remove noise (such as random brightness changes, color spots, etc.) from the image through filtering algorithms, thereby improving the image quality and the accuracy of subsequent processing.

[0033] Background subtraction: remove irrelevant or static background from the image, for example, pre-acquire a background frame without an object, and then differentiate each frame in the video stream from the Beijing frame to obtain an image with the background removed.

[0034] Key Region Extraction: Identify and extract task-specific regions of interest (ROI) from images for further analysis and processing.

[0035] The target effect mapping table is pre-defined, which includes lyrics display effects and lighting display effects corresponding to combinations of different action categories and action intensities.

[0036] S2. Use a target recognition model to perform target recognition on the image to obtain at least one recognized target and corresponding position information.

[0037] In a KTV scene, there may be multiple users at the same time. The target recognition model (such as YOLO, SSD, etc.) can be used to identify all human targets in the KTV box (i.e., identify the target). Of course, the target recognition model can also be obtained by collecting corresponding training sets according to specific circumstances and training through neural networks.

[0038] When all the recognition targets in the box are recognized, the location information corresponding to each recognition target is also obtained at the same time. Specifically, the image captured by the camera has a camera coordinate system, and each recognition target has a square area in the camera coordinate system, that is, the location of the recognition target is framed by a square target frame. The square target frame has four vertices, each vertex corresponds to a coordinate, and the four coordinates together constitute the location information of the recognition target.

[0039] S3. Locate the singing target in at least one recognition target according to the auxiliary positioning information; wherein the auxiliary positioning information is collected by a microphone array or an infrared sensor.

[0040] A microphone array is installed in the KTV box, which can determine the user's location by analyzing the time difference of arrival (TDOA) or angle of arrival (DOA) of the sound when the user sings.

[0041] Arrange multiple microphones (at least 3 or more) in the space and make sure the position of the microphones is known. The microphones can be arranged in a linear array, a two-dimensional planar array, or a three-dimensional stereo array, depending on the application scenario. Each microphone receives the sound signal from the sound source, calculates the time difference or arrival angle when the microphone receives the same sound wave signal, and calculates the position coordinates of the sound source in space through triangulation.

[0042] In addition, multiple infrared transmitters can be installed on the ceiling, wall or other locations of the KTV box, which will periodically emit infrared light to cover various areas in the room. The number and location of the transmitters should be optimized according to the size and layout of the room to ensure that the infrared signal can cover all areas where users are active.

[0043] The user needs to wear a device with an infrared reflector or infrared receiver, which can be a simple bracelet, a head-mounted device, or a reflector mounted on a microphone. This device is used to reflect or receive infrared signals so that the system can capture the user's location.

[0044] Based on the signal transmission between multiple transmitters and receivers, the system can calculate the distance of the target through triangulation. Specifically, the system will calculate the distance between the target and each receiver based on the signal strength or arrival time difference received by different receivers. Combined with the known receiver position, the location coordinates of the sound source can be further calculated.

[0045] Based on the auxiliary positioning information collected by the above microphone array or infrared sensor, that is, the position coordinates of the sound source, the singing target can be located among multiple recognition targets.

[0046] S4. Perform action recognition on the singing target to obtain the action category of the singing target.

[0047] Capture the action feature information of the singing target, for example, the user's skeleton key point information and / or movement speed and other parameters can be captured. The trained model can be used to perform action recognition on the singing target, and the above key point information and / or movement speed information can be used as model input to output the action category of the singing target at the current moment. For example, the action category can be waving, jumping, clapping, shaking, etc.

[0048] S5. Obtain action intensity according to the position information of the singing target at the previous moment and the position information of the singing target at the current moment.

[0049] The target tracking algorithm is used to track the singing target, and its action intensity can be measured according to the position change of the singing target in two adjacent frames.

[0050] S6. Determine the lyrics display effect and the light display effect at the current moment according to the action category, action intensity and target effect mapping table, and perform corresponding lyrics display and light display effect control.

[0051] An action type and an action intensity form an action combination, each action combination corresponds to a lyrics display effect and a light display effect, and the target mapping table contains lyrics display effects and light display effects corresponding to multiple action combinations. Table 1 is a target effect mapping table provided in this embodiment.

[0052] According to the target effect mapping table, the lyrics display effect and lighting display effect corresponding to the current action type and action intensity of the singing target can be determined, and displayed accordingly to render the user's current singing atmosphere.

[0053] Action Type Action intensity Lyrics display effect Lighting display effects Wave Mild The current singing text is enlarged 1.2 times The light color changes to blue Wave Moderate The current singing text is enlarged 1.5 times The light color changes to green Wave severe The current singing text is magnified 2 times The light color changes to red jump Mild The lyrics scrolling speed is increased by 1.2 times Lights flash faster jump Moderate The lyrics scrolling speed is increased by 1.5 times The light flashes faster jump severe Lyrics scrolling speed increased by 2 times Light color changes randomly

[0054] Table 1

[0055] Furthermore, the action category recognition, action intensity calculation and effect triggering of the singing target can be performed in different threads to avoid blocking the main thread and ensure a smooth user experience.

[0056] As an optional implementation, Figure 2 is a flow chart of a singing target positioning method provided by an embodiment of the present invention, such as Figure 2 As shown, locating the singing target in at least one recognition target according to the auxiliary positioning information includes:

[0057] S301. Determine an initial sound source area according to auxiliary positioning information.

[0058] When installing microphone arrays or infrared sensors, these devices need to be calibrated to ensure that their data is consistent in the same coordinate system. Before performing motion recognition, the conversion relationship between different coordinate systems needs to be determined. For example, the relationship between the two coordinate systems can be determined by the origin positions of the two coordinate systems, and the rotation matrix and translation matrix between the two can be calculated.

[0059] Use a microphone array or infrared sensor to determine the user's initial sound source area. Ideally, the sound source position obtained by the microphone array or infrared sensor is an accurate coordinate. However, in actual use, due to noise interference, the sound source position is usually an area, namely the initial sound source area. There are multiple vertices in the initial sound source area. The coordinates of the vertices can be used together as sound source position information to describe the initial sound source area.

[0060] S302: Convert the coordinates in the initial sound source area to the coordinate system where the camera is located to obtain a converted sound source area.

[0061] After that, the coordinates of the initial sound source area need to be converted to the coordinates in the camera coordinate system to obtain the converted sound source area. Taking the microphone array as an example, if the origin of the microphone array is not at the origin of the camera coordinate system, the regional vertices of the initial sound source area are converted to the camera coordinate system. For each point, it is necessary to perform translation, rotation and other transformations according to the relative positions of the two coordinate systems, as shown in the following formula:

[0062] P global =R·P mic +T

[0063] Among them, P mic is the coordinate in the microphone array, R is the rotation matrix, T is the translation vector, P global are coordinates in the global coordinate system.

[0064] S303, respectively calculating the distance between the converted sound source area and each recognition target, and taking the recognition target with the smallest distance as the singing target.

[0065] If multiple recognition targets are recognized in one frame of image during the aforementioned target recognition process, matching is required to determine the singing target among the recognition targets. Specifically, the Euclidean distance between the center coordinates of the converted sound source area and the center coordinates of each recognition target is calculated, and the recognition target with the smallest Euclidean distance is selected as the singing target. Furthermore, a distance threshold can be set to filter out recognition targets with a Euclidean distance greater than the distance threshold, and the recognition target corresponding to the smallest distance is selected from the remaining recognition targets as the singing target to avoid misidentification.

[0066] As an optional implementation, Figure 3 is a flow chart of a method for determining action intensity provided by an embodiment of the present invention. Figure 3 As shown, the action intensity is obtained according to the position information of the singing target at the previous moment and the position information of the singing target at the current moment, including:

[0067] S501, calculating the key point displacement difference according to the key point coordinates in the position information at the previous moment and the key point coordinates in the position information at the current moment.

[0068] Analyze the distance changes and / or angle changes between the user's key points. For example, the displacement difference of the wrist between two frames, the angle difference between the shoulders, etc., which can reflect the amplitude of the action. Quantify these position changes to obtain the key point displacement difference of each key point, such as calculating the wrist movement distance, shoulder movement distance, etc.

[0069] S502, calculating the key point velocity and key point acceleration according to the time difference and displacement difference between the previous moment and the current moment.

[0070] The velocity vector of each key point is calculated by the displacement change of the key points between image frames, the magnitude of the velocity of each key point is calculated, and the average value of the velocity of each key point or the velocity value of a specific key point is taken as the overall velocity feature.

[0071] After obtaining the velocity, the acceleration is calculated by the velocity change between two frames. Similarly, the magnitude of the acceleration of each key point is calculated, and the average value or the acceleration of a specific key point is taken as the overall acceleration feature.

[0072] S503, calculating the strength value according to the key point displacement difference, the key point velocity difference and the key point acceleration difference.

[0073] The key point displacement difference and key point velocity difference, i.e., key point acceleration difference corresponding to all key points are weightedly summed to obtain the strength value.

[0074] As an optional implementation, the calculation formula of the intensity value is as follows:

[0075]

[0076] Where S(t) is the intensity value, N is the number of key points, Δp i (t) is the displacement difference of the key point, Δv i (t) is the velocity difference of the key point, Δa i (t) is the acceleration difference of the key point, and α, β, and γ are weight coefficients.

[0077] S504: Determine the action intensity of the singing target according to the intensity value and the preset intensity level threshold.

[0078] Preset thresholds corresponding to different intensity levels, for example, mild corresponds to the first threshold, moderate corresponds to the second threshold, and severe corresponds to the third threshold. If the intensity value is greater than 0 and not greater than the first threshold, the action intensity of the singing user is considered to be mild. If the intensity value is greater than the first threshold but not greater than the second threshold, the action intensity of the singing user is considered to be moderate. If the intensity value is greater than the third threshold, the action intensity of the singing user is considered to be severe. Of course, the action intensity can be further refined, and the above is only an example of this embodiment.

[0079] As an optional implementation, Figure 4 is a flow chart of the action category recognition method provided by an embodiment of the present invention. Figure 4 As shown, the action recognition of the singing target is performed to obtain the action category of the singing target, including:

[0080] S401, using a skeleton key point recognition algorithm to perform posture recognition on a singing target to obtain posture feature information.

[0081] Skeletal key point based recognition (such as OpenPose, MediaPipe Pose, etc.) recognizes human posture by detecting the positions of key parts of the human body (such as head, shoulders, elbows, wrists, hips, knees, ankles, etc.). These skeletal key points can be used to describe the skeletal structure of the human body and thus determine the posture of the human body.

[0082] After using skeleton key points for identification, the coordinates of multiple skeleton key points, namely posture feature information, can be obtained.

[0083] The skeleton key points here and the above key points can be identified by the skeleton key point algorithm, but in specific use, the skeleton key points used can be the same as or different from the above key points. Usually, the key points and the skeleton key points have intersections, but are not exactly the same. For example, the key points can only use the head, elbow and ankle, while the skeleton key points can use the head, shoulder, elbow, wrist, hip, knee and ankle.

[0084] S402: Use the optical flow method to identify the motion trend of the singing target and obtain motion feature information.

[0085] Use an optical flow algorithm (such as the Lucas-Kanade optical flow method) to analyze the pixel motion between consecutive frames and generate an optical flow vector field. The optical flow vector field is processed to extract features that represent the overall motion trend. For example, information such as the direction, speed, and acceleration of the motion, i.e., motion feature information.

[0086] S403: Concatenate the posture feature information and the motion feature information to obtain category feature information.

[0087] The coordinates of each skeleton key point, the distance between the skeleton key points, the angle and other posture feature information are combined into a feature vector f "#$%oi&' .

[0088] The average motion direction, speed, acceleration and other motion feature information in the optical flow vector field are combined into a feature vector f o%'ical(lo) .

[0089] The skeleton key point feature vector f "#$%oi&' and the optical flow feature vector f o%'ical(lo) Splice them together, connect them front and back to form a longer feature vector, and merge the two vectors into a longer vector, that is, the category feature information. Or the skeleton key point feature vector f "#$%oi&' and the optical flow feature vector f o%'ical(lo) After weighting, the category feature information is obtained.

[0090] S404: Input the category feature information into a preset classification model to obtain the action category corresponding to the singing target.

[0091] The fused feature vector is used as input in advance to train the action recognition model. A deep learning model (such as CNN+LSTM, 3D-CNN, etc.) or a traditional machine learning model (such as SVM, random forest, etc.) is used for classification training to obtain a preset classification model.

[0092] After the category feature information is obtained, it is input into the preset classification model to infer the user's action category.

[0093] As an optional implementation, before obtaining the target effect mapping table corresponding to the current song, the following steps are included:

[0094] S7, defining lyrics display effects and lighting display effects corresponding to different action categories and action intensity combinations, and obtaining at least one effect mapping table;

[0095] Get the target effect mapping table corresponding to the current song, including:

[0096] S101. Determine a corresponding effect mapping table as a current effect mapping table according to the song type of the current song, and modify the lyrics display effect and / or the light display effect in the current effect mapping table according to user modification information to obtain a target effect mapping table.

[0097] A variety of effect mapping table schemes are set in advance, and when facing different types of songs, the corresponding current effect mapping table can be selected in real time. A modification panel can also be provided to the user to modify the current effect mapping table according to the modification information provided by the user to obtain the target effect mapping table.

[0098] The embodiment of the present invention also provides a lyrics display system. Figure 5 is a structural diagram of a lyrics display system provided by an embodiment of the present invention, such as Figure 5 As shown, the system includes:

[0099] The data acquisition module 100 is used to obtain the image captured by the camera at the current moment and the target effect mapping table corresponding to the current song.

[0100] A camera is installed in the KTV box to capture the video of the user singing. The camera can be an ordinary webcam or a depth camera (such as Intel RealSense, Microsoft Kinect, etc.) to obtain richer depth information.

[0101] Use a camera to continuously capture the movement of the target and obtain a series of image frames. These frames are recorded at a fixed time interval (such as 30 frames per second, that is, the frame rate is 30FPS), and the frame images collected at each moment need to be processed.

[0102] The captured video stream is processed by the image preprocessing module, including image noise reduction, background subtraction, and key area extraction, to ensure the image quality for subsequent use.

[0103] Image denoising: Remove noise (such as random brightness changes, color spots, etc.) from the image through filtering algorithms, thereby improving the image quality and the accuracy of subsequent processing.

[0104] Background subtraction: remove irrelevant or static background from the image, for example, pre-acquire a background frame without an object, and then differentiate each frame in the video stream from the Beijing frame to obtain an image with the background removed.

[0105] Key Region Extraction: Identify and extract task-specific regions of interest (ROI) from images for further analysis and processing.

[0106] The target effect mapping table is pre-defined, which includes lyrics display effects and lighting display effects corresponding to combinations of different action categories and action intensities.

[0107] The target recognition module 200 is used to perform target recognition on the image using a target recognition model to obtain at least one recognized target and corresponding position information.

[0108] In a KTV scene, there may be multiple users at the same time. The target recognition model (such as YOLO, SSD, etc.) can be used to identify all human targets in the KTV box, that is, to identify the target. Of course, the target recognition model can also be obtained by collecting corresponding training sets according to specific circumstances and training through neural networks.

[0109] When all the recognition targets in the box are recognized, the location information corresponding to each recognition target is also obtained at the same time. Specifically, the image captured by the camera has a camera coordinate system, and each recognition target has a square area in the camera coordinate system, that is, the location of the recognition target is framed by a square target frame. The square target frame has four vertices, each vertex corresponds to a coordinate, and the four coordinates together constitute the location information of the recognition target.

[0110] The target positioning module 300 is used to locate the singing target in at least one recognition target according to the auxiliary positioning information; wherein the auxiliary positioning information is collected by a microphone array or an infrared sensor.

[0111] A microphone array is installed in the KTV box, which can determine the user's location by analyzing the time difference of arrival (TDOA) or angle of arrival (DOA) of the sound when the user sings.

[0112] Arrange multiple microphones (at least 3 or more) in the space and make sure the position of the microphones is known. The microphones can be arranged in a linear array, a two-dimensional planar array, or a three-dimensional stereo array, depending on the application scenario. Each microphone receives the sound signal from the sound source, calculates the time difference or arrival angle when the microphone receives the same sound wave signal, and calculates the position coordinates of the sound source in space through triangulation.

[0113] In addition, multiple infrared transmitters can be installed on the ceiling, wall or other locations of the KTV box, which will periodically emit infrared light to cover various areas in the room. The number and location of the transmitters should be optimized according to the size and layout of the room to ensure that the infrared signal can cover all areas where users are active.

[0114] The user needs to wear a device with an infrared reflector or infrared receiver, which can be a simple bracelet, a head-mounted device, or a reflector mounted on a microphone. This device is used to reflect or receive infrared signals so that the system can capture the user's location.

[0115] Based on the signal transmission between multiple transmitters and receivers, the system can calculate the distance of the target through triangulation. Specifically, the system will calculate the distance between the target and each receiver based on the signal strength or arrival time difference received by different receivers. Combined with the known receiver position, the location coordinates of the sound source can be further calculated.

[0116] Based on the auxiliary positioning information collected by the above microphone array or infrared sensor, that is, the position coordinates of the sound source, the singing target can be located among multiple recognition targets.

[0117] The category recognition module 400 is used to perform action recognition on the singing target and obtain the action category of the singing target.

[0118] Capture the action feature information of the singing target, for example, the user's skeleton key point information and / or movement speed and other parameters can be captured. The trained model can be used to perform action recognition on the singing target, and the above key point information and / or movement speed information are used as model input to output the action category of the singing target at the current moment. For example, the action category can be waving, jumping, clapping, shaking, etc.

[0119] The intensity recognition module 500 is used to obtain the action intensity according to the position information of the singing target at the previous moment and the position information of the singing target at the current moment;

[0120] The target tracking algorithm is used to track the singing target, and its action intensity can be measured according to the position change of the singing target in two adjacent frames.

[0121] The effect display module 600 is used to determine the lyrics display effect and the light display effect at the current moment according to the action category, action intensity and the target effect mapping table, and to perform corresponding lyrics display and light display effect control.

[0122] An action type and an action intensity form an action combination, each action combination corresponds to a lyrics display effect and a light display effect, and the target mapping table contains lyrics display effects and light display effects corresponding to multiple action combinations. Table 1 is a target effect mapping table provided in this embodiment.

[0123] According to the target effect mapping table, the lyrics display effect and lighting display effect corresponding to the current action type and action intensity of the singing target can be determined, and displayed accordingly to render the user's current singing atmosphere.

[0124] Furthermore, the action category recognition, action intensity calculation and effect triggering of the singing target can be performed in different threads to avoid blocking the main thread and ensure a smooth user experience.

[0125] As an optional implementation, Figure 6is a schematic diagram of the structure of the target positioning module provided by an embodiment of the present invention, such as Figure 6 As shown, the target positioning module 300 includes:

[0126] The auxiliary positioning submodule 3001 is used to determine the initial sound source area according to the auxiliary positioning information.

[0127] When installing microphone arrays or infrared sensors, these devices need to be calibrated to ensure that their data is consistent in the same coordinate system. Before performing motion recognition, the conversion relationship between different coordinate systems needs to be determined. For example, the relationship between the two coordinate systems can be determined by the origin positions of the two coordinate systems, and the rotation matrix and translation matrix between the two can be calculated.

[0128] Use a microphone array or infrared sensor to determine the user's initial sound source area. Ideally, the sound source position obtained by the microphone array or infrared sensor is an accurate coordinate, but in actual use, due to noise interference, the sound source position is usually an area, namely the initial sound source area. The initial sound source area has multiple vertices, and the coordinates of the vertices can be used together as the sound source position information to describe the initial sound source area.

[0129] The conversion submodule 3002 is used to convert the coordinates in the initial sound source area into the coordinate system where the camera is located to obtain a converted sound source area.

[0130] After that, the coordinates of the initial sound source area need to be converted to the coordinates in the camera coordinate system to obtain the converted sound source area. Taking the microphone array as an example, if the origin of the microphone array is not at the origin of the camera coordinate system, the regional vertices of the initial sound source area are converted to the camera coordinate system. For each point, it is necessary to perform translation, rotation and other transformations according to the relative positions of the two coordinate systems, as shown in the following formula:

[0131] P global =R·P mic +T

[0132] Among them, P mic is the coordinate in the microphone array, R is the rotation matrix, T is the translation vector, P global are coordinates in the global coordinate system.

[0133] The matching submodule 3003 is used to calculate the distance between the converted sound source area and each recognition target respectively, and take the recognition target with the smallest distance as the singing target.

[0134] If multiple recognition targets are recognized in one frame of image during the aforementioned target recognition process, matching is required to determine the singing target among the recognition targets. Specifically, the Euclidean distance between the center coordinates of the converted sound source area and the center coordinates of each recognition target is calculated, and the recognition target with the smallest Euclidean distance is selected as the singing target. Furthermore, a distance threshold can be set to filter out recognition targets with a Euclidean distance greater than the distance threshold, and the recognition target corresponding to the smallest distance is selected from the remaining recognition targets as the singing target to avoid misidentification.

[0135] As an optional implementation, Figure 7 is a schematic diagram of the structure of the strength recognition module provided by an embodiment of the present invention. Figure 7 As shown, the strength recognition module 500 includes:

[0136] The displacement difference calculation submodule 5001 is used to calculate the displacement difference of the key point according to the coordinates of the key point in the position information at the previous moment and the coordinates of the key point in the position information at the current moment.

[0137] Analyze the distance changes and / or angle changes between the user's key points. For example, the displacement difference of the wrist between two frames, the angle difference between the shoulders, etc., which can reflect the amplitude of the action. Quantify these position changes to obtain the key point displacement difference of each key point, such as calculating the wrist movement distance, shoulder movement distance, etc.

[0138] The speed calculation submodule 5002 is used to calculate the key point speed and key point acceleration according to the time difference and displacement difference between the previous moment and the current moment.

[0139] The velocity vector of each key point is calculated by the displacement change of the key points between image frames, the magnitude of the velocity of each key point is calculated, and the average value of the velocity of each key point or the velocity value of a specific key point is taken as the overall velocity feature.

[0140] After obtaining the velocity, the acceleration is calculated by the velocity change between two frames. Similarly, the magnitude of the acceleration of each key point is calculated, and the average value or the acceleration of a specific key point is taken as the overall acceleration feature.

[0141] The strength value calculation submodule 5003 is used to calculate the strength value according to the key point displacement difference, the key point velocity difference and the key point acceleration difference.

[0142] The key point displacement difference and key point velocity difference, i.e., key point acceleration difference corresponding to all key points are weightedly summed to obtain the strength value.

[0143] As an optional implementation, the calculation formula of the intensity value is as follows:

[0144]

[0145] Where S(t) is the intensity value, N is the number of key points, Δp i (t) is the displacement difference of the key point, Δv i (t) is the velocity difference of the key point, Δa i (t) is the acceleration difference of the key point, and α, β, and γ are weight coefficients.

[0146] The action intensity determination submodule 5004 is used to determine the action intensity of the singing target according to the intensity value and the preset intensity level threshold.

[0147] Preset thresholds corresponding to different intensity levels, for example, mild corresponds to the first threshold, moderate corresponds to the second threshold, and severe corresponds to the third threshold. If the intensity value is greater than 0 and not greater than the first threshold, the action intensity of the singing user is considered to be mild. If the intensity value is greater than the first threshold but not greater than the second threshold, the action intensity of the singing user is considered to be moderate. If the intensity value is greater than the third threshold, the action intensity of the singing user is considered to be severe. Of course, the action intensity can be further refined, and the above is only an example of this embodiment.

[0148] As an optional implementation, Figure 8 is a schematic diagram of the structure of a category identification module provided by an embodiment of the present invention. Figure 8 As shown, the category identification module 400 includes:

[0149] The posture recognition submodule 4001 is used to perform posture recognition on the singing target by using a skeleton key point recognition algorithm to obtain posture feature information.

[0150] Skeletal key point based recognition (such as OpenPose, MediaPipe Pose, etc.) recognizes human posture by detecting the positions of key parts of the human body (such as head, shoulders, elbows, wrists, hips, knees, ankles, etc.). These skeletal key points can be used to describe the skeletal structure of the human body and thus determine the posture of the human body.

[0151] After using skeleton key points for identification, the coordinates of multiple skeleton key points, namely posture feature information, can be obtained.

[0152] The skeleton key points here and the above key points can be identified by the skeleton key point algorithm, but in specific use, the skeleton key points used can be the same as or different from the above key points. Usually, the key points and the skeleton key points have intersections, but are not exactly the same. For example, the key points can only use the head, elbow and ankle, while the skeleton key points can use the head, shoulder, elbow, wrist, hip, knee and ankle.

[0153] The motion trend recognition submodule 4002 is used to recognize the motion trend of the singing target by using the optical flow method to obtain motion feature information.

[0154] Use an optical flow algorithm (such as the Lucas-Kanade optical flow method) to analyze the pixel motion between consecutive frames and generate an optical flow vector field. The optical flow vector field is processed to extract features that represent the overall motion trend. For example, information such as the direction, speed, and acceleration of the motion, i.e., motion feature information.

[0155] The feature splicing submodule 4003 is used to perform feature splicing on the posture feature information and the motion feature information to obtain category feature information.

[0156] The coordinates of each skeleton key point, the distance between the skeleton key points, the angle and other posture feature information are combined into a feature vector f "#$%oi&' .

[0157] The average motion direction, speed, acceleration and other motion feature information in the optical flow vector field are combined into a feature vector f o%'ical(lo) .

[0158] The skeleton key point feature vector f "#$%oi&' and the optical flow feature vector f o%'ical(lo) Splice them together, connect them front and back to form a longer feature vector, and merge the two vectors into a longer vector, that is, the category feature information. Or the skeleton key point feature vector f "#$%oi&' and the optical flow feature vector f o%'ical(lo) After weighting, the category feature information is obtained.

[0159] The action category identification submodule 4004 is used to input the category feature information into a preset classification model to obtain the action category corresponding to the singing target.

[0160] The fused feature vector is used as input in advance to train the action recognition model. A deep learning model (such as CNN+LSTM, 3D-CNN, etc.) or a traditional machine learning model (such as SVM, random forest, etc.) is used for classification training to obtain a preset classification model.

[0161] As an optional implementation, the system further includes:

[0162] A definition module 700, for defining lyrics display effects and lighting display effects corresponding to different action categories and action intensity combinations, and obtaining at least one effect mapping table;

[0163] The data acquisition module includes:

[0164] Selection module 1001 is used to determine the corresponding effect mapping table as the current effect mapping table according to the song type of the current song, and modify the lyrics display effect and / or light display effect in the current effect mapping table according to user modification information to obtain a target effect mapping table.

[0165] A variety of effect mapping table schemes are set in advance, and when facing different types of songs, the corresponding current effect mapping table can be selected in real time. A modification panel can also be provided to the user to modify the current effect mapping table according to the modification information provided by the user to obtain the target effect mapping table.

[0166] The above technical solution has the following beneficial effects: dynamically adjusting the lyrics display effect and lighting effect according to the user's real-time actions and singing status, so that each action of the user will trigger corresponding feedback, enhancing the user's sense of participation and immersion; by capturing and identifying the user's action type, the user can add more visual and auditory elements to the singing process, making the KTV experience more diverse and vivid.

[0167] The specific implementation methods of the above invention further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above content is only the specific implementation methods of the present invention and is not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A lyrics display method, characterized in that: include: Get the image captured by the camera at the current moment and the target effect mapping table corresponding to the current song; Using a target recognition model to perform target recognition on the image to obtain at least one recognized target and corresponding position information; Locating the singing target in at least one recognition target according to the auxiliary positioning information; wherein the auxiliary positioning information is collected by a microphone array or an infrared sensor; Performing action recognition on the singing target to obtain the action category of the singing target; Obtaining the action intensity according to the position information of the singing target at the previous moment and the position information of the singing target at the current moment; The lyrics display effect and the light display effect at the current moment are determined according to the action category, action intensity and target effect mapping table, and the corresponding lyrics display and light display effect control is performed.

2. The method for displaying lyrics according to claim 1, characterized in that: The step of locating the singing target in at least one recognition target according to the auxiliary positioning information comprises: Determine an initial sound source area according to the auxiliary positioning information; Convert the coordinates in the initial sound source area to the coordinate system of the camera to obtain a converted sound source area; The distance between the converted sound source area and each recognition target is calculated respectively, and the recognition target with the smallest distance is used as the singing target.

3. The lyrics display method according to claim 2, characterized in that: The step of obtaining the action intensity according to the previous moment position information and the current moment position information of the singing target includes: Calculate the key point displacement difference according to the key point coordinates in the position information at the previous moment and the key point coordinates in the position information at the current moment; Calculate the key point velocity and key point acceleration according to the time difference between the previous moment and the current moment and the displacement difference; Calculating the strength value according to the key point displacement difference, key point velocity difference and key point acceleration difference; The action intensity of the singing target is determined according to the intensity value and a preset intensity level threshold.

4. The lyrics display method according to claim 3, characterized in that: The step of performing action recognition on the singing target to obtain the action category of the singing target includes: Using a skeleton key point recognition algorithm to perform posture recognition on the singing target to obtain posture feature information; Using the optical flow method to identify the movement trend of the singing target to obtain movement feature information; Performing feature splicing on the posture feature information and the motion feature information to obtain category feature information; The category feature information is input into a preset classification model to obtain the action category corresponding to the singing target.

5. The lyrics display method according to claim 4, characterized in that: Before obtaining the target effect mapping table corresponding to the current song, the method includes: Define lyrics display effects and lighting display effects corresponding to different action categories and action intensity combinations to obtain at least one effect mapping table; The step of obtaining a target effect mapping table corresponding to the current song includes: According to the song type of the current song, the corresponding effect mapping table is determined as the current effect mapping table, and the lyrics display effect and / or the light display effect in the current effect mapping table are modified according to the user modification information to obtain the target effect mapping table.

6. A lyrics display system, characterized in that: include: A data acquisition module is used to obtain the image collected by the camera at the current moment and the target effect mapping table corresponding to the current song; A target recognition module, used to perform target recognition on the image using a target recognition model to obtain at least one recognized target and corresponding position information; A target positioning module, used to locate the singing target in at least one recognition target according to the auxiliary positioning information; wherein the auxiliary positioning information is collected by a microphone array or an infrared sensor; A category recognition module, used for performing action recognition on the singing target to obtain the action category of the singing target; An intensity recognition module, used to obtain the action intensity according to the position information of the singing target at the previous moment and the position information of the singing target at the current moment; The effect display module is used to determine the lyrics display effect and the light display effect at the current moment according to the action category, action intensity and target effect mapping table, and to perform corresponding lyrics display and light display effect control.

7. The lyrics display system according to claim 6, characterized in that: The target positioning module comprises: An auxiliary positioning submodule, used to determine an initial sound source area according to the auxiliary positioning information; A conversion submodule, used to convert the coordinates in the initial sound source area into the coordinate system where the camera is located to obtain a converted sound source area; The matching submodule is used to respectively calculate the distance between the converted sound source area and each recognition target, and take the recognition target with the smallest distance as the singing target.

8. The lyrics display system according to claim 7, characterized in that: The strength recognition module comprises: The displacement difference calculation submodule is used to calculate the displacement difference of the key point according to the key point coordinates in the position information at the previous moment and the key point coordinates in the position information at the current moment; The speed calculation submodule is used to calculate the key point speed and the key point acceleration according to the time difference between the previous moment and the current moment and the displacement difference; The strength value calculation submodule is used to calculate the strength value according to the key point displacement difference, key point velocity difference and key point acceleration difference; The action intensity determination submodule is used to determine the action intensity of the singing target according to the intensity value and a preset intensity level threshold.

9. The lyrics display system according to claim 8, characterized in that: The category identification module comprises: A posture recognition submodule is used to perform posture recognition on the singing target by using a skeleton key point recognition algorithm to obtain posture feature information; A motion trend recognition submodule is used to recognize the motion trend of the singing target by using an optical flow method to obtain motion feature information; A feature splicing submodule, used for performing feature splicing on the posture feature information and the motion feature information to obtain category feature information; The action category recognition submodule is used to input the category feature information into a preset classification model to obtain the action category corresponding to the singing target.

10. The lyrics display system according to claim 9, characterized in that: Also includes: A definition module, used to define lyrics display effects and lighting display effects corresponding to different action categories and action intensity combinations, and obtain at least one effect mapping table; The data acquisition module includes: a selection submodule, which is used to determine the corresponding effect mapping table as the current effect mapping table according to the song type of the current song, and modify the lyrics display effect and / or light display effect in the current effect mapping table according to user modification information to obtain a target effect mapping table.

Citation Information

Cited By

  • Environmental atmosphere control method and device based on data analysis, equipment and medium

    CN120730586A