A scene recognition method and device, a storage medium and an electronic device

Abstract

The application discloses a scene recognition method and device, a storage medium and an electronic device, relates to the technical field of computers, and comprises the following steps: in response to scene switching of a target live broadcast, obtaining a to-be-recognized video frame of the target live broadcast; extracting multi-dimensional scene features of the to-be-recognized video frame; according to the association strength information between the multi-dimensional scene features and color change information of the to-be-recognized video frame, performing feature fusion on the multi-dimensional scene features to obtain fused scene features; and identifying the live broadcast scene of the to-be-recognized video frame based on the fused scene features to obtain scene information of the to-be-recognized video frame. The application completes feature fusion by combining the association strength of the multi-dimensional scene features and the color change information, effectively integrates the features, and improves the recognition accuracy of live broadcast scenes such as extreme light and rapid motion by identifying the live broadcast scene based on the fused scene features.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a scene recognition method, apparatus, storage medium and electronic device. Background Technology

[0002] With the development of information technology, scene recognition technology can identify scene content from complex and ever-changing live scene images, and is widely used in fields such as intelligent transportation, intelligent monitoring, and content retrieval.

[0003] Currently, scene recognition methods mainly use feature algorithms to perform single feature matching and scene threshold judgment to achieve the recognition of live streaming scenes.

[0004] However, using this method, which relies solely on the number of feature points matched based on a single feature or a fixed threshold for scene recognition, results in low accuracy in recognizing live streaming scenes such as extreme lighting and fast movement. Summary of the Invention

[0005] In view of this, this application provides a scene recognition method, apparatus, storage medium and electronic device, the main purpose of which is to improve the technical problem that the existing technology only relies on the number of feature points matching based on a single feature or a fixed threshold to perform scene recognition, resulting in low recognition accuracy for live streaming scenes such as extreme lighting and fast movement.

[0006] Firstly, this application provides a scene recognition method, including: In response to a scene change in the target live stream, the video frame to be identified in the target live stream is obtained; Extract multi-dimensional scene features from the video frame to be identified; Based on the correlation strength information between the multi-dimensional scene features and the color change information of the video frame to be identified, the multi-dimensional scene features are fused to obtain fused scene features; Based on the fused scene features, the live scene of the video frame to be identified is identified to obtain the scene information of the video frame to be identified.

[0007] Optionally, the extraction of multi-dimensional scene features from the video frame to be identified includes: The hue distribution map corresponding to the video frame to be identified is divided into multiple hue intervals. Based on the pixel quantity information of the multiple hue intervals in the hue distribution map, the hue distribution uniformity of the multiple hue intervals in the hue distribution map is analyzed to determine the first scene feature of the video frame to be identified. Based on the saturation information of each pixel in the video frame to be identified, the difference between the saturation of each pixel and the average saturation of the video frame to be identified is analyzed to determine the second scene feature of the video frame to be identified; Extract the pixel value change gradient information of each pixel, and generate the color change information of each pixel according to the color change of each pixel. Then, fuse the pixel value change gradient information with the color change information to obtain the third scene feature of the video frame to be identified.

[0008] Optionally, the step of fusing the multi-dimensional scene features based on the correlation strength information between the multi-dimensional scene features and the color change information of the video frame to be identified, to obtain fused scene features, includes: Determine the first association relationship between the first scene feature, the second scene feature, and the third scene feature and the color change information, analyze the association strength between each feature and the color change information, and determine the first fusion weight corresponding to the first scene feature, the second scene feature, and the third scene feature based on the first association relationship; Determine the second association relationship between any two features among the first scene feature, the second scene feature, and the third scene feature; analyze the association strength between any two features; and determine the second fusion weight corresponding to any two features based on the second association relationship. Based on the first fusion weight and the second fusion weight, the first scene feature, the second scene feature and the third scene feature are fused to obtain the fused scene feature.

[0009] Optionally, before acquiring the video frame to be identified from the target live stream in response to a scene switch in response to the target live stream, the method further includes: Extract the first-frame feature vector of the current video frame and the second-frame feature vector of the historical video frames from the target live stream; Based on the vector difference information between the first image feature vector and the second image feature vector, the scene difference information between the current video frame and the historical video frame is determined; If the scene difference information meets the scene switching conditions, it is determined that the target live stream has undergone a scene switch.

[0010] Optionally, the step of identifying the live scene of the video frame to be identified based on the fused scene features to obtain the scene information of the video frame to be identified includes: Based on the fused scene features, the probability distribution information of the video frame to be identified in each live streaming scene is determined; The scene information of the video frame to be identified is determined based on the probability distribution information.

[0011] Optionally, determining the scene information of the video frame to be identified based on the probability distribution information includes: Determine the set of video frames corresponding to the video frame to be identified, and perform probability calibration on the probability distribution information of each video frame in the set of video frames to obtain the calibration probability information set of each video frame in the set of video frames in each live broadcast scene. The set of video frames is a series of consecutive video frames containing the video frame to be identified. Based on the calibration probability information set, the target probability information of the target consecutive video frames in the video frame set under the same live broadcast scene is determined; If the target probability information satisfies the continuous frame probability condition, the live scene corresponding to the target probability information is determined as the scene information of the video frame to be identified. If the target probability information does not meet the continuous frame probability condition, the historical live broadcast scene is determined as the scene information of the video frame to be identified.

[0012] Secondly, this application provides a scene recognition device, comprising: The acquisition module is configured to acquire the video frames to be identified in the target live stream in response to scene switching of the target live stream; The extraction module is configured to extract multi-dimensional scene features from the video frame to be identified; The fusion module is configured to perform feature fusion on the multi-dimensional scene features based on the correlation strength information between the multi-dimensional scene features and the color change information of the video frame to be identified, so as to obtain fused scene features; The recognition module is configured to recognize the live scene of the video frame to be recognized based on the fused scene features, and obtain the scene information of the video frame to be recognized.

[0013] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the scene recognition method described in the first aspect.

[0014] Fourthly, this application provides an electronic device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the computer program to implement the scene recognition method described in the first aspect.

[0015] Fifthly, this application provides a computer program product, which includes a computer program that, when executed by a processor, implements the scene recognition method described in the first aspect.

[0016] By employing the above technical solutions, this application provides a scene recognition method, apparatus, storage medium, and electronic device, comprising: acquiring a video frame to be recognized from the target live stream in response to scene switching; extracting multi-dimensional scene features from the video frame to be recognized; fusing the multi-dimensional scene features according to the correlation strength information between the multi-dimensional scene features and the color change information of the video frame to be recognized, to obtain fused scene features; and recognizing the live scene of the video frame to be recognized based on the fused scene features, to obtain scene information of the video frame to be recognized. Compared with the prior art, this application achieves complete acquisition of multi-dimensional feature information of the live stream by extracting multi-dimensional scene features from the video frame to be recognized, and achieves effective feature integration by combining the correlation strength between the multi-dimensional scene features and the color change information. By performing live scene recognition based on the fused scene features, the recognition accuracy of live stream scenes such as extreme lighting and fast movement is improved. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a scene recognition method provided in an embodiment of this application is shown; Figure 2 A flowchart illustrating a scene recognition method provided in an embodiment of this application is shown; Figure 3 This paper shows a schematic diagram of the structure of a scene recognition device provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0020] The embodiments of this application will now be described in more detail with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0021] To address the issue that existing technologies rely solely on a single feature point matching quantity or a fixed threshold for scene recognition, resulting in low accuracy in identifying live streaming scenes with extreme lighting or rapid motion, this embodiment provides a scene recognition method. Figure 1As shown, the method includes: Step 101: In response to the scene switching of the target live stream, obtain the video frame to be identified in the target live stream.

[0022] In this application embodiment, the target live stream can be various types of video live streams such as sports event live streams and studio live streams. Specifically, the target live stream can include, but is not limited to, live streams of football matches, basketball matches, table tennis matches, and comprehensive sports events. The target live stream can be used to provide real-time video playback services to users.

[0023] In the embodiments of this application, scene switching can be a state in which the camera view changes significantly during the live broadcast. Scene switching may include, but is not limited to, camera position switching, shooting object switching, and shooting scene switching from the competition venue to the studio.

[0024] In this embodiment of the application, the video frame to be identified may be a single frame image corresponding to a scene change in the target live video stream.

[0025] Step 102: Extract multi-dimensional scene features from the video frames to be identified.

[0026] In this embodiment of the application, the multi-dimensional scene features can be feature data extracted from different angles such as color distribution, color purity, and contour structure of the video frame to be identified. Specifically, the multi-dimensional scene features can include a first scene feature, a second scene feature, and a third scene feature.

[0027] In the embodiments of this application, the extraction of multi-dimensional scene features can be achieved based on the Hue-Saturation-Value (HSV) color space. The HSV color space can be a video image color expression form that is different from RGB and YUV. The HSV color space can be divided into three dimensions: hue, saturation, and brightness. The HSV color space can accurately represent the color-related attributes of the picture and adapt to complex picture scenes such as extreme lighting, low light, and fast movement in live broadcasts.

[0028] In this embodiment of the application, the acquired video frame to be identified can be converted to the HSV color space, and features can be calculated and extracted from three dimensions: hue distribution, saturation fluctuation, and color combination edge contour, to obtain complete multi-dimensional scene features.

[0029] Step 103: Based on the correlation strength information between the multi-dimensional scene features and the color change information of the video frame to be identified, the multi-dimensional scene features are fused to obtain the fused scene features.

[0030] In this embodiment, the color change information can be the numerical fluctuation of hue, saturation, and brightness of each pixel in the video frame to be identified. Specifically, the color change information can be quantified by a color tendency weight matrix. The color change information can be used to evaluate the degree of matching between different features and the semantics of the image color.

[0031] In the embodiments of this application, the correlation strength information can be the degree of correlation between multi-dimensional scene features and color change information, or between different scene features. The correlation strength can be quantified and calculated using mutual information.

[0032] In this embodiment, feature fusion can be the integration of multi-dimensional scene features according to corresponding weights. Unlike simple feature splicing or fixed-ratio weighting, this embodiment can use an asymmetric weight allocation method to complete feature fusion.

[0033] In this embodiment of the application, the correlation strength between each scene feature and color change information, as well as the correlation strength between any two features, can be calculated. Corresponding fusion weights are configured according to the correlation strength. The multi-dimensional scene features are integrated and calculated in combination with the weights to generate fused scene features.

[0034] Step 104: Based on the fusion scene features, identify the live scene of the video frame to be identified to obtain the scene information of the video frame to be identified.

[0035] In this application embodiment, the live streaming scene can be a classification tag for the content of the live streaming screen. The live streaming scene may specifically include, but is not limited to, football scene, basketball scene, table tennis scene, studio scene, etc.

[0036] In this embodiment of the application, the scene information may be the live scene category to which the video frame to be identified belongs.

[0037] In the embodiments of this application, scene recognition can be based on the calculation and analysis of fused scene features by a trained intelligent model. Specifically, the intelligent model can be a scene recognition model, which can be built on a lightweight convolutional neural network (CNN) architecture to adapt to live scene recognition tasks.

[0038] In this embodiment of the application, the fused scene features can be input into the trained scene recognition model, and the model can perform feature parsing and classification operations to output the corresponding live scene judgment result, that is, the scene information of the video frame to be recognized.

[0039] Compared with existing technologies, the embodiments of this application extract multi-dimensional scene features from the video frames to be identified, thereby achieving complete acquisition of multi-dimensional feature information of the live broadcast image. By combining the correlation strength between multi-dimensional scene features and color change information, feature fusion is completed, achieving effective integration of features. By carrying out live broadcast scene recognition based on fused scene features, the recognition accuracy of live broadcast scenes such as extreme lighting and fast movement is improved.

[0040] As an optional approach, when performing the "extraction of multi-dimensional scene features of the video frame to be identified", the following methods can be used, but are not limited to these: Figure 2 As shown, the method includes: Step 201: Divide the hue distribution map corresponding to the video frame to be identified into multiple hue intervals. Based on the pixel quantity information of the multiple hue intervals in the hue distribution map, analyze the hue distribution uniformity of the multiple hue intervals in the hue distribution map, and determine the first scene feature of the video frame to be identified.

[0041] In this embodiment of the application, the hue distribution map can be a pixel distribution statistical map generated based on the hue dimension of the video frame to be identified in the HSV color space. Hue can be used to characterize the specific type of color in the picture, such as red, green, blue, etc.

[0042] In the embodiments of this application, the hue interval can be a continuous interval obtained by dividing the complete hue range from 0° to 360°. For example, in the embodiments of this application, the complete hue range can be divided into 36 equal hue intervals.

[0043] In the embodiments of this application, the pixel quantity information can be the total number of image pixels contained in a single hue range, and the hue distribution uniformity can be used to describe the distribution density of various colors in the entire image.

[0044] In this embodiment of the application, the first scene feature can be hue fluctuation entropy, which can be used to characterize the uniformity of hue distribution within a single frame image. The first scene feature can be used to resist image interference caused by backlighting, strong light, and other lighting conditions.

[0045] In this embodiment, a hue distribution map can be generated based on the video frame to be identified, several hue intervals can be divided according to preset rules, the number of pixels in each hue interval can be counted, and the hue distribution uniformity can be calculated by combining the pixel ratio to obtain the hue fluctuation entropy as the first scene feature.

[0046] For the embodiments of this application, the calculation formula for the first scene feature can be as shown in Formula 1, where K can represent the total number of hue intervals and K can be 36. It can represent the proportion of the number of pixels corresponding to the kth hue interval in the total effective pixels of the entire frame image; It can represent the entropy of hue fluctuation. A larger value indicates a more dispersed color distribution in a single frame, while a smaller value indicates a more concentrated color distribution in a single frame.

[0047] (Formula 1) Step 202: Based on the saturation information of each pixel in the video frame to be identified, analyze the difference between the saturation of each pixel and the average saturation of the video frame to be identified, and determine the second scene feature of the video frame to be identified.

[0048] In the embodiments of this application, saturation information can be the color purity value corresponding to a single pixel. Saturation can be used to characterize the vividness of a color. The greater the difference in color purity, the more obvious the visual difference in brightness and intensity of the colors in the image. The average saturation can be the average value of the saturation values ​​of all effective pixels in the entire frame image.

[0049] In the embodiments of this application, a pixel can be the smallest unit that constitutes a digital image, and an effective pixel can be the pixel of the main body area of ​​the image after removing invalid areas such as black borders and logos.

[0050] In the embodiments of this application, the second scene feature can be the saturation fluctuation variance, which can be used to characterize the degree of saturation fluctuation within a single frame image and can be adapted to low-light live broadcast scenarios such as studios and stadiums.

[0051] In this embodiment of the application, all valid pixels of the video frame to be identified can be traversed, the saturation value of each pixel can be extracted, the average saturation of the entire frame image can be calculated, and the difference between the saturation of a single pixel and the average saturation can be compared one by one to obtain the second scene feature (saturation fluctuation variance).

[0052] For the embodiments of this application, the calculation formula for the variance of the saturation fluctuation of the second scene feature can be as shown in Formula 2, where N can represent the total number of effective pixels in a single frame image; Si can represent the saturation value corresponding to the i-th pixel; It can represent the average saturation of all valid pixels in a single frame of an image; It can represent the variance of saturation fluctuation. The larger the value, the more obvious the difference in color purity within the image.

[0053] (Formula 2) Step 203: Extract the pixel value change gradient information of each pixel, and generate the color change information of each pixel based on the color change of each pixel. Then, fuse the pixel value change gradient information with the color change information to obtain the third scene feature of the video frame to be identified.

[0054] In this embodiment, the pixel value change gradient information can be an image gradient, which can be used to describe the direction of pixel brightness change and reflect the edge and contour structure of objects in the image.

[0055] In this embodiment, the color change information can be a color weight matrix, which can be a weight data table determined based on the color difference of a single pixel's HSV three channels. The color change information can be used to evaluate the color consistency of a single pixel.

[0056] In the embodiments of this application, the third scene feature can be a color-biased HOG feature. The color-biased HOG feature can combine the image edge contour with color semantics, which can enhance the scene structure features and adapt to the scene of fast movement and camera switching in live broadcast.

[0057] In this embodiment of the application, the execution process of this step can be as follows: extract the image gradient of the video frame to be identified, simultaneously calculate the color tendency weight corresponding to each pixel, and multiply and fuse the image gradient and the color tendency weight element by element to obtain the third scene feature; the element-wise multiplication can be the multiplication operation of two sets of feature data according to their positions.

[0058] For the embodiments of this application, the calculation formula of the color tendency weight matrix can be as shown in Formula 3, where Ci can represent the color weight corresponding to the i-th pixel; , , These can represent the statistical values ​​of color difference of the i-th pixel in the three channels of hue, saturation, and lightness, respectively. , , These can represent the blending weight coefficients for the hue, saturation, and brightness channels, respectively. The initial values ​​of the blending weight coefficients can be set to 0.4, 0.6, and 0.4, respectively.

[0059] (Formula 3) For the embodiments of this application, the calculation formula for the third scene feature color tendency HOG feature can be as shown in Formula 4, wherein, Ci can represent gradient information of pixel value changes; Ci can represent the color bias weight matrix. It can represent element-wise multiplication; It can represent the HOG characteristic of color preference.

[0060] (Formula 4) Optionally, when performing the action of "fusing multi-dimensional scene features based on the correlation strength information between multi-dimensional scene features and color change information of the video frame to be identified to obtain fused scene features", the following method can be used, but is not limited to: determining the first correlation relationship between the first scene feature, the second scene feature, and the third scene feature and the color change information, analyzing the correlation strength between each feature and the color change information, and determining the first fusion weights corresponding to the first scene feature, the second scene feature, and the third scene feature based on the first correlation relationship; determining the second correlation relationship between any two features among the first scene feature, the second scene feature, and the third scene feature, analyzing the correlation strength between any two features, and determining the second fusion weights corresponding to any two features based on the second correlation relationship; and fusing the first scene feature, the second scene feature, and the third scene feature based on the first fusion weight and the second fusion weight to obtain fused scene features.

[0061] In the embodiments of this application, the first scene feature may be hue fluctuation entropy, the second scene feature may be saturation fluctuation variance, and the third scene feature may be color tendency HOG feature.

[0062] In this embodiment, the first association relationship can be a data dependency relationship between a single feature and color change information, and the association strength can be quantified by mutual information.

[0063] In the embodiments of this application, the first fusion weight can be an asymmetric semantic perception weight, which can be used to characterize the response intensity of a single feature to color fluctuation semantics, and the value range can be (0,1).

[0064] In this embodiment of the application, the second association relationship can be the data interaction relationship between any two types of features, and the second fusion weight can be the asymmetric feature interaction weight. The asymmetric feature interaction weight can be used to characterize the interaction strength between two sets of features, and the value range can be (0,1).

[0065] In this embodiment, the mutual information between the three types of features and the color tendency weight matrix can be calculated, and after normalization, asymmetric semantic perception weights can be obtained; the mutual information of the three types of features in pairs can be calculated, and after normalization, asymmetric feature interaction weights can be obtained; the feature weighting operation is completed by combining the two types of weights to generate fused scene features.

[0066] For the embodiments of this application, the calculation formula for the asymmetric semantic perception weight (i.e., the first fusion weight in the embodiments of this application) can be as shown in Formula 5, wherein, Fi can represent the asymmetric semantic perception weight corresponding to the i-th feature; Fi can represent the feature vector, where i can take the values ​​1, 2, and 3, corresponding to the first scene feature, the second scene feature, and the third scene feature, respectively; Ci can represent the color tendency weight matrix; MI(Fi,Ci) can represent the mutual information between the i-th feature and the color tendency weight matrix.

[0067] (Formula 5) For the embodiments of this application, the calculation formula of MI(Fi,Ci) can be as shown in Formula 6, where p(x,y) can represent the joint probability distribution of features and color weights, and p(x) and p(y) can represent the marginal probability distributions corresponding to features and color weights, respectively.

[0068] (Formula 6) For the embodiments of this application, the calculation formula for the asymmetric feature interaction weight (i.e., the second fusion weight in the embodiments of this application) can be as shown in Formula 7, where MI(Fi,Fj) can represent the mutual information between feature i and feature j. It can represent the asymmetric feature interaction weights between feature i and feature j. It can represent the sum of all mutual information after any two of the following three scene features are combined: the first scene feature, the second scene feature, and the third scene feature.

[0069] (Formula 7) For the embodiments of this application, the calculation formula for fusing scene features can be as shown in Formula 8, wherein, It can represent the characteristics of a fused scene; This can represent the asymmetric semantic perception weight corresponding to the i-th feature (i.e., the first fusion weight in the embodiment of this application). The weights can represent asymmetric feature interaction weights (i.e., the second fusion weights in this embodiment); Fi and Fj can represent feature vectors of any two features among the first scene feature, the second scene feature, and the third scene feature.

[0070] (Formula 8) Optionally, before performing "receiving the target live stream video frame to be identified in response to scene switching of the target live stream", the following method can be used, but is not limited to: extracting the first screen feature vector of the current video frame and the second screen feature vector of the historical video frames in the target live stream; determining the scene difference information between the current video frame and the historical video frames based on the vector difference information between the first screen feature vector and the second screen feature vector; and determining that the target live stream has undergone scene switching if the scene difference information meets the scene switching conditions.

[0071] In this embodiment of the application, the current video frame can be the latest frame image captured in real time in a live video stream.

[0072] In this embodiment of the application, the historical video frames can be consecutive past video frames stored in a sliding window; for example, the number of frames in the sliding window can be set to 20 frames.

[0073] In the embodiments of this application, the image feature vector can be a lightweight feature extracted based on the improved lightweight feature extraction network XFeat architecture. The feature dimension can be set to 64 dimensions. The improved XFeat architecture can reduce the number of parameters by 30% on the basis of the original XFeat architecture, thereby achieving lightweight feature extraction.

[0074] In the embodiments of this application, vector difference information can be calculated using Mahalanobis distance. Mahalanobis distance can be used to quantify the degree of difference between two sets of feature vectors, and at the same time, it can eliminate the correlation interference between different feature dimensions.

[0075] In this embodiment, the scene switching condition can be a dynamic adaptive threshold condition, which can be updated in real time by combining the Mahalanobis distance data of historical frames.

[0076] In this embodiment, the image feature vectors of the current video frame and historical video frames can be extracted by improving the XFeat architecture. The difference between the two sets of vectors is calculated using Mahalanobis distance. The difference is then combined with a dynamic threshold to determine whether the difference meets the standard. If the standard is met, it is determined that the live broadcast scene has changed.

[0077] For the embodiments of this application, the formula for calculating the Mahalanobis distance can be as shown in Formula Nine, where, It can represent the feature vector of the current frame; It can represent a vector of historical feature means; It can represent the squared Mahalanobis distance between the current frame and historical frames; It can represent the inverse matrix of the historical feature covariance matrix.

[0078] (Formula 9) For the embodiments of this application, the calculation formula for the dynamic threshold can be as shown in Formula 10, where Th1 can represent the dynamic threshold; max() can represent taking the maximum value of the squared Mahalanobis distance of the most recent 5 frames; std() can represent taking the standard deviation of the squared Mahalanobis distance of the most recent 5 frames. , It can represent weighting coefficients. This can be used to perform weighted scaling on the maximum value of the squared Mahalanobis distance of the most recent 5 frames, determine the baseline component for dynamic thresholding, amplify the largest image differences that have occurred in historical frames, and avoid the problem of small, normal jitter in the live broadcast being mistaken for a camera switch. For example... The value can be 1.2; This can be used to perform a weighted calculation on the standard deviation of the squared Mahalanobis distance of the most recent 5 frames, incorporating the fluctuation range of historical frame differences. When the frame fluctuation is severe, the standard deviation value increases, and the threshold increases accordingly; when the frame is stable, the threshold decreases accordingly, achieving adaptive threshold adjustment. For example... The value can be 0.8.

[0079]

[0080] (Formula 10) For the embodiments of this application, the scene switching determination logic can be as shown in Formula 11, where Len_switch_flag can represent the camera switching flag, a value of 1 can represent that a scene switching has been detected, and a value of 0 can represent that a scene switching has not been detected.

[0081] (Formula Eleven) Optionally, when performing the task of "identifying the live scene of the video frame to be identified based on the fused scene features and obtaining the scene information of the video frame to be identified", the following method can be used, but is not limited to: determining the probability distribution information of the video frame to be identified in each live scene based on the fused scene features; and determining the scene information of the video frame to be identified based on the probability distribution information.

[0082] In this embodiment of the application, the probability distribution information can be the probability value of the video frame to be identified belonging to each preset live streaming scene, and the sum of the probability values ​​corresponding to all scenes can be 1.

[0083] In the embodiments of this application, scene recognition can be implemented based on a scene recognition model. The scene recognition model can be built with the MobileNetV3-Large lightweight CNN architecture as the backbone network. The model can embed a Squeeze-and-Excitation Block (SEBlock) and a Spatial Pyramid Pooling (SPP) module. SEBlock can be used to enhance the weights of key feature channels, and SPP can be used to extract multi-scale scene features.

[0084] In this embodiment, the fused scene features are input into the scene recognition model, and the model's internal feature extraction function and classification matrix complete the calculation, outputting the probability distribution information corresponding to each preset live streaming scene, and determining the scene information based on the probability distribution results.

[0085] For the embodiments of this application, the calculation formula for the scene probability distribution can be as shown in Formula Twelve, where, It can represent a feature vector after concatenating multi-dimensional features; W can represent the feature extraction function; W can represent the classification weight coefficient matrix; b can represent the bias coefficient. It can represent the probability distribution corresponding to each live streaming scenario.

[0086] (Formula 12) Optionally, when performing the "determining the scene information of the video frame to be identified based on the probability distribution information", the following method can be used, but is not limited to: determining the set of video frames corresponding to the video frame to be identified; performing probability calibration on the probability distribution information of each video frame in the set of video frames to obtain a set of calibration probability information for each video frame in the set of video frames in each live scene; the set of video frames is a set of multiple consecutive video frames containing the video frame to be identified; based on the set of calibration probability information, determining the target probability information of the target consecutive video frames in the set of video frames in the same live scene; if the target probability information satisfies the consecutive frame probability condition, determining the live scene corresponding to the target probability information as the scene information of the video frame to be identified; if the target probability information does not satisfy the consecutive frame probability condition, determining the historical live scene as the scene information of the video frame to be identified.

[0087] In this embodiment of the application, the video frame set can be a series of video frames determined based on the scene dynamic complexity. The scene dynamic complexity can be represented by a complexity coefficient C, and the value range of C can be [0,1]. For example, if C is greater than or equal to 0.7, the live scene can be determined as a high complexity scene, and if C is less than or equal to 0.3, the live scene can be determined as a low complexity scene.

[0088] For the embodiments of this application, the formula for calculating the number of frames in the video frame set can be shown in Formula Thirteen, where C can represent the scene complexity coefficient, m can be 15 for high complexity scenes, and m can be 10 for low complexity scenes.

[0089] (Formula Thirteen) In the embodiments of this application, the consecutive frame probability condition can be that the maximum probability of multiple consecutive frames all points to the same scene, and the maximum confidence of a single frame is not less than 0.5.

[0090] In the embodiments of this application, the historical live streaming scene can be a valid scene tag that has been confirmed and stored in the previous process. Using the historical scene can avoid the problem of frequent switching of scene tags and discontinuous judgment under complex scenes.

[0091] In this embodiment, a set of consecutive video frames is determined based on the complexity of the image. The probability distribution of each frame in the set is calibrated using the Platt scaling method, and the judgment results of consecutive frames are statistically analyzed. If multiple consecutive frames point to the same scene and the probability meets the standard, the scene is output as the live scene. If the judgment results are messy and inconsistent, the previous historical live scene is continued to be used.

[0092] For the embodiments of this application, the loss function corresponding to the Platt scaling method can be as shown in Formula Fourteen, where, It can represent the scaling factor for the k-th type of scene. y can represent the offset coefficient of the k-th scene; y can represent the real scene label, with a value of 1 indicating that it belongs to the k-th scene and a value of 0 indicating that it does not belong to the k-th scene. It can represent a loss function used to solve for the optimal calibration parameters.

[0093]

[0094] (Formula Fourteen) In this embodiment of the application, the calculation formula for probability calibration can be as shown in Formula Fifteen, wherein, It can represent the raw score output by the scene recognition model; It can represent the scaling factor for the k-th type of scene. It can represent the offset coefficient of the k-th type of scene; It can represent the scene probability after calibration.

[0095] (Formula 15) For the embodiments of this application, the formula for calculating the dynamic complexity of the scene can be as shown in Formula Sixteen, where, can represent the weight coefficient, and in this embodiment, the value can be 0.6; var() can represent the variance operation, used to characterize the degree of fluctuation of the feature; C can represent the scene dynamic complexity coefficient.

[0096]

[0097] (Formula Sixteen) In this embodiment, the scene recognition model can be a pre-trained artificial intelligence model. Before executing the live scene recognition process, the model needs to complete overall training, optimization and calibration. Model training can include multiple steps such as dataset processing, feature dimension alignment, network parameter tuning, transfer learning, probability calibration and so on.

[0098] For the embodiments of this application, the dataset used for model training can be a dataset of collected live video frames. The dataset can be sampled in layers according to event type, shot type, and lighting conditions. The number of sample frames corresponding to each type of live scene can be no less than 100,000 frames. The dataset can include edge sub-scenes such as backlight, low light, fast movement, and screen occlusion. The proportion of edge sub-scenes can be no less than 20%, thereby ensuring the model's adaptability to complex live scenes.

[0099] In this embodiment, dataset preprocessing may include image cropping, data augmentation, and sample labeling. Image cropping may use an edge detection algorithm to automatically locate the effective region of interest (ROI) of the image, remove invalid content such as black borders and logos in the video frame, and ensure that the ROI area accounts for no less than 90%. Data augmentation may include random horizontal flipping, brightness perturbation, contrast adjustment, Gaussian blur, and HSV channel-specific perturbation. The execution probability of random horizontal flipping may be set to 0.5, the brightness perturbation range may be ±15%, the contrast adjustment range may be ±20%, the Gaussian kernel size may be no greater than 3×3 and the execution probability may be 0.3, the hue shift range may be ±5°, the saturation adjustment range may be ±15%, and the brightness fine-tuning range may be ±10%.

[0100] In the embodiments of this application, sample annotation can adopt AI pre-annotation, manual review, and cross-validation. The annotation accuracy can be no less than 99.5%, and the annotation confidence of each sample frame can be recorded to provide basic data for subsequent model probability calibration.

[0101] In this embodiment of the application, to ensure a balanced distribution of the dataset, the feature distribution entropy of various sub-scenes can also be calculated to complete data verification and supplementation. The formula for calculating the feature distribution entropy is shown in Formula 17, where Hs can represent the feature distribution entropy. It can represent the sample proportion of the i-th sub-scene; when the calculated feature distribution entropy Hs is less than 0.8, additional samples can be collected for the corresponding scene to improve the generalization ability of the dataset.

[0102] (Formula 17) In this embodiment, the backbone network for model training can be a MobileNetV3-Large lightweight CNN architecture. This network can include one input convolutional layer, ten depthwise separable convolutional layers, two bottleneck structures, one global average pooling layer, and a classification head. The total number of network layers can be 14. The pre-trained weights can be obtained based on the Kinetics-400 video dataset. The network can embed an SEBlock channel attention module and an SPP spatial pyramid pooling module. The SEBlock can be placed at the output of the bottleneck structure to enhance the weights of key feature channels through a squeeze-excitation mechanism. The SPP module can be placed before the global average pooling and uses 1×1, 3×3, and 5×5 pooling kernels to extract multi-scale features.

[0103] In this embodiment of the application, after completing the multi-dimensional scene feature extraction, a feature dimension alignment operation needs to be performed to adapt to the input specifications of MobileNetV3-Large. The first scene feature and the second scene feature can be mapped to 64-dimensional features through 1×1 convolution, while the third scene feature retains its original 256-dimensional dimensions. The three types of features are concatenated to form a 384-dimensional feature vector X^((t)). The feature vector expression can be shown in Formula 18, where... It can represent the hue fluctuation entropy corresponding to the current frame. This can represent the variance of saturation fluctuation corresponding to the current frame. It can represent the color tendency HOG feature corresponding to the current frame.

[0104] (Formula 18) In this embodiment, model training can employ a transfer learning strategy, which can be divided into three parts: network layer freezing, hyperparameter configuration, and secondary fine-tuning. Network layer freezing can freeze the first 12 layers of the MobileNetV3-Large network. This part of the network mainly learns general visual features such as edges and textures, and does not require retraining. Only the last two layers, the SEBlock module, the SPP module, and the classification head are unfrozen for specialized training.

[0105] In this embodiment of the application, the training hyperparameters can be fixed, the learning rate can employ a cosine annealing strategy, the initial learning rate can be set to 1e-4, the minimum learning rate can be set to 1e-6, the learning rate variation period can be set to 50 epochs, and the batch size for a single training iteration can be set to 64. During training, the FocalLoss loss function can be used to address the class imbalance problem, and the FocalLoss hyperparameters can be set to... =0.25、 =2.0. When the validation set accuracy no longer improves for three consecutive epochs, the first 8 layers of the network can be unfrozen, and the learning rate can be reduced to one-tenth of the initial value to complete the second fine-tuning of the model.

[0106] In this embodiment, after completing the basic network training, the Platt scaling method can be used to calibrate the model's output probabilities, further improving the model's recognition accuracy. The original output scores of the model from the validation set samples and the true labels can be used to solve for the scaling and offset coefficients corresponding to each scene type through logistic regression. The loss function used in the solution process can be as shown in Equation Fourteen.

[0107] For the embodiments of this application, probability calibration can be performed as shown in Formula 15. After completing all training, fine-tuning and calibration, the scene detection model can meet the performance requirements of Top-3 accuracy of not less than 98.2% and single-frame inference time of not more than 10ms.

[0108] Compared with existing technologies, the embodiments of this application improve the recognition accuracy in extreme lighting and fast scenes by extracting scene features from three dimensions: hue distribution, saturation difference, gradient and color fusion; further ensure the real-time performance of scene recognition by comparing the vector differences between the current video frame and historical video frames to determine scene switching; and avoid frequent changes in recognition results by calibrating the probability of continuous video frames and combining the determination rules to determine the final scene information, thereby effectively improving the continuity of live scene recognition results and video playback.

[0109] Furthermore, as Figure 1 and Figure 2 The specific implementation of the method shown in this embodiment provides a scene recognition device, such as... Figure 3 As shown, the device includes: an acquisition module 31, an extraction module 32, a fusion module 33, and an identification module 34.

[0110] The acquisition module 31 is configured to acquire the target live video frame to be identified in response to the scene switching of the target live stream; Extraction module 32 is configured to extract multi-dimensional scene features from the video frame to be identified; The fusion module 33 is configured to fuse the multi-dimensional scene features based on the correlation strength information between the multi-dimensional scene features and the color change information of the video frame to be identified, so as to obtain the fused scene features. The recognition module 34 is configured to recognize the live scene of the video frame to be recognized based on the fused scene features, and obtain the scene information of the video frame to be recognized.

[0111] In some examples of this embodiment, the extraction module 32 is specifically configured to divide the hue distribution map corresponding to the video frame to be identified into multiple hue intervals, analyze the hue distribution uniformity of the multiple hue intervals in the hue distribution map based on the pixel quantity information of the multiple hue intervals in the hue distribution map, and determine the first scene feature of the video frame to be identified; analyze the difference between the saturation of each pixel and the average saturation of the video frame to be identified based on the saturation information of each pixel in the video frame to be identified, and determine the second scene feature of the video frame to be identified; extract the pixel value change gradient information of each pixel, generate the color change information of each pixel based on the color change of each pixel, and fuse the pixel value change gradient information with the color change information to obtain the third scene feature of the video frame to be identified.

[0112] In some examples of this embodiment, the fusion module 33 is specifically configured to: determine the first association relationship between the first scene feature, the second scene feature, and the third scene feature and the color change information; analyze the association strength between each feature and the color change information; determine the first fusion weights corresponding to the first scene feature, the second scene feature, and the third scene feature based on the first association relationship; determine the second association relationship between any two features among the first scene feature, the second scene feature, and the third scene feature; analyze the association strength between any two features; determine the second fusion weights corresponding to any two features based on the second association relationship; and perform feature fusion on the first scene feature, the second scene feature, and the third scene feature based on the first fusion weight and the second fusion weight to obtain the fused scene feature.

[0113] In some examples of this embodiment, the acquisition module 31 is specifically configured to extract the first screen feature vector of the current video frame and the second screen feature vector of the historical video frames in the target live stream; determine the scene difference information between the current video frame and the historical video frames based on the vector difference information between the first screen feature vector and the second screen feature vector; and determine that the target live stream has undergone a scene switch when the scene difference information meets the scene switching conditions.

[0114] In some examples of this embodiment, the identification module 34 is specifically configured to determine the probability distribution information of the video frame to be identified in each live scene based on the fused scene features; and to determine the scene information of the video frame to be identified based on the probability distribution information.

[0115] In some examples of this embodiment, the identification module 34 is further configured to determine the set of video frames corresponding to the video frame to be identified, perform probability calibration on the probability distribution information of each video frame in the set of video frames to obtain a set of calibration probability information for each video frame in the set of video frames in each live broadcast scene, wherein the set of video frames is a set of multiple consecutive video frames containing the video frame to be identified; based on the set of calibration probability information, determine the target probability information of the target consecutive video frames in the set of video frames in the same live broadcast scene; if the target probability information satisfies the consecutive frame probability condition, determine the live broadcast scene corresponding to the target probability information as the scene information of the video frame to be identified; if the target probability information does not satisfy the consecutive frame probability condition, determine the historical live broadcast scene as the scene information of the video frame to be identified.

[0116] It should be noted that other corresponding descriptions of the functional units involved in the scene recognition device provided in this embodiment can be found in [reference]. Figure 1 and Figure 2 The corresponding description in [the document] will not be repeated here.

[0117] Based on the above, Figure 1 and Figure 2 Accordingly, this embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method. Figure 1 and Figure 2 The method shown.

[0118] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of this application.

[0119] like Figure 4 The diagram shown is a hardware structure schematic of an electronic device according to the present invention, comprising: At least one processor 401; and, Memory 402 is communicatively connected to at least one processor 401; wherein, The memory 402 stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the scene recognition method as described above.

[0120] Figure 4 Take a processor 401 as an example.

[0121] The electronic device may also include an input device 403 and an output device 404.

[0122] The processor 401, memory 402, input device 403, and output device 404 can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0123] Memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the scene recognition method in the embodiments of this application, for example, Figure 1 and Figure 2 The method flow is shown. The processor 401 executes various functional applications and data processing by running non-volatile software programs, instructions, and modules stored in the memory 402, thereby realizing the scene recognition method in the above embodiments.

[0124] The memory 402 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the scene recognition method. Furthermore, the memory 402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 402 may optionally include memory remotely located relative to the processor 401, and these remote memories may be connected to the apparatus performing the scene recognition method via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0125] The input device 403 can receive user clicks and generate signal inputs related to user settings and function control for the scene recognition method. The output device 404 may include a display device such as a screen.

[0126] One or more modules are stored in memory 402, and when run by one or more processors 401, the scene recognition method in any of the above method embodiments is executed.

[0127] Optionally, the aforementioned physical devices may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB interfaces, card reader interfaces, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Wi-Fi interfaces), etc.

[0128] Those skilled in the art will understand that the physical device structure provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0129] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the aforementioned physical device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms, or it can be implemented by hardware. By applying the solution of this embodiment, compared with related technologies, this application embodiment achieves complete acquisition of multi-dimensional feature information of live broadcast images by extracting multi-dimensional scene features of the video frame to be identified, and completes feature fusion by combining the correlation strength of multi-dimensional scene features and color change information to achieve effective feature integration. By carrying out live broadcast scene recognition based on fused scene features, the recognition accuracy of live broadcast scenes such as extreme lighting and fast movement is improved; by comparing the vector difference between the current video frame and historical video frames to determine scene switching, the real-time performance of scene recognition is further guaranteed; by calibrating the probability of continuous video frames and combining the determination rules to determine the final scene information, frequent jumps in recognition results are avoided, effectively improving the continuity of live broadcast scene recognition results and screen playback.

[0131] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0132] The above are merely specific embodiments of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A scene recognition method, characterized in that, include: In response to a scene change in the target live stream, the video frame to be identified in the target live stream is obtained; Extract multi-dimensional scene features from the video frame to be identified; Based on the correlation strength information between the multi-dimensional scene features and the color change information of the video frame to be identified, the multi-dimensional scene features are fused to obtain fused scene features; Based on the fused scene features, the live scene of the video frame to be identified is identified to obtain the scene information of the video frame to be identified.

2. The method according to claim 1, characterized in that, The extraction of multi-dimensional scene features from the video frame to be identified includes: The hue distribution map corresponding to the video frame to be identified is divided into multiple hue intervals. Based on the pixel quantity information of the multiple hue intervals in the hue distribution map, the hue distribution uniformity of the multiple hue intervals in the hue distribution map is analyzed to determine the first scene feature of the video frame to be identified. Based on the saturation information of each pixel in the video frame to be identified, the difference between the saturation of each pixel and the average saturation of the video frame to be identified is analyzed to determine the second scene feature of the video frame to be identified; Extract the pixel value change gradient information of each pixel, and generate the color change information of each pixel according to the color change of each pixel. Then, fuse the pixel value change gradient information with the color change information to obtain the third scene feature of the video frame to be identified.

3. The method according to claim 2, characterized in that, The step of fusing the multi-dimensional scene features based on the correlation strength information between the multi-dimensional scene features and the color change information of the video frame to be identified, to obtain fused scene features, includes: Determine the first association relationship between the first scene feature, the second scene feature, and the third scene feature and the color change information, analyze the association strength between each feature and the color change information, and determine the first fusion weight corresponding to the first scene feature, the second scene feature, and the third scene feature based on the first association relationship; Determine the second association relationship between any two features among the first scene feature, the second scene feature, and the third scene feature; analyze the association strength between any two features; and determine the second fusion weight corresponding to any two features based on the second association relationship. Based on the first fusion weight and the second fusion weight, the first scene feature, the second scene feature and the third scene feature are fused to obtain the fused scene feature.

4. The method according to claim 1, characterized in that, Before acquiring the target live stream's video frame to be identified in response to a scene switch in the target live stream, the method further includes: Extract the first-frame feature vector of the current video frame and the second-frame feature vector of the historical video frames from the target live stream; Based on the vector difference information between the first image feature vector and the second image feature vector, the scene difference information between the current video frame and the historical video frame is determined; If the scene difference information meets the scene switching conditions, it is determined that the target live stream has undergone a scene switch.

5. The method according to claim 1, characterized in that, The step of identifying the live scene of the video frame to be identified based on the fused scene features to obtain the scene information of the video frame to be identified includes: Based on the fused scene features, the probability distribution information of the video frame to be identified in each live streaming scene is determined; The scene information of the video frame to be identified is determined based on the probability distribution information.

6. The method according to claim 5, characterized in that, Determining the scene information of the video frame to be identified based on the probability distribution information includes: Determine the set of video frames corresponding to the video frame to be identified, and perform probability calibration on the probability distribution information of each video frame in the set of video frames to obtain the calibration probability information set of each video frame in the set of video frames in each live broadcast scene. The set of video frames is a series of consecutive video frames containing the video frame to be identified. Based on the calibration probability information set, the target probability information of the target consecutive video frames in the video frame set under the same live broadcast scene is determined; If the target probability information satisfies the continuous frame probability condition, the live scene corresponding to the target probability information is determined as the scene information of the video frame to be identified. If the target probability information does not meet the continuous frame probability condition, the historical live broadcast scene is determined as the scene information of the video frame to be identified.

7. A scene recognition device, characterized in that, include: The acquisition module is configured to acquire the video frames to be identified in the target live stream in response to scene switching of the target live stream; The extraction module is configured to extract multi-dimensional scene features from the video frame to be identified; The fusion module is configured to perform feature fusion on the multi-dimensional scene features based on the correlation strength information between the multi-dimensional scene features and the color change information of the video frame to be identified, so as to obtain fused scene features; The recognition module is configured to recognize the live scene of the video frame to be recognized based on the fused scene features, and obtain the scene information of the video frame to be recognized.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

9. An electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images