Scene recognition method of video, neural network training method, server and medium

Abstract

The application relates to a scene recognition method of a video, a neural network training method, a server and a medium. The method comprises the following steps: obtaining an image to be recognized; determining a plurality of feature vectors in the image to be recognized, and classifying the feature vectors of the plurality of images to be recognized into a first number of image scene categories; the image scene categories comprise a plurality of image scene categories corresponding to indoor scenes and a plurality of image scene categories corresponding to outdoor scenes; based on the number of feature vectors in each feature category, the probability value of the image to be recognized belonging to each kind of image scene category is determined; based on the probability value of each image scene category, a second number of target image scene categories are selected from the first number of image scene categories, and a target video is recognized as an indoor scene or an outdoor scene based on the second number of target image scene categories. The method can improve the accuracy and robustness of scene recognition of a target video.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a video scene recognition method, a neural network training method, a server, a storage medium, and a computer program product. Background Technology

[0002] With the development of internet technology, live entertainment on social networks is becoming increasingly popular, and the dynamic display of the streamer and their environment is also loved by more and more people. Therefore, the identification of the dynamic live streaming scene (such as indoor and outdoor scenes) in which the streamer is located has become a very popular research topic.

[0003] In traditional live streaming scene identification methods, there are generally two approaches. One approach involves using the electronic devices used by the streamer to send location messages to the server in real time, allowing the server to determine the streamer's geographical location and thus the dynamic live streaming scene in which the streamer is located. The other approach involves using a pre-set neural network model to perform similarity matching on cropped images of the live video in the live streaming room. When the pre-set neural network model finds that the similarity between the cropped image and the standard image is greater than a certain value, the server determines that the dynamic live streaming scene in which the streamer is located is the scene corresponding to the standard image.

[0004] However, the identification of live streaming scenes based on location information is often affected by the satellite signals of electronic devices and the spatial dimension of the actual scene, making the location information obtained by the server inaccurate and untimely, thus resulting in inaccurate identification of live streaming scenes; in addition, the identification of live streaming scenes based on similarity is often limited by the number and type of standard images, which also makes the identification of live streaming scenes inaccurate. Summary of the Invention

[0005] Therefore, it is necessary to provide a video scene recognition method, a convolutional neural network training method, a video scene recognition device, a convolutional neural network training device, a server, a storage medium, and a computer program product that can improve the accuracy of live scene recognition, in order to address the above-mentioned technical problems.

[0006] According to a first aspect of the present disclosure, a video scene recognition method is provided, comprising:

[0007] Acquire an image to be identified, wherein the image to be identified is at least one image frame of the target video;

[0008] Multiple feature vectors in the image to be identified are determined, and the multiple feature vectors are classified into a first number of image scene categories; the feature vectors are used to characterize the pixel features of each pixel in the image to be identified, and the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes;

[0009] Based on the number of feature vectors in each of the image scene categories, the probability value of the image to be identified belonging to each of the image scene categories is determined;

[0010] Based on the probability values ​​of each of the image scene categories, a second number of target image scene categories are selected from the first number of image scene categories, and the target video is identified as the indoor scene or the outdoor scene based on the second number of target image scene categories.

[0011] In an exemplary embodiment, identifying the target video as either the indoor scene or the outdoor scene based on the second number of target image scene categories includes:

[0012] In the second number of target image scene categories, a first number proportion of each target image scene category belonging to the indoor scene is determined, and a second number proportion of each target image scene category belonging to the outdoor scene is determined;

[0013] Based on the relationship between the first quantity ratio and the second quantity ratio, the target video is identified as either the indoor scene or the outdoor scene.

[0014] In an exemplary embodiment, identifying the target video as either the indoor scene or the outdoor scene based on the size relationship between the first quantity ratio and the second quantity ratio includes one of the following three items:

[0015] In response to the first quantity ratio being greater than the second quantity ratio, the target video is identified as the indoor scene;

[0016] In response to the first quantity ratio being less than the second quantity ratio, the target video is identified as the outdoor scene;

[0017] In response to the first quantity percentage being equal to the second quantity percentage, the step of acquiring the image to be identified from the target video is re-executed until the first quantity percentage of the target image scene category corresponding to the new image to be identified is greater than the second quantity percentage, or the first quantity percentage is less than the second quantity percentage.

[0018] In an exemplary embodiment, identifying the target video as either the indoor scene or the outdoor scene based on the second number of target image scene categories includes:

[0019] In the second number of target image scene categories, the probability values ​​and values ​​of each target image scene category belonging to the indoor scene are determined as the first probability values ​​and values, and the probability values ​​and values ​​of each target image scene category belonging to the outdoor scene are determined as the second probability values ​​and values.

[0020] Based on the relationship between the first probability value and the second probability value, the target video is identified as either the indoor scene or the outdoor scene.

[0021] In an exemplary embodiment, identifying the target video as either the indoor scene or the outdoor scene based on the magnitude relationship between the first probability value and the second probability value includes one of the following three:

[0022] In response to the first probability value being greater than the second probability value, the target video is identified as the indoor scene;

[0023] In response to the first probability value being less than the second probability value, the target video is identified as the outdoor scene;

[0024] In response to the first probability value being equal to the second probability value, the step of acquiring the image to be identified from the target video is re-executed until the first probability value of the scene category of the new image to be identified is greater than the second probability value, or the first probability value is less than the second probability value.

[0025] In one exemplary embodiment, the number of images to be identified is no less than two; the acquisition of the images to be identified includes:

[0026] Based on a preset frame interval, the target video is subjected to frame extraction to obtain at least two images to be identified;

[0027] The step of selecting a second number of target image scene categories from the first number of image scene categories includes:

[0028] Among the probability values ​​of the first number of image scene categories to which each image to be identified belongs, the second number of target image scene categories with the highest corresponding probability values ​​are selected.

[0029] In an exemplary embodiment, identifying the target video as either the indoor scene or the outdoor scene based on the second number of target image scene categories includes:

[0030] In the second number of target image scene categories corresponding to each of the images to be identified, a first total number proportion and a first total probability value of the target image scene category belonging to the indoor scene are determined, as well as a second total number proportion and a second total probability value of the target image scene category belonging to the outdoor scene.

[0031] Based on the relationship between the first total quantity ratio and the second total quantity ratio, and / or the relationship between the first total probability value and the second total probability value, the target video is identified as either the indoor scene or the outdoor scene.

[0032] In an exemplary embodiment, determining multiple feature vectors in the image to be identified and classifying the multiple feature vectors into a first number of image scene categories includes:

[0033] Based on a pre-trained convolutional neural network, multiple feature vectors corresponding to multiple pixels are extracted from the image to be identified.

[0034] The feature vectors of each pixel are matched with the first number of reference vectors, and the feature vectors of each pixel are classified into image scene categories that meet the matching degree threshold; wherein each reference vector corresponds to an image scene category.

[0035] In an exemplary embodiment, determining the probability value of the image to be identified belonging to each of the image scene categories based on the number of feature vectors in each of the image scene categories includes:

[0036] In each of the image scene categories, based on the number of feature vectors in the image scene category, an exponential value of the image scene category with respect to the number is determined, where the exponential value represents the base of the natural logarithm;

[0037] Determine the exponential values ​​and values ​​of the exponential values ​​among the respective image scene categories;

[0038] Based on the quotient between the index value corresponding to each of the image scene categories and the index value, the probability value of the image to be identified belonging to each of the image scene categories is determined.

[0039] According to a second aspect of the present disclosure, a method for training a convolutional neural network is provided, comprising:

[0040] Obtain a first training image comprising a first number of image scene categories; the first training image is a scene image extracted from an open-source dataset, and the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes;

[0041] The pixel aspect ratio of the first training image is randomly scaled, and the first training image after random scaling is used to train a convolutional neural network model to obtain the initial convolutional neural network after training.

[0042] Acquire a second training image that includes the first number of image scene categories; the second training image is an image frame extracted from multiple target videos, and the image scene categories corresponding to the multiple target videos include the indoor scene and the outdoor scene;

[0043] The initial convolutional neural network is adjusted based on the second training image to obtain a pre-trained convolutional neural network; wherein the pre-trained convolutional neural network is used to perform any one of the above schemes to determine multiple feature vectors in the image to be identified.

[0044] According to a third aspect of the present disclosure, a video scene recognition device is provided, comprising:

[0045] The image acquisition unit is configured to acquire an image to be identified, wherein the image to be identified is at least one image frame of the target video;

[0046] A vector classification unit is configured to determine multiple feature vectors in the image to be identified and classify the multiple feature vectors into a first number of image scene categories; the feature vectors are used to characterize the pixel features of each pixel in the image to be identified, and the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes;

[0047] The probability calculation unit is configured to determine the probability value of the image to be identified belonging to each of the image scene categories based on the number of feature vectors in each of the image scene categories;

[0048] The scene recognition unit is configured to perform a probability value based on each of the image scene categories, select a second number of target image scene categories from the first number of image scene categories, and identify the target video as the indoor scene or the outdoor scene based on the second number of target image scene categories.

[0049] According to a fourth aspect of the present disclosure, a training apparatus for a convolutional neural network is provided, comprising:

[0050] The first acquisition unit is configured to acquire a first training image including a first number of image scene categories; the first training image is a scene image extracted from an open-source dataset, and the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes;

[0051] The model training unit is configured to perform random scaling of the pixel aspect ratio of the first training image and train a convolutional neural network model on the randomly scaled first training image to obtain the initial convolutional neural network after training.

[0052] The second acquisition unit is configured to acquire a second training image including the first number of image scene categories; the second training image is an image frame extracted from multiple target videos, and the image scene categories corresponding to the multiple target videos include the indoor scene and the outdoor scene;

[0053] The model correction unit is configured to adjust the initial convolutional neural network based on the second training image to obtain a pre-trained convolutional neural network; wherein the pre-trained convolutional neural network is used to determine multiple feature vectors in the image to be identified as described in any of the above schemes.

[0054] According to a fifth aspect of the present disclosure, a server is provided, comprising:

[0055] processor;

[0056] Memory for storing the executable instructions of the processor;

[0057] The processor is configured to execute the executable instructions to implement the live streaming scene recognition method and / or the convolutional neural network training method as described in any of the preceding claims.

[0058] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, the computer-readable storage medium including program data, which, when executed by a processor of a server, enables the server to execute the live room scene recognition method and / or the convolutional neural network training method as described in any of the preceding claims.

[0059] According to a seventh aspect of the present disclosure, a computer program product is provided, the computer program product including program instructions, which, when executed by a processor of a server, enable the server to perform the live room scene recognition method and / or the convolutional neural network training method as described in any of the preceding claims.

[0060] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0061] The method first acquires an image to be identified, which is at least one image frame of the target video. Then, it determines multiple feature vectors in the image to be identified and classifies these feature vectors into a first number of image scene categories. The feature vectors characterize the pixel features of each pixel in the image to be identified, and the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes. Then, based on the number of feature vectors in each feature category, it determines the probability value of the image to be identified belonging to each image scene category. Finally, based on the probability values ​​of each image scene category, it selects a second number of target image scene categories from the first number of image scene categories and identifies the target video as an indoor or outdoor scene based on the second number of target image scene categories. In this way, on the one hand, the feature vectors of the target video are first classified, and then the scene recognition of the target video is performed based on the probability values ​​of the classified feature vectors belonging to each image scene category. This optimizes the process of scene recognition of the video, enhances the generalizability of the application scenarios of scene recognition of video, and reduces the consumption of manpower and time costs. On the other hand, by using the multiple target image scene categories with the highest probability of the image to be recognized to identify whether the target video belongs to an indoor scene or an outdoor scene, the accuracy and robustness of scene recognition of the target video can be improved, which is conducive to subsequent video processing based on the accurately classified target video.

[0062] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0063] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0064] Figure 1 This is an application environment diagram illustrating a video scene recognition method according to an exemplary embodiment.

[0065] Figure 2 This is a flowchart illustrating a video scene recognition method according to an exemplary embodiment.

[0066] Figure 3 This is a flowchart illustrating a step of classifying the feature vector of an image to be identified according to an exemplary embodiment.

[0067] Figure 4 This is a flowchart illustrating a step of determining the probability values ​​of an image to be identified belonging to each image scene category, according to an exemplary embodiment.

[0068] Figure 5This is a flowchart illustrating a first step of identifying a target video as an indoor or outdoor scene according to an exemplary embodiment.

[0069] Figure 6 This is a flowchart illustrating a second step for identifying a target video as an indoor or outdoor scene, according to an exemplary embodiment.

[0070] Figure 7 This is a flowchart illustrating a third step in identifying a target video as an indoor or outdoor scene, according to an exemplary embodiment.

[0071] Figure 8 This is a flowchart illustrating a training method for a convolutional neural network according to an exemplary embodiment.

[0072] Figure 9 This is a flowchart illustrating a video scene recognition method according to another exemplary embodiment.

[0073] Figure 10 This is a flowchart illustrating a scene recognition step for an image to be recognized, according to an exemplary embodiment.

[0074] Figure 11 This is a flowchart illustrating a training method for a convolutional neural network according to another exemplary embodiment.

[0075] Figure 12 This is a block diagram of a video scene recognition device according to an exemplary embodiment.

[0076] Figure 13 This is a block diagram of a training apparatus for a convolutional neural network according to an exemplary embodiment.

[0077] Figure 14 This is a block diagram illustrating a server for scene recognition in video according to an exemplary embodiment.

[0078] Figure 15 This is a block diagram illustrating a computer-readable storage medium for scene recognition in video according to an exemplary embodiment.

[0079] Figure 16 This is a block diagram illustrating a computer program product for scene recognition in video, according to an exemplary embodiment. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0081] The term "and / or" in the embodiments of this application refers to any and all possible combinations including one or more of the associated listed items. It should also be noted that, when used in this specification, "including / comprising" specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, and / or components and / or groups thereof.

[0082] The terms "first," "second," etc., used in this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0083] Furthermore, although the terms "first," "second," etc., are used repeatedly in this application to describe various operations (or various elements, or various applications, or various instructions, or various data), these operations (or elements, or applications, or instructions, or data) should not be limited by these terms. These terms are only used to distinguish one operation (or element, or application, or instruction, or data) from another operation (or element, or application, or instruction, or data). For example, a first image scene category can be called a second image scene category, and a second image scene category can be called a first image scene category; the only difference is the scope they encompass, but this does not depart from the scope of this application. Both the first image scene category and the second image scene category are sets of image scene categories corresponding to various feature categories, but they are not image scene categories corresponding to the same feature categories.

[0084] The video scene recognition method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a communication network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other network servers.

[0085] In some embodiments, reference Figure 1First, server 104 acquires the image to be identified, wherein the image to be identified is at least one image frame of the target video. Then, server 104 determines multiple feature vectors in the image to be identified and classifies the multiple feature vectors into a first number of image scene categories. The feature vectors are used to characterize the pixel features of each pixel in the image to be identified, and the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes. Then, server 104 determines the probability value of the image to be identified belonging to each image scene category based on the number of feature vectors in each feature category. Finally, server 104 selects a second number of target image scene categories from the first number of image scene categories based on the probability values ​​of each image scene category, and identifies the target video as an indoor scene or an outdoor scene based on the second number of target image scene categories.

[0086] In some embodiments, terminal 102 (such as a mobile terminal or a fixed terminal) can be implemented in various forms. Terminal 102 can be a mobile terminal, including mobile phones, smartphones, laptops, portable handheld devices, personal digital assistants (PDAs), tablet computers (PADs), etc., capable of using live video from a live streaming room and identifying whether the live streaming room is an indoor or outdoor scene based on the live video. Terminal 102 can also be a fixed terminal, including Automated Teller Machines (ATMs), self-service machines, digital TVs, desktop computers, fixed-line computers, etc., capable of using live video from a live streaming room and identifying whether the live streaming room is an indoor or outdoor scene based on the live video.

[0087] Hereinafter, it is assumed that terminal 102 is a fixed terminal. However, those skilled in the art will understand that, if there are operations or elements specifically designed for mobile purposes, the construction according to the embodiments disclosed in this application can also be applied to mobile type terminal 102.

[0088] In some embodiments, the data processing component running on server 104 may load any of the various additional server applications and / or middleware applications being executed, such as HTTP (Hypertext Transfer Protocol), FTP (File Transfer Protocol), CGI (Common Gateway Interface), RDBMS (Relational Database Management System), etc.

[0089] In some embodiments, server 104 may be implemented using a standalone server or a server cluster consisting of multiple servers. Server 104 may be adapted to run one or more application services or software components that provide the terminal 102 described in the foregoing disclosure.

[0090] In some embodiments, the application service may include a service interface for providing users with a live video playback interface (e.g., a display interface for showing users the scene category of the streamer in the live room), and corresponding program services, etc. The software components may include, for example, an application (SDK) or client (APP) with the function of identifying the scene (including indoor or outdoor scene) of the live room based on image frames of the live video.

[0091] In some embodiments, the application or client provided by server 104, which has the function of identifying the scene of the live room based on the image frames of the live video, includes a portal port that provides one-to-one application services to users in the foreground and multiple business systems that perform data processing in the background, so as to extend the application of the functions associated with the identified scene category (such as live room recommendation function, live room grouping function, etc.) to the APP or client, so that users can use and access the functions associated with the scene category of the live room at any time and any place.

[0092] In some embodiments, the resource transfer function of an APP or client can be a computer program running in user mode to complete one or more specific tasks, which can interact with the user and has a visual user interface. The APP or client can include two parts: a graphical user interface (GUI) and an engine, which together provide users with a variety of application services in the form of a user interface in a digital client system.

[0093] In some embodiments, users can input corresponding code data or control parameters into the APP or client through a preset input device or automatic control program to execute application services of the computer program in the server 104 and display application services in the user interface.

[0094] As an example, when a user needs to group a live stream room P displayed in a live streaming platform APP running on terminal 102 according to scene categories, the user can input control parameters for extracting the live stream video into terminal 102 through an input device. Then, server 104 identifies indoor or outdoor scenes in the extracted live stream video, thereby grouping the scene category of live stream room P. Finally, server 104 sends code data about the grouped display of live stream room P to terminal 102, so that live stream room P is displayed in the indoor or outdoor scene category of the live streaming platform APP running on terminal 102. Optionally, the input method corresponding to the input device can be touch screen input, button input, voice input, or related control program input, etc.

[0095] In some embodiments, the operating system running the app or client may include various versions of Microsoft... Apple and / or Linux operating system, various commercial or similar Operating systems (including but not limited to various GNU / Linux operating systems, Google) OS and / or mobile operating systems, such as Phone OS OS OS operating systems, as well as other online or offline operating systems, are not specifically limited here.

[0096] In some embodiments, such as Figure 2 As shown, a scene recognition method for videos is provided, which can be applied to... Figure 1 Taking server 104 as an example, the method includes the following steps:

[0097] Step S11: Obtain the image to be recognized.

[0098] In one embodiment, the image to be identified is at least one image frame of the target video.

[0099] In some embodiments, the server extracts a segment of live video of a preset duration from the target live room of the live streaming platform, or the server extracts the video to be identified from a dedicated database to obtain the target video; then, the server performs video frame extraction on the target video at preset time intervals (e.g., frame intervals) to extract the image to be identified. The extracted image to be identified can be a single image or multiple images.

[0100] Understandably, for operational needs based on efficient video scene recognition, the server can identify the scene of the target video based on a small number of images to be recognized (e.g., one image to be recognized); for operational needs based on high-precision video scene recognition, the server can identify the scene of the target video based on a large number of images to be recognized (e.g., ten images to be recognized).

[0101] Step S12: Determine multiple feature vectors in the image to be identified, and classify the multiple feature vectors into a first number of image scene categories.

[0102] In some embodiments, each image to be identified has multiple feature information, all of which can be represented by feature vectors (e.g., using entity vectors to represent "embedding"). The feature vectors are used to characterize the pixel features of each pixel in the image to be identified.

[0103] In some embodiments, the server can input the image to be recognized as image data into a pre-trained deep neural network model (e.g., a residual convolution model, a translation model), and use the high-dimensional feature vector (N, C, H, W) of the output image data as the feature vector of the image to be recognized.

[0104] Where N represents the number of images to be identified corresponding to the image data. For example, if there is only one image to be identified, then N = 1; if there are five images to be identified, then N = 5. C represents the number of data channels of the image data to be identified. That is, C = 3 when the image to be identified is an RGB image, and C = 2 when the image to be identified is a grayscale image. H and W represent the width and height pixel values ​​of the image data to be identified. For example, H = 224 and W = 224.

[0105] In other embodiments, the server may also use other methods to obtain the feature vector of the image to be recognized, as long as the feature information of these images can be expressed as a feature vector. For example, the feature vector of the image to be recognized can be obtained similarly using open-source datasets (e.g., the Open Image dataset, Image Net dataset) and / or other structured models (e.g., other large-scale classification models or image metric learning models); the server may also obtain the feature vector of the image to be recognized by acquiring traditional image features based on SIFT, color histogram, HOG, etc.

[0106] In some embodiments, the server uses a preset clustering algorithm or computer network model to classify the feature vector of the image to be identified into an image scene category. For example, the server uses the K-means algorithm, mean-shift clustering algorithm, density-based clustering method, or convolutional neural network model (e.g., a convolutional neural network based on CNN or RNN) to classify the feature vector of the image to be identified into a preset first number of image scene categories.

[0107] As an example, the server first extracts X feature vectors from the image to be recognized using a pre-trained residual convolutional model. Then, the residual convolutional model classifies the X feature vectors into P pre-prepared image scene categories. Each image scene category represents a set of feature vectors.

[0108] In one embodiment, the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes.

[0109] In some embodiments, the server organizes image data using an open-source dataset and generates multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes using the organized image data.

[0110] In some embodiments, in order to increase the stability and generalization of image scene categories, the server sets an indoor blank scene category among the various image scene categories corresponding to indoor scenes, and sets an outdoor blank scene category among the various image scene categories corresponding to outdoor scenes.

[0111] Specifically, the indoor blank scene category includes processed image data from multiple indoor scene categories, and these processed image data do not belong to any other image scene category belonging to indoor scenes; and the outdoor blank scene category includes processed image data from multiple outdoor scene categories, and these processed image data do not belong to any other image scene category belonging to outdoor scenes.

[0112] Step S13: Based on the number of feature vectors in each image scene category, determine the probability value of the image to be identified belonging to each image scene category.

[0113] In some embodiments, the server may input the feature vectors of a first number of image scene categories into a preset probability statistics algorithm (or confidence algorithm) to perform feature vector normalization processing in order to determine the probability value (or confidence level) of the image to be identified belonging to each image scene category.

[0114] The normalization process includes using a pre-defined confidence algorithm or probability statistics algorithm to map the real number domain to the effective real number space [0, X] representing the probability distribution, based on the number of feature vectors in each image scene category. Here, X is a real number, such as X = 1, 2, etc.

[0115] Step S14: Based on the probability values ​​of each image scene category, select a second number of target image scene categories from the first number of image scene categories, and identify the target video as an indoor scene or an outdoor scene based on the second number of target image scene categories.

[0116] In some embodiments, the server first sorts a first number of image scene categories according to the order of their probability values. Then, it identifies a second number of target image scene categories with the second largest probability values ​​in the sorting. Then, based on the probability values ​​and / or number of indoor scenes and the probability values ​​and / or number of outdoor scenes in these second number of target image scene categories, the server identifies the scene of the image to be identified as an indoor scene or an outdoor scene, thereby identifying the target video as an indoor scene or an outdoor scene.

[0117] In the aforementioned video scene recognition process, the server first acquires the image to be recognized; wherein the image to be recognized is at least one image frame of the target video; then, the server determines multiple feature vectors in the image to be recognized and classifies the multiple feature vectors into a first number of image scene categories; wherein, the feature vectors are used to characterize the pixel features of each pixel in the image to be recognized, and the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes; then, the server determines the probability value of the image to be recognized belonging to each image scene category based on the number of feature vectors in each feature category; finally, the server selects a second number of target image scene categories from the first number of image scene categories based on the probability values ​​of each image scene category, and identifies the target video as an indoor scene or an outdoor scene based on the second number of target image scene categories. In this way, on the one hand, the feature vectors of the target video are first classified, and then the scene recognition of the target video is performed based on the probability values ​​of the classified feature vectors belonging to each image scene category. This optimizes the process of scene recognition of the video, enhances the generalizability of the application scenarios of scene recognition of video, and reduces the consumption of manpower and time costs. On the other hand, by using the multiple target image scene categories with the highest probability of the image to be recognized to identify whether the target video belongs to an indoor scene or an outdoor scene, the accuracy and robustness of scene recognition of the target video can be improved, which is conducive to subsequent video processing based on the accurately classified target video.

[0118] Those skilled in the art will understand that the methods disclosed in the above-described specific embodiments can be implemented in more specific ways. For example, the embodiments described above for determining multiple feature vectors in an image to be identified and classifying the multiple feature vectors of the image to be identified into a first number of image scene categories are merely illustrative.

[0119] For example, the server determines the probability that the image to be identified belongs to each image scene category based on the number of feature vectors in each feature category; or the server identifies the target video as an indoor scene or an outdoor scene based on a second number of target image scene categories, etc. This is just one way of setting up the categories. In actual implementation, there may be other ways of dividing the categories. For example, the first number of image scene categories and the second number of target image scene categories may be combined or set into another system, or some features may be ignored or not executed.

[0120] In one exemplary embodiment, see Figure 3 , Figure 3This is a flowchart illustrating an embodiment of classifying feature vectors of an image to be recognized in this application. In step S12, the server determines each feature vector in the image to be recognized and classifies the feature vectors of the image to be recognized into a first number of image scene categories. This process can be implemented in the following way:

[0121] Step S121: Based on a pre-trained convolutional neural network, extract multiple feature vectors corresponding to multiple pixels from the image to be identified.

[0122] In some embodiments, the convolutional neural network pre-trained by the server is a residual convolutional neural network, which identifies the deep feature map (i.e., the feature map obtained after convolving the image and the filter) on each pixel of the image to be identified through multiple convolutional layers (which can be represented by Conv_x), and then performs pooling processing on the deep feature map through pooling layers (which can be represented by pool_cr), thereby extracting the one-dimensional feature vector corresponding to each pixel in the image to be identified.

[0123] Specifically, as described in steps 1 to 5 below, the pre-trained convolutional neural network can extract the feature vectors of each pixel in the image to be identified using the following method.

[0124] Step 1: The server first inputs the image data corresponding to the image to be recognized into the first convolutional layer (Conv_1) to perform the first convolution operation, and obtains the image data corresponding to the image to be recognized after the first convolution operation.

[0125] The first convolution operation includes sequentially performing downsampling, max pooling, residual convolution, and unit convolution on the image data corresponding to the image to be recognized.

[0126] The system consists of five images to be identified. The first convolutional layer comprises a 7x7 convolutional kernel, a stride of 2 for downsampling, three Block 1 layers, one Block 2 layer, and a 2x2 Max pooling layer. In the first convolutional operation, downsampling is achieved through the stride of Conv_1. Taking an image with pixel dimensions of (224, 224) as an example, the feature size after the first convolutional downsampling layer is 109x109. After the Max pooling layer, a feature map of size 53x53 is obtained, increasing the number of channels from 3 to 64. Then, three Block 1 residual convolutions and one Block 2 residual convolution increase the dimension from 64 to 128. The image data after the first convolutional operation is represented by a vector (N, 128, H, W), where N = 5, C1 = 128, H = 224, and W = 224.

[0127] The values ​​of the above data (such as the number of images to be recognized, the number of convolutional layers, pixel size, and number of channels) are for illustrative purposes only and can be set to other values ​​according to actual application requirements.

[0128] The residual convolution processing includes three sets of first convolutional blocks sequentially arranged in the first convolutional layer, and the unit convolution processing includes a set of second convolutional blocks arranged after the three sets of first convolutional blocks. Each of the three sets of first convolutional blocks encapsulates a batch normalization function and a ReLU activation function, and includes a shot cut channel to form residual blocks within the first convolutional blocks. These residual blocks are used to perform local feature fusion on the image data to extract local texture information. The set of second convolutional blocks is used to perform dimension reduction, non-linear activation, and cross-channel information interaction on the image data after local feature fusion, thereby increasing the number of data channels and maintaining the pixel size of the image data after local feature fusion.

[0129] Step 2: The server sequentially inputs the image data after the first convolution operation into the second and third convolution layers to perform the second convolution operation twice, thereby obtaining the image data corresponding to the two second convolution operations of the image to be recognized.

[0130] The second convolution operation includes channel up-processing and size down-processing on the image data after the first convolution operation.

[0131] In this process, the image data after each second convolution operation is represented by the vector (N, 2×C1, H / 2, W / 2), that is, N=5, C2=256, H=112, W=112.

[0132] Step 3: The server inputs the image data after the two second convolution operations into the fourth convolution layer to perform the third convolution operation, resulting in image data after the third convolution operation for 5 images to be recognized.

[0133] The third convolution operation involves performing global average pooling on the image data after the second convolution operation. Global average pooling is used to scale the global features of the image data. The image data after the third convolution operation is represented by the vector (N, 1024, 1, 1), i.e., N = 5, C3 = 1024, H = 1, W = 1.

[0134] The values ​​of the above data (such as the number of images to be identified, the vector of image data, etc.) are for illustrative purposes only and can be set to other values ​​according to actual application requirements.

[0135] Step S122: Match the feature vector of each pixel with the first number of reference vectors, and classify the feature vector of each pixel into the image scene category that meets the matching degree threshold.

[0136] In one embodiment, each reference vector corresponds to an image scene category.

[0137] In one embodiment, the server can classify the one-dimensional feature vectors of the image to be recognized into a preset first number of image scene categories through a fully connected layer (which can be represented by Fc) connected to the last convolutional layer of a pre-trained convolutional neural network. The fully connected layer is used to match the feature vectors of each pixel with the first number of reference vectors and output the classification result of the feature vectors of each pixel.

[0138] As an example, as described in step 6 below, the fully connected layer can classify the feature vectors of each pixel into the corresponding image scene category using the following method.

[0139] Step 6: The server inputs the image data after the third convolution operation into the fully connected layer to perform the fourth convolution operation, thereby obtaining the image data corresponding to the fourth convolution operation of the image to be recognized.

[0140] The fourth convolution operation involves classifying the image data after the third convolution operation into 257 preset feature categories. The number of images to be identified is 5, and the image data after the fourth convolution operation is represented by the vector (N, 257, 1, 1), i.e., N = 5, C4 = 257, H = 1, W = 1. The number of feature categories is illustrative and can be set to other values ​​according to actual application requirements.

[0141] In one exemplary embodiment, see Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of determining the probability values ​​of an image to be identified belonging to various image scene categories in this application. In step S13, the server determines the probability values ​​of the image to be identified belonging to a first number of image scene categories based on the number of feature vectors in each feature category. This process can be implemented in the following way:

[0142] Step S131: In each image scene category, determine the exponential value of the image scene category with respect to the number of feature vectors in the image scene category.

[0143] In one embodiment, the exponent value is used to characterize the base value of the natural logarithm.

[0144] Step S132: Determine the exponent values ​​and values ​​of each image scene category with respect to the exponent values.

[0145] Step S133: Based on the quotient between the index value corresponding to each image scene category and the index value and the sum value, determine the probability value of the image to be identified belonging to each image scene category.

[0146] In one embodiment, the server uses the Soft max function to normalize the image data after the fourth convolution operation to obtain the probability values ​​of each image to be identified corresponding to a first number of preset scene categories.

[0147] As an example, the Soft max function maps the real-valued domain (i.e., the exponential values ​​of each feature category) of the output of a linear model (i.e., a convolutional neural network) to the effective real-valued space [0, 1] representing the probability distribution. The Soft max function can be characterized by the following expression:

[0148]

[0149] Among them, e Zi That is, the index value of each feature category i; This is the sum of the exponential values ​​of each feature category i.

[0150] In this context, the probability values ​​of each image to be identified corresponding to a first number of preset scene categories are distributed between (0, 1), and the sum of the probability values ​​of the first number of scene categories is equal to 1.

[0151] In one embodiment, the scene category to which the image to be identified belongs includes two main categories: indoor and outdoor. The indoor category includes various specific indoor subcategories (i.e., preset scene categories), and the outdoor category includes various specific outdoor subcategories (i.e., preset scene categories). Therefore, each preset scene category in the above embodiment belongs to either the indoor or outdoor category.

[0152] In one embodiment, the first scene recognition result is the probability value of each image to be recognized corresponding to a first number of preset scene categories.

[0153] In one exemplary embodiment, see Figure 5 , Figure 5 This is a schematic flowchart illustrating the first embodiment of identifying a target video as an indoor or outdoor scene in this application. In step S14, the server identifies the target video as an indoor or outdoor scene based on a second number of target image scene categories. This process can be implemented in the following ways:

[0154] Step S141: In the second number of target image scene categories, determine the first proportion of each target image scene category belonging to indoor scenes, and determine the second proportion of each target image scene category belonging to outdoor scenes.

[0155] As an example, the server selects a total of 10 target image scene categories for the second number of times. Among them, there are 3 target image scene categories belonging to indoor scenes, so the first number accounts for 3 / 10; there are 7 target image scene categories belonging to outdoor scenes, so the second number accounts for 7 / 10.

[0156] Step S142: Based on the relationship between the first quantity ratio and the second quantity ratio, identify whether the target video is an indoor scene or an outdoor scene.

[0157] In some embodiments, the server identifies the target video as an indoor or outdoor scene based on the relationship between a first quantity ratio and a second quantity ratio, including one of the following three possibilities:

[0158] The first possibility is that, in response to the first quantity percentage being greater than the second quantity percentage, the target video is identified as an indoor scene. The second possibility is that, in response to the first quantity percentage being less than the second quantity percentage, the target video is identified as an outdoor scene. The third possibility is that, in response to the first quantity percentage being equal to the second quantity percentage, the step of acquiring the image to be identified from the target video is repeated until the first quantity percentage of the target image scene category corresponding to the new image to be identified is greater than the second quantity percentage, or the first quantity percentage is less than the second quantity percentage.

[0159] In one exemplary embodiment, see Figure 6 , Figure 6 This is a schematic flowchart illustrating the second embodiment of identifying a target video as an indoor or outdoor scene in this application. In step S14, the process by which the server identifies the target video as an indoor or outdoor scene based on a second number of target image scene categories can be implemented in the following way:

[0160] Step S143: In the second number of target image scene categories, determine the probability values ​​and sums of each target image scene category belonging to indoor scenes as the first probability values ​​and sums, and determine the probability values ​​and sums of each target image scene category belonging to outdoor scenes as the second probability values ​​and sums.

[0161] As an example, the server selects a second number of target image scene categories, totaling 10. Among them, there are 7 target image scene categories belonging to indoor scenes, with probability values ​​of A1 = 0.01, A2 = 0.34, A3 = 0.12, A4 = 0.01, A5 = 0.14, A6 = 0.02, and A7 = 0.01, respectively. The sum of the first probability values ​​is 0.65. There are 3 target image scene categories belonging to outdoor scenes, with probability values ​​of A1 = 0.08, A2 = 0.24, and A3 = 0.03, respectively. The sum of the second probability values ​​is 0.35.

[0162] The values ​​of the above data (such as the number of image scene categories, probability values, probability values ​​and values, etc.) are for illustrative purposes only and can be set to other values ​​according to actual application needs.

[0163] Step S144: Based on the relationship between the sum of the first probability value and the sum of the second probability value, identify whether the target video is an indoor scene or an outdoor scene.

[0164] In some embodiments, the server identifies the target video as an indoor or outdoor scene based on the magnitude relationship between the sum of a first probability value and the sum of a second probability value, including one of the following three possibilities:

[0165] The first possibility is that, in response to the sum of the first probability values ​​being greater than the sum of the second probability values, the target video is identified as an indoor scene. The second possibility is that, in response to the sum of the first probability values ​​being less than the sum of the second probability values, the target video is identified as an outdoor scene. The third possibility is that, in response to the sum of the first probability values ​​being equal to the sum of the second probability values, the step of acquiring the image to be identified from the target video is re-executed until the sum of the first probability values ​​for the scene category corresponding to the new image to be identified is greater than the sum of the second probability values, or the sum of the first probability values ​​is less than the sum of the second probability values.

[0166] In one embodiment, the number of images to be identified is no less than two. That is, the server extracts more than one single-frame image from the image frames of the target video.

[0167] In one embodiment, the server acquires the image to be identified by: extracting frames from the target video based on a preset frame interval to obtain at least two images to be identified.

[0168] In one embodiment, after acquiring the image to be identified, the server further includes: adjusting the pixel size of at least two of the acquired images to be identified.

[0169] As an example, after the server extracts five frames from a short live video clip to obtain five images for recognition, these images are scaled using OpenCV according to the h / w ratio to (224, 224), meaning each side of the image is 224 pixels. For sides smaller than 224, padding is added with 0 pixels to bring the total to 224. Finally, the server stitches these five images together, and the stitched image data is represented by a vector (N, C, H, W). Here, N represents the number of images to be recognized (N = 5); C represents the number of data channels in the image data (C = 3); and H and W represent the width and height pixel values ​​of the image data (H = 224, W = 224).

[0170] The values ​​of the above data (such as the number of images to be identified, pixel ratio, and vector of image data) are for illustrative purposes only and can be set to other values ​​according to actual application requirements.

[0171] In one embodiment, the server selects a second number of target image scene categories from a first number of image scene categories, including: selecting the second number of target image scene categories with the highest probability values ​​from the probability values ​​of the first number of image scene categories to which each image to be identified belongs.

[0172] In one exemplary embodiment, see Figure 7 , Figure 7 This is a schematic flowchart illustrating the third embodiment of identifying a target video as an indoor or outdoor scene in this application. In step S14, the process by which the server identifies the target video as an indoor or outdoor scene based on a second number of target image scene categories can be implemented in the following way:

[0173] Step S145: Among the second number of target image scene categories corresponding to each image to be identified, determine the first total number proportion and the first total probability value of the target image scene category belonging to the indoor scene, and the second total number proportion and the second total probability value of the target image scene category belonging to the outdoor scene.

[0174] As an example, there are 5 images to be identified. The server selects 10 target image scene categories for each image. Specifically, for the first image, the percentage of the first category belonging to indoor scenes is 3 / 10, and the percentage of the second category belonging to outdoor scenes is 7 / 10; for the second image, the percentage of the first category belonging to indoor scenes is 1 / 10, and the percentage of the second category belonging to outdoor scenes is 9 / 10; for the third image, the percentage of the first category belonging to indoor scenes is 5 / 10, and the percentage of the second category belonging to outdoor scenes is 5 / 10; for the fourth image, the percentage of the first category belonging to indoor scenes is 3 / 10, and the percentage of the second category belonging to outdoor scenes is 7 / 10; and for the fifth image, the percentage of the first category belonging to indoor scenes is 8 / 10, and the percentage of the second category belonging to outdoor scenes is 2 / 10. From the above, we can conclude that the first total quantity percentage is 3 / 10 + 1 / 10 + 5 / 10 + 3 / 10 + 8 / 10 = 2; the second total quantity percentage is 7 / 10 + 9 / 10 + 5 / 10 + 7 / 10 + 2 / 10 = 3.

[0175] As another example, there are 5 images to be identified. The server selects 10 target image scene categories for each image to be identified. The first image to be identified has a first probability value of 0.3 for each target image scene category belonging to an indoor scene and a second probability value of 0.7 for each target image scene category belonging to an outdoor scene. The second image to be identified has a first probability value of 0.1 for each target image scene category belonging to an indoor scene and a second probability value of 0.9 for each target image scene category belonging to an outdoor scene. The third image to be identified has a first probability value of 0.5 for each target image scene category belonging to an indoor scene and a second probability value of 0.5 for each target image scene category belonging to an outdoor scene. The fourth image to be identified has a first probability value of 0.3 for each target image scene category belonging to an indoor scene and a second probability value of 0.7 for each target image scene category belonging to an outdoor scene. The fifth image to be identified has a first probability value of 0.8 for each target image scene category belonging to an indoor scene and a second probability value of 0.2 for each target image scene category belonging to an outdoor scene. From the above, we can conclude that the sum of the first total probability values ​​is 0.3 + 0.1 + 0.5 + 0.3 + 0.8 = 2; and the sum of the second total probability values ​​is 0.7 + 0.9 + 0.5 + 0.7 + 0.2 = 3.

[0176] The values ​​of the above data (such as the number of images to be identified, the number of image scene categories, and the percentage of each category) are for illustrative purposes only and can be set to other values ​​according to actual application needs.

[0177] Step S146: Based on the relationship between the first total quantity ratio and the second total quantity ratio, and / or the relationship between the first total probability value and the second total probability value, identify whether the target video is an indoor scene or an outdoor scene.

[0178] The server identifies the target video as an indoor scene or an outdoor scene based on the relationship between the first total quantity ratio and the second total quantity ratio, and / or the relationship between the first total probability value and the second total probability value. This is similar to steps S142 and S144 in the above embodiments, or it is a combination of steps S142 and S144. No specific limitation is made here.

[0179] In some embodiments, such as Figure 8 As shown, a training method for a convolutional neural network is provided, which can be applied to... Figure 1 Taking server 104 as an example, the method includes the following steps:

[0180] Step S21: Obtain a first training image that includes a first number of image scene categories.

[0181] In one embodiment, the first training image is a scene image extracted from an open-source dataset.

[0182] In one embodiment, the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes.

[0183] As an example, the server extracts 500,000 images containing real-world scenes from an open-source database. These 500,000 images are then categorized by the server into 257 predefined image scene categories to obtain the first training images comprising these 257 categories. The number of image scene categories is illustrative and can be set to other values ​​based on actual application requirements.

[0184] Step S22: Randomly scale the pixel aspect ratio of the first training image, and train the convolutional neural network model on the randomly scaled first training image to obtain the initial convolutional neural network after training.

[0185] In one embodiment, random scaling is used to adjust the pixel aspect ratio of the first training image under a randomly indicated display logic.

[0186] In one embodiment, the display ratio of the randomly scaled first training image in the display window is consistent with the randomly indicated display logic; wherein, the display logic includes landscape display logic and portrait display logic.

[0187] As an example, the server adjusts the pixel aspect ratio of the first training image according to the display ratio of landscape mode (16:4, 16:8, etc.) or portrait mode (8:16, 3:4, etc.). Then, the server trains a convolutional neural network model (e.g., a convolutional neural network based on CNN, RNN, etc.) on the randomly scaled first training image to obtain the initial convolutional neural network after training.

[0188] The purpose of random scaling is to simulate the switching between landscape and portrait modes during mobile live streaming, in order to improve the generalization ability of the subsequent training model.

[0189] Step S23: Obtain a second training image that includes a first number of image scene categories.

[0190] In one embodiment, the second training image is an image frame extracted from multiple target videos.

[0191] In one embodiment, the image scene categories corresponding to the multiple target videos include indoor scenes and outdoor scenes.

[0192] As an example, the server retrieves multiple target videos from a live streaming platform and extracts 5000 image frames from these videos using a video frame extraction method. These 5000 image frames are then categorized by the server into 257 preset image scene categories to obtain a second training image set comprising these 257 categories. The number of image scene categories is merely illustrative and can be set to other values ​​based on actual application requirements.

[0193] Step S24: Adjust the initial convolutional neural network based on the second training image to obtain a pre-trained convolutional neural network.

[0194] In one embodiment, a pre-trained convolutional neural network is used to perform the step of determining multiple feature vectors in the video scene recognition method described above, and the specific implementation will not be repeated here.

[0195] To more clearly illustrate the video scene recognition method provided in this disclosure, a specific embodiment will be used to describe the video scene recognition method in detail below. In an exemplary embodiment, reference is made to... Figure 9 , Figure 9 The flowchart illustrates a video scene recognition method according to another exemplary embodiment. This video scene recognition method is used in server 104 and specifically includes the following:

[0196] Step S31: Extract a 10-second live stream segment from a live stream room on a live streaming platform.

[0197] Step S32: Extract frames from the live stream segment at 2-second intervals to obtain 5 live stream images.

[0198] Step S33: Scale the width and height pixel values ​​of the 5 live images proportionally to obtain 5 images to be recognized, each with a pixel value ratio of 224×224.

[0199] The scaling process includes pixel padding of the width and height of the live image to make the corresponding attribute pixel size reach 224 pixels.

[0200] The first scene recognition result is the probability value of each image to be recognized corresponding to 257 preset scene categories. The number of image scene categories, pixel size, pixel value ratio, number of live images, etc. are only illustrative examples and can be set to other values ​​according to actual application needs.

[0201] Each preset scene category belongs to either the indoor or outdoor category.

[0202] Step S34: Stitch together the 5 images to be recognized, and input the image data corresponding to the 5 stitched images into a preset convolutional neural network for scene recognition to obtain the first scene recognition result for each image to be recognized.

[0203] The image data input into the convolutional neural network is represented by a vector (N, C, H, W).

[0204] Wherein, N represents the number of images to be identified corresponding to the image data, i.e., N=5; C represents the number of data channels of the image data to be identified corresponding to the image data, i.e., C=3 (Note: The images in the embodiments of this application are all RGB images. If the application conditions require, grayscale images with a data channel number of C=2 can also be used); H and W represent the width pixel value and height pixel value of the image data to be identified corresponding to the image data, i.e., H=224 and W=224.

[0205] In one exemplary embodiment, see Figure 10 , Figure 10 This is a flowchart illustrating an embodiment of scene recognition of an image to be recognized in this application. In step S34, the server inputs the image data corresponding to the five stitched images to be recognized into a preset convolutional neural network for scene recognition to obtain the first scene recognition result for each image to be recognized. This process can be specifically implemented in the following way:

[0206] Step S341: Input the image data corresponding to the five images to be identified after stitching into the first convolutional layer to perform the first convolution operation, and obtain the image data corresponding to the five images to be identified after the first convolution operation.

[0207] The first convolution operation includes performing downsampling, max pooling, residual convolution, and unit convolution on the image data corresponding to the five images to be recognized in sequence.

[0208] The residual convolution processing includes three sets of first convolution blocks arranged sequentially in the first convolution layer, and the unit convolution processing includes a second convolution block arranged after the three sets of first convolution blocks.

[0209] The first convolutional block of each of the three groups encapsulates a normalization function (Batch Normal) and an activation function (ReLU), as well as a shortcut channel (shot cut) to form a residual block in the first convolutional block. The residual block is used to perform local feature fusion on the image data to extract local texture information from the image data.

[0210] In this group, the second convolutional block is used to perform dimensionality reduction, non-linear activation, and cross-channel information interaction on the image data after local feature fusion, so that the number of data channels of the image data after local feature fusion increases while the pixel size remains unchanged.

[0211] The image data after the first convolution operation is represented by the vector (N, 128, H, W), that is, N = 5, C1 = 128, H = 224, W = 224.

[0212] Step S342: The image data after the first convolution operation is sequentially input into the second and third convolution layers to perform the second convolution operation twice, resulting in image data after the two second convolution operations for 5 images to be recognized.

[0213] The second convolution operation includes channel up-processing and size down-processing on the image data after the first convolution operation.

[0214] In this process, the image data after each second convolution operation is represented by the vector (N, 2×C1, H / 2, W / 2), that is, N=5, C2=256, H=112, W=112.

[0215] Step S343: Input the image data after the two second convolution operations into the fourth convolution layer to perform the third convolution operation, and obtain the image data after the third convolution operation for the five images to be recognized.

[0216] The third convolution operation includes global average pooling of the image data after the second convolution operation.

[0217] Global average pooling is used to scale the global features of the image data.

[0218] The image data after the third convolution operation is represented by the vector (N, 1024, 1, 1), that is, N = 5, C3 = 1024, H = 1, W = 1.

[0219] Step S344: Input the image data after the third convolution operation into the fully connected layer for the fourth convolution operation to obtain the image data after the fourth convolution operation for the five images to be recognized.

[0220] The fourth convolution operation involves classifying the image data after the third convolution operation into 257 preset feature categories. The number of feature categories and the vector of the image data are merely illustrative and can be set to other values ​​according to actual application requirements.

[0221] The image data after the fourth convolution operation is represented by the vector (N, 257, 1, 1), that is, N = 5, C4 = 257, H = 1, W = 1.

[0222] Step S345: The image data after the fourth convolution operation is normalized using the Softmax function to obtain the probability values ​​of each of the 5 images to be recognized corresponding to 257 preset scene categories.

[0223] In this context, the probability values ​​of each image to be identified corresponding to 257 preset scene categories are distributed between (0,1), and the sum of the probability values ​​of the 257 scene categories is equal to 1.

[0224] Each preset scene category belongs to either the indoor or outdoor category.

[0225] The first scene recognition result is the probability value of each image to be recognized corresponding to one of 257 preset scene categories.

[0226] The Soft max function maps the real-valued output of a linear model (i.e., a convolutional neural network) to the effective real-valued space [0, 1] representing the probability distribution. The Soft max function can be represented by the following expression:

[0227]

[0228] Step S35: Among the probability values ​​of 257 preset scene categories for each image to be identified, determine the top ten scene categories with the highest probability values.

[0229] As an example, among the probability values ​​of the 257 preset scene categories corresponding to the image P1 to be identified, the top ten scene categories with the highest probability values ​​and their probability values ​​are as follows: (Scene A1, 0.09), (Scene A2, 0.19), (Scene A3, 0.41), (Scene A4, 0.22), (Scene A5, 0.36), (Scene A6, 0.89), (Scene A7, 0.69), (Scene A8, 0.41), (Scene A9, 0.01), (Scene A10, 0.05).

[0230] In some embodiments, the server determines the second scene recognition result of the five images to be recognized based on the top ten scene categories with the highest probability values ​​for each image to be recognized, thereby determining the scene where the live broadcast room is located. This can be done through one of the following two methods: steps S36a-S38a or steps S36b-S38b, or a combination of these two methods.

[0231] The values ​​mentioned above (such as probability values, the top ten scenario categories with the highest probability values, the number of scenario categories, etc.) are for illustrative purposes only and can be set to other values ​​according to actual application needs.

[0232] Step S36a: Among the scene categories with the ten highest probability values ​​for each image to be identified, determine the ratio of indoor category numbers and the ratio of outdoor category numbers for each image to be identified.

[0233] Among them, the indoor category ratio is the proportion of indoor scene categories among the ten most probable scene categories, and the outdoor category ratio is the proportion of outdoor scene categories among the ten most probable scene categories.

[0234] Step S37a: Determine the first average of the ratio of the number of images belonging to the indoor category among the 5 images to be identified and the second average of the ratio of the number of images belonging to the outdoor category among the 5 images to be identified.

[0235] As an example, among the scene categories with the ten highest probability values ​​for the image P1 to be identified, the indoor category has a ratio of 4 / 10 and 6 / 10; among the scene categories with the ten highest probability values ​​for the image P2 to be identified, the indoor category has a ratio of 8 / 10 and 2 / 10; among the scene categories with the ten highest probability values ​​for the image P3 to be identified, the indoor category has a ratio of 4 / 10 and 6 / 10; among the scene categories with the ten highest probability values ​​for the image P4 to be identified, the indoor category has a ratio of 5 / 10 and 5 / 10; among the scene categories with the ten highest probability values ​​for the image P5 to be identified, the indoor category has a ratio of 3 / 10 and 7 / 10. The first average of the ratio of indoor category counts among the 5 images to be identified is (4 / 10+8 / 10+4 / 10+5 / 10+3 / 10) / 5 = 0.48, and the second average of the ratio of outdoor category counts among the 5 images to be identified is (6 / 10+2 / 10+6 / 10+5 / 10+7 / 10) / 5 = 0.52.

[0236] The values ​​of the above data (such as average value, indoor category ratio, number of scene categories, etc.) are for illustrative purposes only and can be set to other values ​​according to actual application needs.

[0237] Step S38a: Based on the relationship between the first average value and the second average value of the five images to be identified, determine the second scene recognition result of the five images to be identified, thereby determining the scene where the live broadcast room is located.

[0238] Specifically, if the first average value is greater than the second average value, the scene recognition result of the 5 images to be recognized is an indoor scene; if the first average value is less than the second average value, the scene recognition result of the 5 images to be recognized is an outdoor scene; if the first average value is equal to the second average value, a new live broadcast segment is selected, and 5 new images to be recognized are extracted for scene recognition again, until the first average value is greater than the second average value, or the first average value is less than the second average value.

[0239] Step S36b: Among the scene categories with the ten highest probability values ​​for each image to be identified, determine the probability sum of the indoor category and the probability sum of the outdoor category for each image to be identified.

[0240] The probability sum of the indoor category is the sum of the probability values ​​of the scene categories that belong to indoor scenes among the ten scene categories with the highest probability values, and the probability sum of the outdoor category is the sum of the probability values ​​of the scene categories that belong to outdoor scenes among the ten scene categories with the highest probability values.

[0241] Step S37b: Determine the third average of the sum of probabilities of the five images to be identified belonging to the indoor category and the fourth average of the sum of probabilities of the five images to be identified belonging to the outdoor category.

[0242] Step S38b: Based on the relationship between the third average value and the fourth average value of the five images to be identified, determine the third scene recognition result of the five images to be identified, thereby determining the scene where the live broadcast room is located.

[0243] If the third average value is greater than the fourth average value, the scene recognition result of the five images to be recognized is an indoor scene; if the third average value is less than the fourth average value, the scene recognition result of the five images to be recognized is an outdoor scene; if the third average value is equal to the fourth average value, a new live broadcast segment is selected, and five new images to be recognized are extracted for scene recognition again, until the third average value is greater than the fourth average value, or the third average value is less than the fourth average value.

[0244] To more clearly illustrate the training process of the convolutional neural network provided in the embodiments of this disclosure, the training process of the convolutional neural network will be specifically described below with reference to a specific embodiment. In an exemplary embodiment, referring to... Figure 11 , Figure 11 The flowchart illustrates a training method for a convolutional neural network according to another exemplary embodiment. The training process of this convolutional neural network is used in server 104 and specifically includes the following:

[0245] Step S41: Extract the first training images of the 257 scene categories that have been classified from the database.

[0246] The number of image scene categories is for illustrative purposes only and can be set to other values ​​according to actual application needs.

[0247] Among the 257 scene categories, there is one indoor blank scene category and one outdoor blank scene category;

[0248] Among them, the indoor blank scene category includes training images of multiple indoor scene categories, and the training images of these multiple indoor scene categories do not belong to any of the remaining 256 scene categories; the outdoor blank scene category includes training images of multiple outdoor scene categories, and the training images of these multiple outdoor scene categories do not belong to any of the remaining 256 scene categories.

[0249] Step S42: Randomly scale the pixel ratio of the first training image for 257 scene categories to obtain the randomly scaled first training image.

[0250] The purpose of random scaling is to simulate the switching between landscape and portrait modes during mobile live streaming, in order to improve the generalization ability of the subsequent training model.

[0251] Step S43: Train the model on the randomly scaled first training image to obtain the trained first convolutional neural network.

[0252] Step S44: Extract the second training image from the database.

[0253] The second training images include live images of various indoor scene categories and live images of various outdoor scene categories.

[0254] Step S45: Based on the second training image, the first convolutional neural network is modified to obtain the modified second convolutional neural network, and the second convolutional neural network is used to perform scene recognition on the image to be recognized corresponding to the live broadcast segment.

[0255] The above scheme, on the one hand, first classifies the feature vectors of the target video, and then performs scene recognition on the target video based on the probability values ​​of the classified feature vectors belonging to each image scene category. This optimizes the process of scene recognition for the video, enhances the generalizability of the application scenarios for scene recognition of video, and reduces the consumption of manpower and time costs. On the other hand, by using the multiple target image scene categories with the highest probability of the image to be recognized to identify whether the target video belongs to an indoor or outdoor scene, it can improve the accuracy and robustness of scene recognition of the target video, thus facilitating subsequent video processing based on the accurately classified target video.

[0256] It should be understood that, although Figures 2-11The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2-11 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0257] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0258] Figure 12 This is a block diagram of a video scene recognition device provided in an embodiment of this application. (Refer to...) Figure 12 The scene recognition device 10 for the video includes: an image acquisition unit 11, a vector classification unit 12, a probability calculation unit 13, and a scene recognition unit 14.

[0259] The image acquisition unit 11 is configured to acquire an image to be identified, wherein the image to be identified is at least one image frame of the target video.

[0260] The vector classification unit 12 is configured to determine multiple feature vectors in the image to be identified and classify the multiple feature vectors into a first number of image scene categories; the feature vectors are used to characterize the pixel features of each pixel in the image to be identified, and the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes.

[0261] The probability calculation unit 13 is configured to determine the probability value of the image to be identified belonging to each of the image scene categories based on the number of feature vectors in each of the feature categories.

[0262] The scene recognition unit 14 is configured to perform a probability value based on each of the image scene categories, select a second number of target image scene categories from the first number of image scene categories, and identify the target video as the indoor scene or the outdoor scene based on the second number of target image scene categories.

[0263] In some embodiments, in identifying the target video as the indoor scene or the outdoor scene based on the second number of target image scene categories, the scene recognition unit 14 is specifically used for:

[0264] In the second number of target image scene categories, a first number proportion of each target image scene category belonging to the indoor scene is determined, and a second number proportion of each target image scene category belonging to the outdoor scene is determined;

[0265] Based on the relationship between the first quantity ratio and the second quantity ratio, the target video is identified as either the indoor scene or the outdoor scene.

[0266] In some embodiments, regarding the identification of the target video as either the indoor scene or the outdoor scene based on the size relationship between the first quantity ratio and the second quantity ratio, the scene identification unit 14 includes one of the following three items:

[0267] In response to the first quantity ratio being greater than the second quantity ratio, the target video is identified as the indoor scene;

[0268] In response to the first quantity ratio being less than the second quantity ratio, the target video is identified as the outdoor scene;

[0269] In response to the first quantity percentage being equal to the second quantity percentage, the step of acquiring the image to be identified from the target video is re-executed until the first quantity percentage of the target image scene category corresponding to the new image to be identified is greater than the second quantity percentage, or the first quantity percentage is less than the second quantity percentage.

[0270] In some embodiments, in identifying the target video as the indoor scene or the outdoor scene based on the second number of target image scene categories, the scene recognition unit 14 is specifically used for:

[0271] In the second number of target image scene categories, the probability values ​​and values ​​of each target image scene category belonging to the indoor scene are determined as the first probability values ​​and values, and the probability values ​​and values ​​of each target image scene category belonging to the outdoor scene are determined as the second probability values ​​and values.

[0272] Based on the relationship between the first probability value and the second probability value, the target video is identified as either the indoor scene or the outdoor scene.

[0273] In some embodiments, in identifying the target video as either the indoor scene or the outdoor scene based on the magnitude relationship between the first probability value and the second probability value, the scene recognition unit 14 includes one of the following three:

[0274] In response to the first probability value being greater than the second probability value, the target video is identified as the indoor scene;

[0275] In response to the first probability value being less than the second probability value, the target video is identified as the outdoor scene;

[0276] In response to the first probability value being equal to the second probability value, the step of acquiring the image to be identified from the target video is re-executed until the first probability value of the scene category of the new image to be identified is greater than the second probability value, or the first probability value is less than the second probability value.

[0277] In some embodiments, the number of images to be identified is not less than two; in terms of acquiring the images to be identified, the image acquisition unit 11 is further configured to:

[0278] Based on a preset frame interval, the target video is subjected to frame extraction to obtain at least two images to be identified;

[0279] In selecting the second number of target image scene categories with the highest probability values ​​from the first number of image scene categories, the scene recognition unit 14 is further configured to:

[0280] Among the probability values ​​of the first number of image scene categories to which each image to be identified belongs, the second number of target image scene categories with the highest corresponding probability values ​​are selected.

[0281] In some embodiments, in identifying the target video as the indoor scene or the outdoor scene based on the second number of target image scene categories, the scene recognition unit 14 is further configured to:

[0282] In the second number of target image scene categories corresponding to each of the images to be identified, a first total number proportion and a first total probability value of the target image scene category belonging to the indoor scene are determined, as well as a second total number proportion and a second total probability value of the target image scene category belonging to the outdoor scene.

[0283] Based on the relationship between the first total quantity ratio and the second total quantity ratio, and / or the relationship between the first total probability value and the second total probability value, the target video is identified as either the indoor scene or the outdoor scene.

[0284] In some embodiments, in determining multiple feature vectors in the image to be identified and classifying the multiple feature vectors into a first number of image scene categories, the vector classification unit 12 is further configured to:

[0285] Based on a pre-trained convolutional neural network, multiple feature vectors corresponding to multiple pixels are extracted from the image to be identified.

[0286] The feature vectors of each pixel are matched with the first number of reference vectors, and the feature vectors of each pixel are classified into image scene categories that meet the matching degree threshold; wherein each reference vector corresponds to a feature category.

[0287] In some embodiments, in determining the probability value of the image to be identified belonging to each of the image scene categories based on the number of feature vectors in each of the feature categories, the probability calculation unit 13 is further configured to:

[0288] In each of the image scene categories, based on the number of feature vectors in the image scene category, an exponential value of the image scene category with respect to the number is determined, where the exponential value represents the base of the natural logarithm;

[0289] Determine the exponential values ​​and values ​​of the exponential values ​​among the respective image scene categories;

[0290] Based on the quotient between the index value corresponding to each of the image scene categories and the index value, the probability value of the image to be identified belonging to each of the image scene categories is determined.

[0291] Figure 13 This is a block diagram of a training device for a convolutional neural network provided in an embodiment of this application. (Refer to...) Figure 13 The training device 10A for the convolutional neural network includes: a first acquisition unit 11A, a model training unit 12A, a second acquisition unit 13A, and a model correction unit 14A.

[0292] The first acquisition unit 11A is configured to acquire a first training image including a first number of image scene categories; the first training image is a scene image extracted from an open-source dataset, and the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes.

[0293] The model training unit 12A is configured to perform random scaling of the pixel aspect ratio of the first training image and train a convolutional neural network model on the randomly scaled first training image to obtain the trained initial convolutional neural network.

[0294] The second acquisition unit 13A is configured to acquire a second training image including the first number of image scene categories; the second training image is an image frame extracted from multiple target videos, and the image scene categories corresponding to the multiple target videos include the indoor scene and the outdoor scene.

[0295] The model correction unit 14A is configured to adjust the initial convolutional neural network based on the second training image to obtain a pre-trained convolutional neural network.

[0296] The pre-trained convolutional neural network is used to perform the step of determining multiple feature vectors in the video scene recognition method described above.

[0297] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0298] Figure 14 This is a block diagram of a server 20 provided in an embodiment of this application. For example, server 20 can be an electronic device, an electronic component, or a server array, etc. (Refer to...) Figure 14 Server 20 includes processor 21, which may be a processor set, including one or more processors. Server 20 also includes memory resources represented by memory 22, where computer programs, such as application programs, are stored. The computer programs stored in memory 22 may include one or more modules, each corresponding to a set of executable instructions. Furthermore, processor 21 is configured to implement, when executing the computer programs, scene recognition methods for video as described above, and / or convolutional neural network training methods.

[0299] In some embodiments, server 20 is an electronic device whose computing system can run one or more operating systems, including any of the operating systems discussed above and any commercially available server operating system. Server 20 can also run any of a variety of additional server applications and / or middleware applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, super servers, database servers, etc. Exemplary database servers include, but are not limited to, commercially available database servers from companies such as IBM.

[0300] In some embodiments, processor 21 typically controls the overall operation of server 20, such as operations associated with display, data processing, data communication, and recording operations. Processor 21 may include one or more processor components to execute computer programs to perform all or part of the steps of the methods described above. Furthermore, processor components may include one or more modules to facilitate interaction between processor components and other components. For example, processor components may include a multimedia module to facilitate control of the interaction between user server 20 and processor 21 using multimedia components.

[0301] In some embodiments, the processor component in processor 21 may also be referred to as a CPU (Central Processing Unit). The processor component may be an electronic chip with signal processing capabilities. The processor may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor component. Furthermore, the processor component may be implemented using integrated circuit chips.

[0302] In some embodiments, memory 22 is configured to store various types of data to support operation on server 20. Examples of such data include instructions for any application or method operating on server 20, acquired data, messages, images, videos, etc. Memory 22 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene storage.

[0303] In some embodiments, the memory 22 can be a memory module, TF card, etc., and can store all information in the server 20, including the input raw data, computer programs, intermediate running results, and final running results. In some embodiments, it stores and retrieves information according to the location specified by the processor. In some embodiments, the server 20 has a memory function and can ensure normal operation because of the memory 22. In some embodiments, the memory 22 of the server 20 can be classified into main memory (RAM) and auxiliary memory (external memory) according to its purpose, or it can be classified into external memory and internal memory. External memory is usually magnetic media or optical discs, which can store information for a long time. RAM refers to the storage component on the motherboard, which is used to store the currently executing data and programs, but it is only used to temporarily store programs and data. The data will be lost when the power is turned off or the power is cut off.

[0304] In some embodiments, server 20 may further include: a power supply component 23 configured to perform power management of server 20, a wired or wireless network interface 24 configured to connect server 20 to a network, and an input / output (I / O) interface 25. Server 20 may operate on an operating system stored in memory 22, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or similar.

[0305] In some embodiments, power supply component 23 provides power to various components of server 20. Power supply component 23 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to server 20.

[0306] In some embodiments, the wired or wireless network interface 24 is configured to facilitate wired or wireless communication between the server 20 and other devices. The server 20 may access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof.

[0307] In some embodiments, the wired or wireless network interface 24 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the wired or wireless network interface 24 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0308] In some embodiments, the input / output (I / O) interface 25 provides an interface between the processor 21 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, volume buttons, a power button, and a lock button.

[0309] Figure 15 This is a block diagram of a computer-readable storage medium 30 provided in an embodiment of this application. The computer-readable storage medium 30 stores a computer program 31, which, when executed by a processor, implements the video scene recognition method and / or convolutional neural network training method as described above.

[0310] If the integrated units of the various functional units in the various embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium 30. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer-readable storage medium 30 includes a computer program 31, which includes several instructions to cause a computer device (which may be a personal computer, system server, or network device, etc.), an electronic device (e.g., MP3, MP4, etc., or a mobile phone, tablet computer, wearable device, etc., or a desktop computer, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of this application.

[0311] Figure 16 This is a block diagram of a computer program product 40 provided in an embodiment of this application. The computer program product 40 includes program instructions 41, which can be executed by the processor of server 20 to implement the video scene recognition method and / or convolutional neural network training method as described above.

[0312] Those skilled in the art will understand that embodiments of this application may provide a video scene recognition method, a convolutional neural network training method, a video scene recognition device 10, a convolutional neural network training device 10A, a server 20, a computer-readable storage medium 30, or a computer program product 40. Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product 40 embodied on one or more computer program instructions 41 (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0313] This application is described with reference to flowchart illustrations and / or block diagrams of a video scene recognition method, a convolutional neural network training method, a video scene recognition device 10, a convolutional neural network training device 10A, a server 20, a computer-readable storage medium 30, or a computer program product 40 according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by the computer program product 40. These computer program products 40 can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that program instructions 41, executable by the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the scene recognition method in the flowchart illustration. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0314] These computer program products 40 may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that program instructions 41 stored in the computer program product 40 produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0315] These program instructions 41 may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing the program instructions 41 that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0316] It should be noted that the various methods, apparatuses, electronic devices, computer-readable storage media, computer program products, etc. described above may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0317] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0318] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A scene recognition method for video, characterized in that, The method includes: Acquire an image to be identified, wherein the image to be identified is at least one image frame of the target video; Multiple feature vectors in the image to be identified are determined, and the multiple feature vectors are classified into a first number of image scene categories; the feature vectors are used to characterize the pixel features of each pixel in the image to be identified, and the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes; Based on the number of feature vectors in each of the image scene categories, the probability value of the image to be identified belonging to each of the image scene categories is determined; Based on the probability values ​​of each of the image scene categories, a second number of target image scene categories are selected from the first number of image scene categories, and the target video is identified as either an indoor scene or an outdoor scene based on the second number of target image scene categories; this includes: determining a first proportion of each target image scene category belonging to the indoor scene and a second proportion of each target image scene category belonging to the outdoor scene in the second number of target image scene categories; identifying the target video as either an indoor scene or an outdoor scene based on the relationship between the first proportion and the second proportion; or determining the sum of the probability values ​​of each target image scene category belonging to the indoor scene as a first probability value and the sum of the probability values ​​of each target image scene category belonging to the outdoor scene as a second probability value and the sum of the probability values ​​of each target image scene category belonging to the outdoor scene in the second number of target image scene categories; identifying the target video as either an indoor scene or an outdoor scene based on the relationship between the first probability value and the second probability value.

2. The method according to claim 1, characterized in that, The step of identifying the target video as either the indoor scene or the outdoor scene based on the size relationship between the first quantity ratio and the second quantity ratio includes one of the following three items: In response to the first quantity ratio being greater than the second quantity ratio, the target video is identified as the indoor scene; In response to the first quantity ratio being less than the second quantity ratio, the target video is identified as the outdoor scene; In response to the first quantity percentage being equal to the second quantity percentage, the step of acquiring the image to be identified from the target video is re-executed until the first quantity percentage of the target image scene category corresponding to the new image to be identified is greater than the second quantity percentage, or the first quantity percentage is less than the second quantity percentage.

3. The method according to claim 1, characterized in that, The step of identifying the target video as either the indoor scene or the outdoor scene based on the magnitude relationship between the first probability value and the second probability value includes one of the following three items: In response to the first probability value being greater than the second probability value, the target video is identified as the indoor scene; In response to the first probability value being less than the second probability value, the target video is identified as the outdoor scene; In response to the first probability value being equal to the second probability value, the step of acquiring the image to be identified from the target video is re-executed until the first probability value of the scene category of the new image to be identified is greater than the second probability value, or the first probability value is less than the second probability value.

4. The method according to claim 1, characterized in that, The number of images to be identified is no less than two; the acquisition of the images to be identified includes: Based on a preset frame interval, the target video is subjected to frame extraction to obtain at least two images to be identified; The step of selecting a second number of target image scene categories from the first number of image scene categories includes: Among the probability values ​​of the first number of image scene categories to which each image to be identified belongs, the second number of target image scene categories with the highest corresponding probability values ​​are selected.

5. The method according to claim 4, characterized in that, The step of identifying the target video as either the indoor scene or the outdoor scene based on the second number of target image scene categories includes: In the second number of target image scene categories corresponding to each of the images to be identified, a first total number proportion and a first total probability value of the target image scene category belonging to the indoor scene are determined, as well as a second total number proportion and a second total probability value of the target image scene category belonging to the outdoor scene. Based on the relationship between the first total quantity ratio and the second total quantity ratio, and / or the relationship between the first total probability value and the second total probability value, the target video is identified as either the indoor scene or the outdoor scene.

6. The method according to claim 1, characterized in that, The step of determining multiple feature vectors in the image to be identified and classifying the multiple feature vectors into a first number of image scene categories includes: Based on a pre-trained convolutional neural network, multiple feature vectors corresponding to multiple pixels are extracted from the image to be identified. The feature vectors of each pixel are matched with the first number of reference vectors, and the feature vectors of each pixel are classified into image scene categories that meet the matching degree threshold; wherein each reference vector corresponds to an image scene category.

7. The method according to claim 1, characterized in that, The step of determining the probability value of the image to be identified belonging to each of the image scene categories based on the number of feature vectors in each of the image scene categories includes: In each of the image scene categories, based on the number of feature vectors in the image scene category, an exponential value of the image scene category with respect to the number is determined, where the exponential value represents the base of the natural logarithm; Determine the exponential values ​​and values ​​of the exponential values ​​among the respective image scene categories; Based on the quotient between the index value corresponding to each of the image scene categories and the index value, the probability value of the image to be identified belonging to each of the image scene categories is determined.

8. A method for training a convolutional neural network, characterized in that, The method includes: Obtain a first training image comprising a first number of image scene categories; the first training image is a scene image extracted from an open-source dataset, and the image scene categories include multiple image scene categories corresponding to indoor scenes and multiple image scene categories corresponding to outdoor scenes; The pixel aspect ratio of the first training image is randomly scaled, and the first training image after random scaling is used to train a convolutional neural network model to obtain the initial convolutional neural network after training. Acquire a second training image that includes the first number of image scene categories; the second training image is an image frame extracted from multiple target videos, and the image scene categories corresponding to the multiple target videos include the indoor scene and the outdoor scene; The initial convolutional neural network is adjusted based on the second training image to obtain a pre-trained convolutional neural network; wherein the pre-trained convolutional neural network is used to perform the determination of multiple feature vectors in the image to be identified as described in any one of claims 1-7.

9. A server, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the executable instructions to implement the video scene recognition method as described in any one of claims 1 to 7, and / or the convolutional neural network training method as described in claim 8.

10. A computer-readable storage medium comprising program data, characterized in that, When the program data is executed by the server's processor, the server is able to execute the scene recognition method for the video as described in any one of claims 1 to 7, and / or the convolutional neural network training method as described in claim 8.

Images