Scene category determination method, electronic device and storage medium

Abstract

The present application discloses a scene category determination method, electronic device, and storage medium. The scene category determination method includes: obtaining at least one current image frame and initial classification requirements corresponding to several scene categories, at least some of the scene categories corresponding to different image processing methods; based on the target scene category to which several frames of historical images belong, updating at least some of the initial classification requirements to obtain new classification requirements corresponding to each scene category; and determining the final scene category of at least one current image frame according to the new classification requirements corresponding to each scene category. The above scheme can improve the accuracy of the determined scene category.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a scene category determination method, electronic device and storage medium. Background Art

[0002] In recent years, scene recognition is an important branch in the field of image processing technology. Image acquisition devices usually perform different image processing functions in different image acquisition scenarios. For example, an image acquisition device includes an image sensor and other functional modules other than the image sensor. Other functional modules may be related modules that can perform image processing on the acquired image data. Specifically, other functional modules may be image signal processors (ISPs). However, parameter debugging and picture style adjustment of image signal processors in complex scenes usually rely on pre-identification of image acquisition scenes in image data, and selection of appropriate algorithms and parameters according to different image acquisition scenes. The computer vision industry needs to make different adjustments to the acquired images according to different image acquisition scene categories of the image acquisition device. For example, when a group of animals pass by the lens of the image acquisition device, the image scene may be identified as a special scene because the animals cover a large area of ​​the image, but the animals quickly walk out of the screen and will be identified as a normal scene. The image signal processor switches back and forth between turning on and off the image processing function corresponding to the special scene, which may cause the image acquisition device to use the wrong image processing function to process the acquired image, which may cause abnormal image processing results. It can be seen that accurate pre-identification of the image acquisition scene is particularly important for the activation of subsequent image processing related functions.

[0003] In view of the existing technical defects, how to provide a more accurate solution for image scene determination is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0004] The present application at least provides a scene category determination method, an electronic device, and a storage medium.

[0005] The present application provides a method for determining a scene category, including: obtaining at least one frame of current image and initial classification requirements corresponding to several scene categories, wherein at least some of the scene categories correspond to different image processing methods; based on the target scene categories to which several frames of historical images belong, updating at least some of the initial classification requirements to obtain new classification requirements corresponding to each scene category; and determining the final scene category of at least one frame of current image according to the new classification requirements corresponding to each scene category.

[0006] The present application provides a scene category determination device, including: an acquisition module, an update module and a determination module; the acquisition module is used to acquire at least one frame of current image and initial classification requirements corresponding to several scene categories, and at least some of the scene categories have different image processing methods; the update module is used to update at least some of the initial classification requirements based on the target scene categories to which several frames of historical images belong, and obtain new classification requirements corresponding to each scene category; the determination module is used to determine the final scene category of at least one frame of current image according to the new classification requirements corresponding to each scene category.

[0007] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned scene category determination method.

[0008] The present application provides a computer-readable storage medium having program instructions stored thereon, and the program instructions implement the above-mentioned scene category determination method when executed by a processor.

[0009] The above scheme updates at least part of the initial classification requirements based on the target scene categories to which several frames of historical images belong, and obtains new classification requirements corresponding to each scene category, so that when the scene category of the current image is determined, the new classification requirements are highly adapted to each scene category, and then the final scene category of at least one frame of the current image is determined according to the new classification requirements corresponding to each scene category. The final scene category of each current image can be determined by using classification requirements with a higher degree of adaptation, thereby improving the accuracy of the final scene category determined for each current image.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0012] Figure 1 This is a flow chart of an embodiment of the method for determining the scene category of the present application. Figure 1 ;

[0013] Figure 2 This is a flow chart of an embodiment of the method for determining the scene category of the present application. Figure 2 ;

[0014] Figure 3 This is a flow chart of an embodiment of the method for determining the scene category of the present application. Figure 3 ;

[0015] Figure 4This is a flow chart of an embodiment of the method for determining the scene category of the present application. Figure 4 ;

[0016] Figure 5 It is a structural diagram of an embodiment of a device for determining a scene category of the present application;

[0017] Figure 6 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0018] Figure 7 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0019] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0020] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0021] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0022] The present application provides some scene category determination methods and scene category determination devices. The application scenarios of the scene category determination method include but are not limited to intelligent image acquisition, scene recognition of images, image processing of images by image acquisition devices, etc. The executor of the scene category determination method may be a scene category determination device. For example, the scene category determination device may be set in a terminal device or a server or other processing device, wherein the terminal device may be a device for scene category determination, a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, etc. In some possible implementations, the scene category determination method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0023] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the method for determining the scene category of the present application. Figure 1 Specifically, the scene category determination method may include the following steps:

[0024] Step S11: obtaining at least one current image frame and initial classification requirements corresponding to several scene categories.

[0025] The current image may be an image that needs to be processed. The at least one frame of the current image may be one frame of the current image or multiple frames of the current image. Specifically, the current image may be at least one frame of the image that is captured by the image capture device within the current time period and used for image processing.

[0026] Several scene categories may be one scene category or multiple scene categories. Scene categories may refer to the classification of the environment and background contained in the image during the image recognition process. Several scene categories may be scene categories that can be set by user, or they may be various preset scene categories. Specifically, scene categories may be preset scene categories that each frame of image may be classified into. In some application scenarios, scene categories may be preset scene categories used to indicate the belonging of an image. Specifically, several scene categories may be natural landscapes (such as forests, mountains, beaches) and urban landscapes (such as streets, buildings, squares) belonging to outdoor scenes, or homes (such as living rooms, kitchens, bedrooms) and businesses (such as shops, offices, restaurants) belonging to indoor landscapes, or roads (such as highways, city streets) and weather (such as fog, rain, snow, clear) belonging to traffic scenes, etc. Exemplarily, several scene categories may be foggy scenes, rainy scenes, and backlit scenes, respectively. It is understandable that several scene categories are only exemplified here, and are not limited to the specific scene categories of several scene categories when several scene categories are preset scene categories.

[0027] The initial classification requirement may be a preset classification requirement. The classification requirement may be an evaluation index for indicating whether an image belongs to each scene category. Several scene categories may correspond to the classification requirements of each scene category respectively. When the image to be classified meets the classification requirement corresponding to any one of the several scene categories, any one of the several scene categories is determined as the scene category to which the image to be classified belongs. The image to be classified may be a current image or a historical image. Specifically, the classification requirement may be an evaluation index of whether the image features of the image meet each scene category. When the prediction confidence that the image to be classified belongs to a certain scene category reaches a threshold confidence during the classification decision process, the certain scene category is determined as the scene category to which the image to be classified belongs. The prediction confidence that the image to be classified belongs to a certain scene category may be obtained by normalizing or mapping the feature extraction result of the image to be classified. Exemplarily, the prediction confidence may be the confidence of the image features of the image to be classified. The threshold confidence may be a confidence threshold. In some application scenarios, the prediction confidence that the image to be classified belongs to a certain scene category may be obtained by directly normalizing or mapping the feature extraction result of the image to be classified. In other application scenarios, the degree of matching obtained by matching the feature extraction results of the image to be classified with the reference features of each scene category can be normalized or mapped to obtain the prediction confidence that the image to be classified belongs to a certain scene category. At this time, the classification requirement can be a threshold confidence. The classification requirement can be preset or customized. The specific classification requirement can be an evaluation index for evaluating whether the prediction confidence obtained by normalizing or mapping the feature extraction results of the image to be classified belongs to a certain scene category. It can be understood that different scene categories correspond to different classification requirements, and different classification requirements can distinguish the image to be classified into different scene categories.

[0028] At least some scene categories correspond to different image processing modes. The image processing corresponding to different scene categories in several scene categories may be the same or different. The image processing mode is used to indicate the mode in which the functional module or image processing module in the image acquisition device can be, in which the functional module or image processing module can perform corresponding image processing on the image to be processed. The image to be processed may be a current image or a historical image. Exemplarily, when several scene categories may be foggy scenes, rainy scenes, and backlit scenes, the image processing modes corresponding to each scene category are different. For example, when the scene category is a backlit scene, the functional module or image processing module is in a wide dynamic mode; when the scene category is a foggy scene, the functional module or image processing module is in a fog-penetrating mode. The wide dynamic mode can be used to process high-contrast scenes and can capture the details of the bright and dark parts of the image to be processed. The fog-penetrating mode can remove the haze in the image to be processed and improve the clarity and contrast of the image to be processed.

[0029] It is understandable that the initial classification requirements for each scene category are different, and at least some scene categories have different image processing methods.

[0030] Step S12: Based on the target scene categories to which several frames of historical images belong, at least part of the initial classification requirements are updated to obtain new classification requirements corresponding to each scene category.

[0031] Several frames of historical images may be one frame of historical images or multiple frames of historical images. The historical images may be images collected by the image acquisition device in the historical time period before the current time period in which at least one frame of current image is located. The scene categories to which each historical image in several frames of historical images belongs may be the same or different. In some application scenarios, when the scene categories to which each historical image in several frames of historical images belongs are the same, the scene categories to which each historical image belongs are taken as the target scene category. In other application scenarios, when the scene categories to which each historical image in several frames of historical images belongs are different, at least one of the scene categories to which each historical image belongs is taken as the target scene category. Specifically, at least one of the scene categories to which each historical image belongs may be the scene category to which the historical image that is closest in time to the current time period belongs, or may be the scene category that appears the most times in the scene categories to which each historical image belongs. It can be understood that the target scene category is at least one of the final scene categories to which several frames of historical images belong.

[0032] In some application scenarios, updating at least part of the initial classification requirements to obtain new classification requirements corresponding to each scene category may be: when the target scene category is at least one of several scene categories, updating the initial classification requirements corresponding to the target scene category to obtain new classification requirements corresponding to the target scene category. At this time, the initial classification requirements corresponding to the scene categories other than the target scene category in several scene categories are maintained as the new classification requirements corresponding to the scene category. Exemplarily, the target scene categories are foggy scenes and rainy scenes. The initial classification requirements of foggy scenes and the initial classification requirements of rainy scenes are updated respectively to obtain new classification requirements of foggy scenes and new classification requirements of rainy scenes. The initial classification requirements of backlight scenes are maintained to obtain new classification requirements of backlight scenes. In other application scenarios, updating at least part of the initial classification requirements to obtain new classification requirements corresponding to each scene category may be: when the target scene category is at least one of several scene categories, maintaining the initial classification requirements corresponding to the target scene category to obtain new classification requirements corresponding to the target scene category. At this time, the initial classification requirements corresponding to the scene categories other than the target scene category among the several scene categories are updated to obtain new classification requirements corresponding to the scene categories.

[0033] It can be understood that updating the initial classification requirements corresponding to any scene category to obtain the new classification requirements corresponding to the scene category can be achieved by fine-tuning the threshold confidence in the above-mentioned initial classification requirements according to usage requirements. Specifically, it can be to increase the threshold confidence value or to lower the threshold confidence value. The updated threshold confidence value is used as the threshold confidence value of the new classification requirement.

[0034] Step S13: determining the final scene category of at least one frame of the current image according to the new classification requirements corresponding to each scene category.

[0035] For each current image, the final scene category of the current image is determined according to the new classification requirements corresponding to each scene category. The final scene category may be the scene category to which the current image belongs determined based on the new classification requirements, and the final scene category is at least one of several scene categories. It is understandable that the scene category of the current image determined according to the initial classification requirements corresponding to each scene category may be the same as or different from the final scene category.

[0036] If the prediction confidence that the current image belongs to a certain scene category reaches the threshold confidence value of the new classification requirement, the scene category is directly used as the final category of the current category. When the actual scene has not changed significantly, but the collected continuous multiple frames of images often appear in the initial classification requirements corresponding to the target scene category and contain fluctuations near the discrimination threshold, the scene category of the continuous multiple frames of images is determined to be constantly switching between belonging to the target scene category and not belonging to the target scene category, resulting in the inability to accurately define the scene category of the image. For each current image, determining the final scene category of the current image according to the new classification requirements corresponding to each scene category can improve the accuracy of the final scene category determined.

[0037] The above scheme updates at least part of the initial classification requirements based on the target scene categories to which several frames of historical images belong, and obtains new classification requirements corresponding to each scene category, so that when the scene category of the current image is determined, the new classification requirements are highly adapted to each scene category, and then the final scene category of at least one frame of the current image is determined according to the new classification requirements corresponding to each scene category. The final scene category of each current image can be determined by using classification requirements with a higher degree of adaptation, thereby improving the accuracy of the final scene category determined for each current image.

[0038] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of the method for determining the scene category of the present application. Figure 2 .

[0039] In some embodiments, the initial classification requirement includes an initial confidence requirement, and the new classification requirement includes a new confidence requirement. The above step S12 may include the following steps: Step S21: Reduce the initial confidence requirement corresponding to the target scene category to obtain a new confidence requirement corresponding to the target scene category.

[0040] The classification requirements include confidence requirements. The threshold confidence corresponding to each scene category may specifically be the confidence threshold corresponding to each scene category. The confidence requirements may include confidence thresholds corresponding to each scene category. The initial confidence requirements included in the initial classification requirements include initial confidence thresholds corresponding to each scene category. The new confidence requirements included in the new classification requirements include new confidence thresholds corresponding to each scene category. Reducing the initial confidence requirement corresponding to the target scene category to obtain a new confidence requirement corresponding to the target scene category may be: reducing the initial confidence threshold in the initial confidence requirement corresponding to the target scene category to obtain a new confidence threshold in the new confidence requirement corresponding to the target scene category. Exemplarily, the target scene categories are foggy scenes and rainy scenes. The initial confidence threshold in the initial classification requirements for foggy scenes and the initial confidence threshold in the initial classification requirements for rainy scenes are respectively reduced to obtain new confidence thresholds in the new classification requirements for foggy scenes and new confidence thresholds in the new classification requirements for rainy scenes. The initial confidence threshold in the initial classification requirement of the backlight scene remains unchanged and is used as the new confidence threshold in the new classification requirement of the backlight scene. Specifically, in the process of lowering the initial confidence requirement corresponding to the target scene category and obtaining the new confidence requirement corresponding to the target scene category, this application takes the initial confidence threshold in the initial confidence requirement multiplied by a preset value to obtain a new confidence threshold as an example, and the preset value is a value less than 1. For example, the preset value may be 0.8.

[0041] The new classification requirement is used to indicate that if the above prediction confidence value of the current image reaches the new confidence threshold in the new classification requirement corresponding to any scene category, the scene category is determined as the scene category to which the current image belongs.

[0042] In some embodiments, the above step S13 may include the following steps: Step S22: Determine the confidence that each current image belongs to each scene category respectively. Step S23: For each current image, determine the final scene category to which the current image belongs based on the relationship between the confidence between the current image and each scene category and the new confidence requirement of each scene category.

[0043] For each current image, the confidence that the current image belongs to each scene category can be determined by taking the image to be classified as the current image and executing the above steps of predicting the confidence that the image to be classified belongs to a certain scene category. Specifically, the confidence that the current image belongs to each scene category can be obtained by normalizing or mapping the feature extraction result of the current image. Exemplarily, the current image is input into a deep learning model, and the deep learning model can output the confidence that the current image belongs to each scene category.

[0044] In some embodiments, the above step S22 may include the following steps: performing the following steps on each current image: first, extracting features from the current image to obtain target features of the current image. Then, obtaining the similarity between the target features of the current image and the reference features corresponding to each scene category. Subsequently, the similarity between the target features and each reference feature is used as the confidence that the current image belongs to each scene category.

[0045] The target feature of the current image may be a feature extraction result obtained by extracting features from the current image. The current image is input into the backbone network of the deep learning model for feature extraction to obtain the target feature corresponding to the current image. Among them, the deep learning model is a trained deep learning model obtained by pre-training. The backbone network may be a preset feature extraction network. The feature extraction method may be to input the current image into the preset feature extraction network. Among them, the preset feature extraction network may be a MobileNet feature extraction network, an Inception feature extraction network, etc. The setting method of the feature extraction network is not limited here. The similarity between the target feature of the current image and the reference feature corresponding to each scene category may be the above-mentioned matching degree. The reference feature may include preset image feature information corresponding to each scene category. The reference features corresponding to different scene categories are different. The reference feature corresponding to each scene category may have scene image feature information specific to the scene category. The method for determining the similarity between the target feature and each reference feature may be a method for calculating the cosine similarity or Euclidean distance between the two. It can be understood that the method for calculating the similarity is not limited here. For each reference feature, the similarity between the target feature and the reference feature is determined as the similarity corresponding to the reference feature, and the similarity corresponding to the reference feature is used as the confidence that the current image belongs to the scene category corresponding to the reference feature.

[0046] See also Figure 3 , Figure 3 This is a flow chart of an embodiment of the method for determining the scene category of the present application. Figure 3 .

[0047] In some embodiments, the above step S23 may include the following steps: Step S31: In response to the confidence corresponding to each scene category reaching the target confidence requirement, the scene category corresponding to the target confidence is determined as the initial scene category of the current image. The target confidence requirement is at least one of the new confidence requirements. Step S32: Based on the initial scene category of each current image, the final scene category of each current image is determined.

[0048] Determine whether the confidence corresponding to each scene category meets the target confidence requirement. Specifically, for the confidence corresponding to each scene category, determine whether the confidence corresponding to the scene category meets the new confidence threshold in the new confidence requirement corresponding to the scene category. In the case that the confidence corresponding to the scene category reaches the new confidence threshold in the new confidence requirement corresponding to the scene category, the scene category is determined as the initial scene category to which the current image belongs. Because the initial scene category to which each current image belongs can be at least one of several scene categories, the confidence requirement that can be met by the confidence corresponding to each scene category of the current image can also be the new confidence requirement corresponding to at least one scene category.

[0049] In some application scenarios, after obtaining the initial scene category to which each current image belongs, the above step S32 may be to randomly specify, forcibly specify, or custom specify at least part of the initial scene categories to which each current image belongs as the final scene category to which each current image belongs. In other application scenarios, the final scene category of each current image is determined based on the distribution of the initial scene categories of each current image. The distribution is determined based on the statistical value of the initial scene category to which each current image belongs. Exemplarily, when the number of each current image is 10 frames, the statistical value of the initial scene category may represent 8 first scene categories, 1 second scene category, and 1 third scene category. Among them, the first scene category may be a rainy scene and a foggy scene. The second scene category may be a rainy scene. The third scene category may be a backlight scene. At this time, the scene category with the largest number of occurrences in the statistical value of the initial scene category is determined as the final scene category of each current image. Exemplarily, when the number of current images is 10 frames, the statistical value of the initial scene category may indicate that the first five frames of the current image have the first scene category five times in a row, and the second scene category has the second scene category five times in a row in the last five frames of the current image. Therefore, the final scene category to which the first five frames of the current image belong is determined as the first scene category, and the final scene category to which the last five frames of the current image belong is determined as the second scene category. It is understandable that, when the initial scene category to which the current images of consecutive preset frames belong is the same scene category, the final scene category to which the current images of the consecutive preset frames belong is determined as the same scene category.

[0050] In some embodiments, the number of at least one current image frame is multiple frames, and the above step S32 may include the following steps: in response to the inconsistency of the initial scene categories of the current images, the target scene category is used as the updated scene category of each current image. Or, in response to the consistency of the initial scene categories of the current images, the initial scene category of each current image is used as the updated scene category of each current image. Subsequently, based on the updated scene category of each current image, the final scene category of each current image is determined.

[0051] The number of at least one current image is multiple frames. That is, at least one current image is at least two current images. The step of updating the initial scene categories of each current image to obtain the updated scene categories of each current image may be to determine whether the initial scene categories of each current image are consistent. Exemplarily, taking at least one current image as two current images as an example, if the initial scene category corresponding to the first current image is a foggy scene, and the initial scene category corresponding to the second current image is a foggy scene and a rainy scene, then in response to the inconsistency of the initial scene categories of each current image, the target scene category (foggy scene and rainy scene) is used as the updated scene category of the first current image and the second current image, respectively. Exemplarily, taking at least one current image as two current images as an example, if the initial scene category corresponding to the first current image is a foggy scene, and the initial scene category corresponding to the second current image is a foggy scene, then in response to the consistency of the initial scene categories of each current image, the initial scene categories of the first current image and the second current image are maintained, and the initial scene category is maintained as the updated scene category of each current image. Therefore, at this time, the updated scene categories corresponding to each current image are the same scene category. It is understandable that the present application does not limit the number of current images. Exemplarily, the target scene category is the 10 consecutive historical images closest to the current image, that is, the several historical images are the 10 consecutive historical images. And the final scene category to which the several historical images belong is the same, and the final scene category is determined as the target scene category.

[0052] In some application scenarios, the step of determining the final scene category of each current image based on the updated scene category of each current image may be: directly determining the final scene category of each current image by using the updated scene category of each current image. In other application scenarios, for each current image, one of the initial scene category and the updated scene category corresponding to the current image may be randomly specified or custom set and forcibly specified as the final scene category of the current image. Specifically, in the case of custom setting, a custom weight is set for each scene category. For example, the custom weights of the rainy scene and the foggy scene are the same, and the custom weights of the rainy scene and the foggy scene are greater than the custom weight of the backlight scene. That is, when the updated scene category to which the current image belongs includes the rainy scene and the backlight scene, the final scene category to which the current image belongs is determined to be the rainy scene. Specifically, the mandatory designation may directly designate the final scene category to which the current image belongs as at least one of the scene categories when the updated scene category to which the current image belongs includes at least one preset scene category. In other application scenarios, based on the distribution of the update scene categories to which each current image belongs, one is selected from the initial scene category to which each current image belongs or the update scene category to which each current image belongs as the final scene category to which each current image belongs. Specifically, the distribution of the update scene categories to which each current image belongs may be the scene category distribution of the update scene categories to which each current image belongs and the final scene category of multiple frames of historical images. The scene category distribution may be a statistical value of the scene categories of a preset number of images, for example, the preset number of images may be 10 frames of current images and 190 frames of historical images. The scene category distribution is obtained by the update scene categories to which the 10 frames of current images belong and the final scene category of the 190 frames of historical images. The scene category distribution may be the number of occurrences of various scene category combinations, and the scene category combination may be at least one scene category. The scene category distribution may be the number of scene switching times in a preset number of images, and the inconsistency of the scene categories to which two consecutive frames of images belong is recorded as one scene switching.

[0053] It can be considered that the scene determination of the actual current image cannot avoid the occurrence of a few misjudgments, or when the variable image acquisition scene changes instantly, the actual scene category does not want to change. Therefore, by updating or maintaining the initial scene category to which the current image belongs, the updated scene category to which the current image belongs can ensure the accuracy of the subsequent final scene category determination.

[0054] In some embodiments, the above-mentioned step of determining the final scene category of each current image based on the updated scene category of each current image may include the following steps: obtaining the number of scene category switching within a preset time period when the final scene category of the current image is the updated scene category, and at least one frame of current image and several frames of historical images can be collected within the preset time period, wherein the case where two adjacent scene categories are inconsistent is recorded as one scene category switching. Subsequently, in response to the number of switching reaching the number threshold, the initial scene category of each current image is respectively determined as the final scene category of each current image.

[0055] The number of scene category switching times is used to indicate the number of scene category switching times obtained by counting whether two adjacent scene categories in the scene category to which a preset number of images belong are consistent. Wherein, if two adjacent scene categories are inconsistent, it is recorded as one scene category switching. The preset number of images may include at least one current image and several historical images that can be collected within a preset time period. The preset number of images may also include at least one current image and multiple historical images including several historical images that can be collected within a preset time period. The number of multiple historical images is greater than the number of multiple historical images. Under the assumption that the final scene category of the current image is the updated scene category of the current image, the scene category to which the preset number of images belongs may be the final scene category of each image. Wherein, the scene category to which the preset number of images belongs may include the final scene category of each current image and the final scene category to which several historical images belong or the final scene category to which multiple historical images belong. The number threshold may be a preset number or a dynamically adjusted number. When the number threshold is a dynamically adjusted number, the number threshold is dynamically adjusted based on the number of the preset number of images, and the greater the number of the preset number of images, the greater the number of the number threshold.

[0056] In some embodiments, the scene category determination method may further include the following steps: in response to the switching times not reaching the times threshold, determining the updated scene category of each current image as the final scene category of each current image.

[0057] Exemplarily, the preset number of images may be each current image and multiple frames of historical images. Multiple frames of historical images may be multiple frames of historical images or more historical images including multiple frames of historical images, that is, the number of historical images contained in multiple frames of historical images may be more than the above-mentioned multiple frames of historical images. Multiple frames of historical images are images adjacent to each current image and acquired before the current time period in which each current image is located. Multiple frames of historical images are images including multiple frames of historical images and acquired before the current time period in which each current image is located. This application takes the multiple frames of historical images as multiple frames of historical images including multiple frames of historical images as an example. For example, multiple frames of historical images may be 190 frames of historical images. The preset number of images may be 10 frames of current images and 190 frames of historical images. Assuming that the final scene category of the current image is the updated scene category of the current image, the updated scene category to which the 10 frames of current images belong and the final scene category of the 190 frames of historical images obtain the scene category distribution. The scene category distribution may be the number of occurrences of various scene category combinations, and the scene category combination may be at least one scene category. The scene category distribution may be the number of scene switching times in a preset number of images, and the inconsistency of the scene categories of two consecutive frames of images is recorded as one scene switching. It is determined whether the number of switching times reaches a number threshold. For example, the number threshold may be 10 times. When the number of switching times reaches the number threshold, the initial scene category of each current image is determined as the final scene category of each current image. When the number of switching times does not reach the number threshold, the updated scene category of each current image is determined as the final scene category of each current image.

[0058] It is understandable that actual image acquisition scenarios often involve short-term continuous scene switching. For example, if an image acquisition device is facing the entrance and exit of an animal enclosure area and animals frequently appear at the entrance and exit, the scene category of the image may frequently switch between backlit scenes and non-backlit scenes. The corresponding image processing module will frequently switch between wide dynamic mode, which may easily lead to resource consumption of the image acquisition device and abnormal image processing results.

[0059] In the dimension of time domain, the queue corresponding to the preset number of images is first-in-first-out according to the order of acquisition time, and the queue of the final scene category to which the preset number of images belongs is retained. Among them, the queue of the final scene category to which the preset number of images belongs can be represented as a State_steady queue. When the number of scene category switching in the State_steady queue is greater than or equal to the number threshold, the scene state of the preset number of images is a fluctuating state. In the fluctuating state, the initial scene category of each current image is determined as the final scene category of each current image. When the number of scene category switching in the State_steady queue is less than the number threshold, the scene state of the preset number of images is a stable state. In the stable state, the updated scene category of each current image is determined as the final scene category of each current image.

[0060] In some embodiments, after step S13, the scene category determination method may further include the following steps: first, the image processing method of the final scene category to which each current image belongs is used as the target image processing method. Then, for each current image, the target image processing method is used to perform image processing on the current image to obtain an image processing result.

[0061] For each current image, the image processing method of the final scene category to which the current image belongs is used as the target image processing method corresponding to the current image. Exemplarily, when the scene category is a backlit scene, the functional module or the image processing module is in a wide dynamic mode; when the scene category is a foggy scene, the functional module or the image processing module is in a fog-penetrating mode. When the final scene category to which the current image belongs is a backlit scene, the image processing module in the wide dynamic mode is used to perform image processing on the current image to obtain an image processing result. When the final scene category to which the current image belongs is a foggy scene, the image processing module in the fog-penetrating mode is used to perform image processing on the current image to obtain an image processing result.

[0062] See also Figure 4 , Figure 4 This is a flow chart of an embodiment of the method for determining the scene category of the present application. Figure 4 .

[0063] Execute steps S401 to S409 in sequence. Step S401: Obtain at least one frame of current image and initial classification requirements corresponding to several scene categories. Step S402: Reduce the initial confidence requirement corresponding to the target scene category to obtain a new confidence requirement corresponding to the target scene category. Step S403: Perform feature extraction on the current image to obtain the target feature of the current image. Step S404: Obtain the similarity between the target feature of the current image and the reference feature corresponding to each scene category. Step S405: Use the similarity between the target feature and each reference feature as the confidence that the current image belongs to each scene category. Step S406: In response to the confidence corresponding to each scene category reaching the target confidence requirement, determine the scene category corresponding to the target confidence as the initial scene category of the current image. Step S407: Determine whether the initial scene categories of each current image are consistent. If the judgment result of step S407 is no, execute step S408. Step S408: In response to the inconsistency of the initial scene categories of each current image, the target scene category is used as the update scene category of each current image. If the judgment result of step S407 is yes, step S408 is executed. Step S409: In response to the consistency of the initial scene categories of each current image, the initial scene category of each current image is used as the update scene category of each current image. Step S410: Determine whether the number of switching times reaches the number threshold. If the judgment result of step S410 is reached, step S411 is executed. Step S411: In response to the number of switching times reaching the number threshold, the initial scene category of each current image is determined as the final scene category of each current image. If the judgment result of step S410 is not reached, step S412 is executed. Step S412: In response to the number of switching times not reaching the number threshold, the update scene category of each current image is determined as the final scene category of each current image. Step S413: For each current image, the current image is processed using the target image processing method to obtain an image processing result.

[0064] It is understandable that after the current image captured by the actual image acquisition device is identified as certain special categories, the current image will be processed accordingly, such as turning on wide dynamic and fog penetration processing. When the scene recognition algorithm makes a misjudgment or in a changeable image acquisition scene, the scene category to which the current image belongs fluctuates frequently, which may have a serious impact on the image processing results collected by the image acquisition device. From the dimension of time domain changes, the present application proposes to use the initial image category to which the current image belongs or the updated image category to which the current image belongs as the final image category when the scene state of a preset number of images is a fluctuating state or a stable state, which can ensure the accuracy and stability of the final scene category of the current image when abnormal judgment and the scene state of the image are in the case of time domain fluctuations.

[0065] The above step S22 and the above step S31 are both performed by a deep learning model. The scene category determination method may further include a training step for the deep learning model. The training step includes the following contents:

[0066] It is understandable that in the scene category determination task, the extreme imbalance in the amount of training data between scene categories is very common in real deep training scenarios, that is, in the training set, the amount of data for a certain common scene label is very large, while some uncommon classification scene labels are relatively small, and the ratio is very disparate, which will cause the deep learning model to easily misclassify uncommon scene categories into common scene labels. Common processing methods are to add more scarce category data, increase the loss weight of scarce category samples, use Focal Loss loss function and other solutions. Specifically, the process of using Focal Loss loss function can refer to formula (1):

[0067] Formula (1);

[0068] Where P is the sample confidence corresponding to each sample image. The sample confidence corresponding to the sample image is used to indicate the similarity between the sample target features corresponding to each sample image and the sample reference features corresponding to each scene category. a is the category weight, and λ is the difficult sample weight. Both are hyperparameters and can be set according to training requirements. Used to represent the classification loss function. Specifically, the classification loss function can be a Focal Loss loss function. However, the use of a single FocalLoss loss function has the defects of being sensitive to abnormal samples and not handling abnormal points at the classification boundary ideally. This application uses the Focal Loss loss function in combination with the multi-center loss function SoftTriple to adjust the parameters in the deep learning model to obtain a trained deep learning model. After the deep learning model is trained, when the scene category is determined, the trained deep model is first applied to infer the current image to obtain the initial scene category to which each current image belongs.

[0069] In some application scenarios, the network structure of the deep learning model is first determined; the feature extraction network of the deep learning network can select a preset feature extraction network according to the needs. Subsequently, a classification Focal Loss loss function containing the feature vector of the category center cluster is constructed to perform deep learning training. It includes constructing a multi-center loss function SoftTriple, determining the distance measure between the to-be-classified example and the specific category as the loss function. And combining the loss function of Focal Loss to increase the classification weight of difficult samples. Specifically, the process of constructing the multi-center loss function can be to obtain sample target features corresponding to several sample images, each target feature is obtained by inputting several sample images into the deep learning model for feature extraction; determine several center features corresponding to each scene category; determine the multi-center loss according to the multi-center loss function and the center similarity, the center similarity is the similarity between the sample target features corresponding to each sample image and the several center features corresponding to each scene category; use the multi-center loss to adjust the parameters in the deep learning model to obtain a trained deep learning model. In other application scenarios, the classification loss obtained by the multi-center loss and the Focal Loss loss function is weighted summed to obtain the target loss. The target loss is then used to adjust the parameters in the deep learning model to obtain a trained deep learning model.

[0070] It is understandable that the present application uses the central loss function SoftTriple combined with the FocalLoss loss function for the deep learning model to train a feature extraction network and a set of scene category cluster central measurement feature vectors. The trained deep learning model directly eliminates the prior information of data classification sample imbalance and converts it into a measurement problem between the feature vectors of each scene category cluster, which can effectively solve the problem of scene category determination misdetection caused by extreme sample imbalance.

[0071] The above scheme updates at least part of the initial classification requirements based on the target scene categories to which several frames of historical images belong, and obtains new classification requirements corresponding to each scene category, so that when the scene category of the current image is determined, the new classification requirements are highly adapted to each scene category, and then the final scene category of at least one frame of the current image is determined according to the new classification requirements corresponding to each scene category. The final scene category of each current image can be determined by using classification requirements with a higher degree of adaptation, thereby improving the accuracy of the final scene category determined for each current image.

[0072] See also Figure 5 , Figure 5It is a structural diagram of an embodiment of a scene category determination device of the present application. The scene category determination device 50 includes an acquisition module 51, an update module 52 and a determination module 53. The acquisition module 51 is used to acquire at least one frame of current image and initial classification requirements corresponding to several scene categories, and at least some of the scene categories have different image processing methods; the update module 52 is used to update at least some of the initial classification requirements based on the target scene category to which several frames of historical images belong, and obtain new classification requirements corresponding to each scene category; the determination module 53 is used to determine the final scene category of at least one frame of current image according to the new classification requirements corresponding to each scene category.

[0073] The above scheme updates at least part of the initial classification requirements based on the target scene categories to which several frames of historical images belong, and obtains new classification requirements corresponding to each scene category, so that when the scene category of the current image is determined, the new classification requirements are highly adapted to each scene category, and then the final scene category of at least one frame of the current image is determined according to the new classification requirements corresponding to each scene category. The final scene category of each current image can be determined by using classification requirements with a higher degree of adaptation, thereby improving the accuracy of the final scene category determined for each current image.

[0074] Please refer to the scene category determination method for the functions performed by each module, which will not be repeated here.

[0075] See also Figure 6 , Figure 6 6 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 60 includes a memory 61 and a processor 62, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps in the above-mentioned scene category determination method embodiment. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a multi-camera device, a microcomputer, and a server. In addition, the electronic device 60 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0076] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps in the above-mentioned scene category determination method embodiment. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0077] The above scheme updates at least part of the initial classification requirements based on the target scene categories to which several frames of historical images belong, and obtains new classification requirements corresponding to each scene category, so that when the scene category of the current image is determined, the new classification requirements are highly adapted to each scene category, and then the final scene category of at least one frame of the current image is determined according to the new classification requirements corresponding to each scene category. The final scene category of each current image can be determined by using classification requirements with a higher degree of adaptation, thereby improving the accuracy of the final scene category determined for each current image.

[0078] See also Figure 7 , Figure 7 The computer-readable storage medium 70 stores program instructions 701, which implement the steps in any of the above-mentioned scene category determination method embodiments when executed by a processor.

[0079] The above scheme updates at least part of the initial classification requirements based on the target scene categories to which several frames of historical images belong, and obtains new classification requirements corresponding to each scene category, so that when the scene category of the current image is determined, the new classification requirements are highly adapted to each scene category, and then the final scene category of at least one frame of the current image is determined according to the new classification requirements corresponding to each scene category. The final scene category of each current image can be determined by using classification requirements with a higher degree of adaptation, thereby improving the accuracy of the final scene category determined for each current image.

[0080] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0081] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0082] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0083] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0084] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A method for determining a scene category, characterized in that: The method comprises: Obtaining at least one current image frame and initial classification requirements corresponding to a number of scene categories, wherein at least some of the scene categories correspond to different image processing methods; Based on the target scene categories to which the plurality of frames of historical images belong, at least part of the initial classification requirements are updated to obtain new classification requirements corresponding to each of the scene categories; Determining a final scene category of the at least one frame of the current image according to new classification requirements corresponding to each of the scene categories; The initial classification requirement includes an initial confidence requirement, the new classification requirement includes a new confidence requirement, and the updating of at least part of the initial classification requirements based on the target scene categories to which the plurality of frames of historical images belong to obtain new classification requirements corresponding to each of the scene categories includes: lowering the initial confidence requirement corresponding to the target scene category to obtain the new confidence requirement corresponding to the target scene category; Among them, determining the final scene category of the at least one frame of the current image according to the new classification requirements corresponding to each of the scene categories includes: separately determining the confidence that each of the current images belongs to each of the scene categories; for each of the current images, determining the final scene category to which the current image belongs based on the relationship between the confidence of the current image and each of the scene categories and the new confidence requirements of each of the scene categories.

2. The method according to claim 1, characterized in that The respectively determining the confidence that each of the current images belongs to each of the scene categories comprises: For each of the current images, perform the following steps: Performing feature extraction on the current image to obtain target features of the current image; Obtaining the similarity between the target feature of the current image and the reference features corresponding to each of the scene categories; The similarity between the target feature and each of the reference features is used as the confidence that the current image belongs to each of the scene categories.

3. The method according to claim 1, characterized in that The step of determining, for each of the current images, a final scene category to which the current image belongs based on a relationship between the confidences of the current image and each of the scene categories and a new confidence requirement of each of the scene categories comprises: In response to the confidence corresponding to each of the scene categories reaching a target confidence requirement, determining the scene category corresponding to the target confidence as the initial scene category of the current image, the target confidence requirement being at least one of the new confidence requirements; Based on the initial scene category of each of the current images, a final scene category of each of the current images is determined.

4. The method according to claim 3, characterized in that The number of the at least one current image frame is multiple frames, and determining the final scene category of each current image based on the initial scene category of each current image includes: In response to the inconsistency of the initial scene categories of the current images, the target scene categories are used as the updated scene categories of the current images; or, in response to the consistency of the initial scene categories of the current images, the initial scene categories of the current images are used as the updated scene categories of the current images; Based on the updated scene category of each of the current images, a final scene category of each of the current images is determined.

5. The method according to claim 4, characterized in that The determining, based on the updated scene category of each of the current images, the final scene category of each of the current images comprises: Obtaining the number of scene category switchings within a preset time period when the final scene category of the current image is the updated scene category, wherein the at least one frame of the current image and the plurality of frames of historical images can be collected within the preset time period, wherein the inconsistency of two adjacent scene categories is recorded as one scene category switching; In response to the switching times reaching a times threshold, the initial scene categories of the current images are respectively determined as the final scene categories of the current images.

6. The method according to claim 5, characterized in that The method further comprises: In response to the switching times not reaching the times threshold, the updated scene category of each of the current images is determined as the final scene category of each of the current images.

7. The method according to any one of claims 1 to 6, characterized in that After determining the final scene category of the at least one frame of the current image according to the new classification requirements corresponding to each of the scene categories, the method includes: Using the image processing method of the final scene category to which each current image belongs as the target image processing method; For each of the current images, the current image is processed using a target image processing method to obtain an image processing result.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 7.

Images