Image data optimization method and device, electronic equipment and storage medium

Abstract

The invention provides an image data optimization method and device, electronic equipment and a storage medium, and the method comprises the steps: responding to a calling behavior of a third-party application for a camera service of an operation system, and recognizing the type of a camera application scene corresponding to the calling behavior; and determining a preset optimization strategy corresponding to the identified type so as to optimize original image data generated by the camera service in response to the calling behavior. By identifying the calling intention of the third-party application for the camera service, corresponding image data optimization can be performed according to the types of different camera application scenes, and the actual requirements of the third-party application are met.

Description

Technical Field

[0001] The present disclosure relates to the field of digital imaging technology, and in particular to an image data optimization method, device, electronic device and storage medium. Background Art

[0002] Third-party applications are allowed to use the camera API (Application Programming Interface) to customize the camera's interface and functions. By calling the camera service, third-party applications can interact with the camera device and customize corresponding camera functions for different camera application scene types. For example, camera functions for different scene types such as taking photos, recording videos, video calls, and scanning applications can be implemented within the application.

[0003] In the related art, the operating system does not optimize the image data specifically according to the scene type in which the camera function is located, but the third-party application decides which optimization strategy to use to optimize the obtained image data.

[0004] However, designing reasonable optimization strategies for camera applications in different scene types requires high professional requirements, and due to the professional level of most third-party application developers, third-party applications often do not optimize image data specifically for the scene types in which the camera function is located. Of course, even if the third-party application developers have sufficient professional level, since each third-party application needs to set an optimization strategy separately, it will still lead to low development efficiency and waste of a lot of development resources. Summary of the invention

[0005] In view of this, the present disclosure provides an image data optimization method, device, electronic device and storage medium to solve the above technical problems.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for optimizing image data is provided, the method comprising:

[0007] In response to a calling behavior of a third-party application on a camera service of the operating system, identifying a type of a camera application scenario corresponding to the calling behavior;

[0008] A preset optimization strategy corresponding to the identified type is determined to optimize the raw image data generated by the camera service in response to the calling behavior.

[0009] Optionally, the identifying the type of camera application scenario corresponding to the calling behavior includes:

[0010] Acquire characteristic information for describing the calling behavior;

[0011] Based on the acquired feature information, the type of camera application scenario corresponding to the calling behavior is identified.

[0012] Optionally, the acquiring characteristic information for describing the calling behavior includes:

[0013] A data stream sent by the third-party application to the camera hardware abstraction layer of the operating system is obtained, and the feature information is extracted from the obtained data stream.

[0014] Optionally, the preset optimization strategy is defined in the camera hardware abstraction layer.

[0015] Optionally, the feature information includes multi-dimensional sub-features with time sequence; and the identifying the type of camera application scenario corresponding to the calling behavior includes:

[0016] According to the temporal sequence between the multi-dimensional sub-features, the type of the camera application scene is iteratively identified in multiple rounds according to the sub-features of each dimension;

[0017] The input for the first round of iterative recognition is the sub-features of the first dimension, and the input for non-first round of iterative recognition is the sub-features of the corresponding dimension and the recognition result of the previous round of iterative recognition.

[0018] Optionally, the performing multiple rounds of iterative identification on the type of the camera application scene according to the sub-features of each dimension in turn includes:

[0019] Inputting the multi-dimensional sub-features into a pre-trained hidden Markov model to generate hidden state nodes corresponding to the sub-features of each dimension respectively;

[0020] At each hidden state node, a round of iteration is performed on the type of the camera application scene according to the sub-feature corresponding to the hidden state node, and the recognition result generated by the previous hidden state node is transferred to the next hidden state node through the transfer matrix;

[0021] Among them, the recognition result generated by the hidden state node arranged at the end is the recognition result of the multiple rounds of iterative recognition.

[0022] Optionally, the multidimensional sub-features include:

[0023] Behavioral characteristics of camera activity;

[0024] Configuration information of the camera stream, including the size, number and encoding format of the camera stream;

[0025] The data transfer method used by the codec and the process of building the graphics buffer source;

[0026] Take a picture of a binary frame and request metadata information;

[0027] Camera streaming use case setup.

[0028] Optionally, the identifying the type of camera application scenario corresponding to the calling behavior includes:

[0029] Identifying the calling behavior through a pre-trained scene classifier to determine the type of camera application scene corresponding to the calling behavior;

[0030] The scene classifier is trained by capturing historical data generated by historical calling behaviors of the camera service by a third-party application.

[0031] Optionally, the method further includes: updating the scene classifier according to model update data sent by the cloud server.

[0032] Optionally, the original image data includes: preview image data and / or captured image data.

[0033] According to a second aspect of an embodiment of the present disclosure, there is provided an image data optimization device, the device comprising:

[0034] an identification module, configured to identify, in response to a calling behavior of a third-party application for a camera service of an operating system, a type of a camera application scenario corresponding to the calling behavior;

[0035] The optimization module is used to determine a preset optimization strategy corresponding to the identified type to optimize the raw image data generated by the camera service in response to the calling behavior.

[0036] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0037] processor;

[0038] a memory for storing a computer program executable by the processor;

[0039] The processor is configured to execute the computer program in the memory to implement the steps of the method described in the first aspect.

[0040] According to a fourth aspect of an embodiment of the present disclosure, there is provided a non-temporary computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0041] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0042] The present disclosure divides camera application scenarios into different types, and designs corresponding preset optimization strategies for the optimization requirements of different scenario types. Then, in response to the calling behavior of a third-party application for the camera service of the operating system, the corresponding preset optimization strategy can be determined according to the type of camera application scenario corresponding to the calling behavior, so as to optimize the original image data generated by the camera service in response to the calling behavior. Through the technical solution of the present disclosure, there is no need for third-party application developers to customize corresponding optimization strategies for different scene types of camera applications, nor is there any need for third-party applications to make any adjustments or changes. The operating system can automatically identify the type of camera application scenario corresponding to the calling behavior of the third-party application, and automatically select and execute the corresponding preset optimization strategy according to the type, which helps to save the development resources of third-party application developers and improve the development efficiency of third-party applications. At the same time, since the optimization operation is uniformly implemented at the operating system level, it can ensure that users can obtain a uniform optimization effect when using all third-party applications, and there will be no gap in the user experience.

[0043] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0045] Figure 1 is a flow chart of an image data optimization method according to an exemplary embodiment of the present disclosure;

[0046] Figure 2 is a schematic diagram of a scene classifier based on a hidden Markov model according to an exemplary embodiment of the present disclosure;

[0047] Figure 3 is a schematic diagram of another method for optimizing image data according to an exemplary embodiment of the present disclosure;

[0048] Figure 4 is a block diagram of an image data optimization device according to an exemplary embodiment of the present disclosure;

[0049] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0051] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0052] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0053] Third-party applications implement customized camera functions by calling the camera service in the operating system. Different camera functions may have different optimization requirements for image data, and each optimization requirement can be summarized as a type of camera application scenario. For example, many third-party applications have different types of application scenarios such as scanning, video calling, and high-definition photography. For scanning scenarios, there is a general optimization requirement for rapid response to third-party application requests and rapid identification; for video calling scenarios, there is a general optimization requirement for reducing power consumption by sacrificing image resolution; and unlike video calling scenarios, for high-definition photography scenarios, there is a general optimization requirement for improving image resolution.

[0054] In the related art, the operating system does not optimize the image data specifically according to the scene type in which the camera function is located, but the third-party application decides which optimization strategy to use to optimize the obtained image data.

[0055] Therefore, when third-party application developers are faced with camera applications in different scene types, they need to customize the corresponding image data optimization strategy for the scene type in the third-party application based on the identified scene type of the camera application, so that the operating system can accurately execute the image data optimization strategy customized by the third-party application.

[0056] However, designing reasonable optimization strategies for camera applications in different scene types requires high professional requirements, and due to the professional level of most third-party application developers, third-party applications often do not optimize image data specifically for the scene types in which the camera function is located. Of course, even if the third-party application developers have sufficient professional level, since each third-party application needs to set an optimization strategy separately, it will still lead to low development efficiency and waste of a lot of development resources.

[0057] In order to solve the above problems, the present disclosure proposes an image data optimization method, which does not require third-party application developers to customize corresponding optimization strategies for different scene types of camera applications. The operating system automatically identifies the type of camera application scenario corresponding to the third-party camera's call to the camera service, and automatically executes the corresponding preset optimization strategy according to the scene type.

[0058] Next, the embodiments of the present disclosure are described in detail.

[0059] like Figure 1 As shown, Figure 1 1 is a flowchart of an image data optimization method according to an exemplary embodiment of the present disclosure, comprising the following steps:

[0060] Step 101: In response to a calling behavior of a third-party application for a camera service of an operating system, identify a type of camera application scenario corresponding to the calling behavior.

[0061] Third-party applications can refer to software applications created and provided by independent developers, organizations or companies that are not directly associated with a specific operating system, platform or service provider. These applications are usually designed and developed to run on a specific operating system, platform or service and provide specific functions or services.

[0062] In an illustrated embodiment, feature information for describing the calling behavior may be obtained; based on the obtained feature information, the type of camera application scenario corresponding to the calling behavior may be identified. For example, the calling behavior of a third-party application for the camera service of the operating system may be detected by monitoring the camera service calling event. By monitoring the calling event of the camera service, in response to the calling behavior of the third-party application for the camera service of the operating system, feature information for describing the calling behavior may be obtained, and based on the obtained feature information, the type of camera application scenario corresponding to the calling behavior may be identified.

[0063] The specific content of the characteristic information of this embodiment will be described in detail in the following embodiments and will not be elaborated here.

[0064] In an illustrated embodiment, the feature information may be extracted by acquiring a data stream sent by the third-party application to the camera hardware abstraction layer of the operating system, and extracting the feature information from the acquired data stream.

[0065] In this embodiment, the Camera Hardware Abstraction Layer (Camera HAL) is located between the operating system and the camera hardware, and provides an access interface for different camera hardware, so that different third-party applications can interact with the camera hardware in a consistent manner.

[0066] The data stream sent by the third-party application to the camera hardware abstraction layer of the operating system may refer to a specific command or operation request issued by the third-party application to the camera hardware abstraction layer by calling the camera API to control the behavior and function of the camera hardware.

[0067] In one embodiment, the camera application scenarios of third-party applications can be divided into multiple types such as video calling, code scanning, high-definition photo taking, video recording, and ordinary preview photo taking. Different types of camera application scenarios have different optimization requirements:

[0068] For example, for application scenarios such as video calls, since third-party applications need to call the camera function for a long time, the preset optimization strategy for video calls needs to consider reducing power consumption and sacrificing some image and digital zoom quality to provide users with a better video experience.

[0069] For example, for application scenarios such as code scanning, third-party applications have optimization requirements for quickly obtaining images with stable quality. Therefore, the preset optimization strategy for code scanning needs to consider quickly responding to requests from third-party applications and providing high-quality images to third-party applications to enhance the user's experience of quickly scanning codes.

[0070] For example, for application scenarios such as high-definition photography, third-party applications have optimization requirements for obtaining high-quality, high-resolution static photos. Therefore, the preset optimization strategy for high-definition photography needs to consider providing high-quality, high-resolution photos to third-party applications to enhance the user's photography experience.

[0071] For example, for video recording application scenarios, third-party applications have optimization requirements for obtaining high-quality and stable video recordings. Therefore, the preset optimization strategy for video recording needs to consider providing high-quality and stable video recordings to third-party applications to improve the user's video recording experience.

[0072] For example, for general preview and photo-taking application scenarios, third-party applications have default basic camera optimization requirements. Therefore, the preset optimization strategy for general preview and photo-taking needs to consider providing basic camera functions to third-party applications to give users a balanced camera experience.

[0073] Those skilled in the art can design corresponding preset optimization strategies according to the application requirements of the divided scene types, and the present disclosure does not impose any restrictions on this. The above division of camera application scene types is only for illustration, and the present disclosure does not impose any restrictions on the division method and number of camera application scene types.

[0074] In an illustrated embodiment, the feature information includes multi-dimensional sub-features with temporal sequence. In this embodiment, the extracted feature information is divided into multi-dimensional sub-features with temporal sequence according to the temporal sequence relationship between each other. In this embodiment, the temporal sequence relationship between adjacent sub-features means that the actual content of the previous one-dimensional sub-feature will affect the generated content of the latter one-dimensional sub-feature. For example, the actual content of the previous one-dimensional sub-feature can be the behavioral feature of the camera activity, and the behavioral feature of the camera activity actually reflects that the camera function used by the third-party application is a video function, and the content of the latter one-dimensional sub-feature can be the configuration information of the camera stream. Then, the configuration information will be affected by the camera activity defined by the previous one-dimensional sub-feature, and the generated configuration information reflects more the information of the video stream rather than the information of other scene types.

[0075] According to the temporal sequence between the multidimensional sub-features, multiple rounds of iterative recognition are performed on the type of the scene of the camera application according to the sub-features of each dimension, wherein the input in the first round of iterative recognition is the sub-features of the first dimension, and the input in the non-first round of iterative recognition is the sub-features of the corresponding dimension and the recognition result of the previous round of iterative recognition.

[0076] For example, the feature information is divided into five sub-features with a temporal correlation according to the time sequence. First, the scene type of the camera application is identified in the first round according to sub-feature 1. Then, the second round of identification is performed together with sub-feature 2 using the first round of identification results. Then, the third round of identification is performed together with sub-feature 3 using the second round of identification results, and so on. Finally, the fifth round of identification is performed together with sub-feature 5 using the fourth round of identification results.

[0077] Regarding the implementation of this embodiment, a network architecture that can utilize temporal information, such as a recurrent neural network (RNN), a convolutional neural network (CNN), a hidden Markov model, and a Transformer, can be used to implement the technical solution of the above embodiment. Taking RNN as an example, the multidimensional sub-features can be input into the RNN as input vectors according to the temporal sequence relationship. The input of the RNN in the first round of iterative recognition is the input of the first time step, and the input of the RNN in the non-first round of iterative recognition is the input of the current time step and the hidden state of the previous time step as input, and then the hidden state of the current time step and the corresponding output are output. This loop structure can integrate the recognition information of the previous round into the processing of the current time step, thereby capturing the dependency of the time series.

[0078] In this embodiment, the temporal relationship between multi-dimensional sub-features is fully utilized. Through multiple rounds of iterative recognition, the input information of each round contains the recognition results of the previous round. Context associations are established during the recognition process, which can better explain the sub-features of the current dimension. Each round of iteration uses the recognition results of the previous round of iteration as input. As the iteration proceeds, the recognition results will be gradually refined and improved, thereby improving the accuracy of recognition.

[0079] In another illustrated embodiment, the acquired feature information may also be divided into mutually independent multidimensional sub-features, and multiple rounds of iterative recognition may be performed on the type of scene applied by the camera according to the sub-features of each dimension, wherein the input in the first round of iterative recognition is the sub-feature of the first dimension, and the input in the non-first round of iterative recognition is the sub-feature of the corresponding dimension and the recognition result of the previous round of iterative recognition.

[0080] In an illustrated embodiment, a hidden Markov model may be used to perform multiple rounds of iterative identification on the type of the camera application scene according to the sub-features of each dimension in turn.

[0081] Figure 2 FIG. 1 is a schematic diagram of a scene classifier based on a hidden Markov model according to an exemplary embodiment of the present disclosure. Figure 2 The implementation process of the technical solution is described.

[0082] like Figure 2 As shown in , the multi-dimensional sub-features can be input into the pre-trained hidden Markov model to generate hidden state nodes corresponding to the sub-features of each dimension. Figure 2Only the hidden state nodes corresponding to sub-feature 1 and sub-feature 2 are shown in FIG. According to the content reflected by the feature information of sub-feature 1, the probabilities of different scene types to which the camera service calling behavior of the third-party application belongs can be inferred. For example, according to the content of sub-feature 1, the probabilities of the video call category, scanning category, high-definition picture quality photo category, video recording category and ordinary preview photo category to which the camera service calling behavior of the third-party application belongs can be inferred to be P11, P12, P13, P14 and P15 respectively, and the probability value of P11 is the largest, that is, according to the content of sub-feature 1, the probability of inferring that the application scene of the camera is the video call category is the largest. According to the sub-feature 1 corresponding to the first hidden state node, the type of the camera application scenario is iterated for one round, and the recognition result generated by the hidden state node is transferred to the second hidden state node through the transfer matrix. The sub-feature 2 on the second hidden state node is combined with the recognition result of the previous hidden state node to re-infer the probability of the video call class, scanning class, high-definition picture quality photo class, video class and ordinary preview photo class to which the camera service call behavior of the third-party application belongs, which are P21, P22, P23, P24 and P25 respectively, and the probability value of P25 is the largest, that is, according to the content of sub-feature 2, it is inferred that the probability of the camera application scenario being the ordinary photo preview class is the largest. According to the above calculation method, the recognition result generated by the hidden state node arranged at the end is the recognition result of the multiple rounds of iterative recognition. For example, the last hidden state node infers that the probability of the camera service call behavior belonging to the ordinary preview photo class is the highest, and the ordinary preview photo class is used as the final recognition result.

[0083] In an illustrated embodiment, the feature information of the calling behavior acquired in step 101 is divided into multi-dimensional sub-features with time sequence according to the time sequence relationship. Next, the specific meaning of each dimensional sub-feature used in this embodiment will be described in detail:

[0084] Feature 1: Behavioral features of camera activity. This refers to the specific activities initiated on the application interface when a third-party application uses the camera function, such as taking photos, recording videos, or scanning codes.

[0085] Feature 2: Camera stream configuration information refers to the configuration parameters of the camera stream (image or video) when the third-party application uses the camera function. These configuration parameters include the configuration of key information such as the size, quantity, and encoding format of the camera stream. Through these camera stream configuration information, the type of camera application scenario corresponding to the current third-party application's calling behavior can be reflected. For example, when a larger camera stream is obtained, it usually reflects that the camera application scenario may focus on the capture of high-quality images or videos; when multiple images or video streams are obtained, it usually reflects that the camera application scenario involves multi-angle shooting or depth information acquisition; if the encoding format of the camera stream supports efficient video encoding formats, such as H.265 or VP9, ​​it usually reflects that the camera application scenario involves applications with high-quality video transmission.

[0086] Feature 3: The data transmission method used by the codec and the process of building the GraphicBufferSource. For example, when a third-party application is detected to use the H.264 encoder for data streaming and the GraphicBufferSource for image processing and rendering, it is very likely that the camera application scenario involves real-time video communication. It should be noted that this is only a preliminary judgment, and the specific application scenario needs further analysis and confirmation.

[0087] Feature 4: Capture Blob Frame and Request MetaInformat ion. Capture Blob Frame may refer to a special type of image data obtained when taking a photo through the camera API in a camera application. Request MetaInformat ion may be additional information used to describe a camera request in a camera application. It contains various metadata related to the camera request to provide more context and parameter settings, such as camera parameters such as exposure time and focal length, shooting scene information, and other information such as shooting timestamp. By combining the capture blob frame and request metadata information, the application scenario of the camera can be inferred. For example, if the capture binary frame data has high resolution and low compression loss, and the metadata information contains parameters related to high-definition photography, it can be reflected that the camera application scenario involves high-definition photography.

[0088] Feature 5: Camera Stream Usecase Configuration. This refers to configuring and defining the use cases and scenarios of the camera stream in third-party camera applications. For example, third-party applications can configure different camera stream use cases such as preview stream, photo stream, and video stream. Based on the characteristics of the camera stream use case, the types of use scenarios reflected by the third-party camera application can be reflected.

[0089] It should be noted that, in addition to the above five sub-features provided in this embodiment, in fact, other feature information can be extracted from the calling behavior of the camera service of the operating system to reflect the type of camera application scenario, and the present disclosure is not limited to this.

[0090] Next, a method for identifying the type of camera application scenario corresponding to the calling behavior is described.

[0091] In an illustrated embodiment, the calling behavior can be identified by a pre-trained scene classifier to determine the type of camera application scene corresponding to the calling behavior. The scene classifier is trained by capturing historical data generated by the historical calling behavior of the camera service by a third-party application. By capturing the actual historical data generated by the historical calling behavior of the camera service by a third-party application as training samples, it can be ensured that the trained model is closer to the current actual classification scene, thereby improving the accuracy of classification.

[0092] In an illustrated embodiment, feature information for training a scenario classifier can be extracted from data generated by historical call behaviors by means of feature extraction, and the extracted feature information can be stored in a training sample database so that the training sample data in the database can be reused to train the scenario classifier.

[0093] In an illustrated embodiment, the scene classifier may be updated according to the model update data sent by the cloud server. It should be noted that the model update data may be a new scene classifier to replace the current scene classifier, or may be an updated model parameter to update the model parameter of the current scene classifier.

[0094] Step 102: Determine a preset optimization strategy corresponding to the identified type to optimize the raw image data generated by the camera service in response to the calling behavior.

[0095] In an illustrated embodiment, a preset optimization strategy may be defined in the camera hardware abstraction layer. By defining the preset optimization strategy in the camera hardware abstraction layer, third-party applications do not need to understand the underlying implementation details and can directly use the preset optimization strategy to optimize the original image data.

[0096] In an illustrated embodiment, the original image data includes: preview image data and / or captured image data.

[0097] In this embodiment, the camera application scenes are divided into different types according to the general requirements of third-party applications calling camera functions, and corresponding preset optimization strategies are designed according to the optimization requirements of different scene types. The technical solution of this embodiment can solve the optimization pain points of third-party camera applications. Third-party application developers can automatically identify the scene type of the camera application of the third-party application when the third-party application calls the camera service of the operating system without changing the original camera API calling program, and automatically execute the corresponding preset optimization strategy according to the scene type.

[0098] Figure 3 It is a schematic diagram of a system architecture for identifying camera application scenarios and realizing image data optimization according to an exemplary embodiment of the present disclosure.

[0099] like Figure 3 As shown, at the user layer, when the third-party application 310 of the electronic device 30 calls the camera service of the operating system, in response to the calling behavior of the third-party application 310 for the camera service of the operating system, the camera service layer obtains feature information for describing the calling behavior. At the same time, the historical data generated by the historical calling behavior of the third-party application 310 for the camera service is uploaded to the cloud server 40, and the cloud server 40 extracts the feature information through the feature information extraction unit 410 as a training sample for the scene classifier 320, wherein the extracted feature information can be stored in a training sample database (not shown in the figure) so that the training sample data in the database can be reused to train the scene classifier 320.

[0100] The cloud server 40 sends the scene classifier 320 to the electronic device 30 via the cloud 420, and the electronic device 30 deploys the scene classifier 320 in the camera service layer. In addition, the scene classifier 320 can be updated according to the model update data sent by the cloud server.

[0101] Based on the acquired feature information, the scene classifier 320 identifies the type of camera application scene corresponding to the calling behavior, and passes the result of the identified camera application scene type to the camera hardware abstraction layer. The camera hardware abstraction layer determines the preset optimization strategy 330 corresponding to the identified type, and calls the chip 340 of the hardware layer according to the preset optimization strategy 330 to optimize the original image data generated by the calling behavior.

[0102] Through this embodiment, for all different third-party applications that call the operating system's camera service, the type of camera application scenario corresponding to the calling behavior is automatically identified, and the original image data generated by the calling behavior is optimized according to the preset optimization strategy corresponding to the identified scene type.

[0103] For the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that the present disclosure is not limited by the described order of actions, because according to the present disclosure, certain steps can be performed in other orders or simultaneously.

[0104] Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0105] Corresponding to the aforementioned embodiment of an image data optimization method, the present disclosure also provides an embodiment of an image data optimization device and a corresponding terminal.

[0106] like Figure 4 As shown, Figure 4 1 is a block diagram of an image data optimization device according to an exemplary embodiment of the present disclosure. The device may include:

[0107] An identification module 402 is used to identify the type of camera application scenario corresponding to the calling behavior of the camera service of the operating system in response to the calling behavior of the third-party application;

[0108] The optimization module 404 is used to determine a preset optimization strategy corresponding to the identified type to optimize the raw image data generated by the camera service in response to the calling behavior.

[0109] Optionally, the identification module 402 is specifically configured to:

[0110] Acquire characteristic information used to describe the calling behavior, and identify the type of camera application scenario corresponding to the calling behavior based on the acquired characteristic information.

[0111] Optionally, the identification module 402 is specifically configured to:

[0112] A data stream sent by the third-party application to the camera hardware abstraction layer of the operating system is obtained, and the feature information is extracted from the obtained data stream.

[0113] Optionally, the preset optimization strategy is defined in the camera hardware abstraction layer.

[0114] Optionally, the feature information includes multi-dimensional sub-features with time sequence; the identification module 402 is specifically used for:

[0115] According to the temporal sequence between the multi-dimensional sub-features, the type of the camera application scene is iteratively identified in multiple rounds according to the sub-features of each dimension;

[0116] The input for the first round of iterative recognition is the sub-features of the first dimension, and the input for non-first round of iterative recognition is the sub-features of the corresponding dimension and the recognition result of the previous round of iterative recognition.

[0117] Optionally, the identification module 402 is specifically configured to:

[0118] Inputting the multi-dimensional sub-features into a pre-trained hidden Markov model to generate hidden state nodes corresponding to the sub-features of each dimension respectively;

[0119] At each hidden state node, a round of iteration is performed on the type of the camera application scene according to the sub-feature corresponding to the hidden state node, and the recognition result generated by the previous hidden state node is transferred to the next hidden state node through the transfer matrix;

[0120] Among them, the recognition result generated by the hidden state node arranged at the end is the recognition result of the multiple rounds of iterative recognition.

[0121] Optionally, the multidimensional sub-features include:

[0122] Behavioral characteristics of camera activity;

[0123] Configuration information of the camera stream, including the size, number and encoding format of the camera stream;

[0124] The data transfer method used by the codec and the process of building the graphics buffer source;

[0125] Take a picture of a binary frame and request metadata information;

[0126] Camera streaming use case setup.

[0127] Optionally, the identification module 402 is specifically configured to:

[0128] Identifying the calling behavior through a pre-trained scene classifier to determine the type of camera application scene corresponding to the calling behavior;

[0129] The scene classifier is trained by capturing historical data generated by historical calling behaviors of the camera service by a third-party application.

[0130] Optionally, the device further includes a data update module (not shown in the figure), which is used to update the scene classifier according to the model update data sent by the cloud server.

[0131] Optionally, the original image data includes: preview image data and / or captured image data.

[0132] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0133] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0134] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. The electronic device 30 may be a user device, which may specifically be a mobile phone, a computer, a digital broadcast terminal, a message transceiver device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, a wearable device such as a smart watch, a smart glasses, a smart bracelet, a smart running shoe, etc.

[0135] like Figure 5 As shown, the electronic device 30 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .

[0136] The processing component 502 generally controls the overall operation of the electronic device 30, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.

[0137] The memory 504 is configured to store various types of data to support operations on the electronic device 30. Examples of such data include instructions for any application or method operating on the electronic device 30, contact data, phone book data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0138] The power supply component 506 provides power to the various components of the electronic device 30. The power supply component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 30.

[0139] The multimedia component 508 includes a screen that provides an output interface between the above-mentioned electronic device 30 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The above-mentioned touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the above-mentioned touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the electronic device 30 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0140] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the electronic device 30 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.

[0141] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0142] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the electronic device 30. For example, the sensor assembly 514 can detect the open / closed state of the electronic device 30, the relative positioning of the components, such as the display and keypad of the electronic device 30, and the sensor assembly 514 can also detect the position change of the electronic device 30 or a component of the electronic device 30, the presence or absence of user contact with the electronic device 30, the orientation or acceleration / deceleration of the electronic device 30, and the temperature change of the electronic device 30. The sensor assembly 514 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0143] The communication component 516 is configured to facilitate wired or wireless communication between the electronic device 30 and other devices. The electronic device 30 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G LTE, 5G NR or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0144] In an exemplary embodiment, the electronic device 500 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above methods.

[0145] The present disclosure also provides an executable non-transitory computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any of the above-mentioned image data optimization methods provided by the present disclosure are implemented. The non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0146] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0147] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for optimizing image data, characterized in that: The method comprises: In response to a calling behavior of a third-party application on a camera service of the operating system, identifying a type of a camera application scenario corresponding to the calling behavior; A preset optimization strategy corresponding to the identified type is determined to optimize the raw image data generated by the camera service in response to the calling behavior.

2. The method according to claim 1, characterized in that The identifying the type of camera application scenario corresponding to the calling behavior includes: Acquire characteristic information for describing the calling behavior; Based on the acquired feature information, the type of camera application scenario corresponding to the calling behavior is identified.

3. The method according to claim 2, characterized in that The acquiring of characteristic information for describing the calling behavior includes: A data stream sent by the third-party application to the camera hardware abstraction layer of the operating system is obtained, and the feature information is extracted from the obtained data stream.

4. The method according to claim 3, characterized in that: The preset optimization strategy is defined in the camera hardware abstraction layer.

5. The method according to claim 2, characterized in that: The feature information includes multi-dimensional sub-features with temporal characteristics; The identifying the type of camera application scenario corresponding to the calling behavior includes: According to the temporal sequence between the multi-dimensional sub-features, the type of the camera application scene is iteratively identified in multiple rounds according to the sub-features of each dimension; The input for the first round of iterative recognition is the sub-features of the first dimension, and the input for non-first round of iterative recognition is the sub-features of the corresponding dimension and the recognition result of the previous round of iterative recognition.

6. The method according to claim 5, characterized in that The performing multiple rounds of iterative identification on the type of the camera application scene according to the sub-features of each dimension in turn includes: Inputting the multi-dimensional sub-features into a pre-trained hidden Markov model to generate hidden state nodes corresponding to the sub-features of each dimension respectively; At each hidden state node, a round of iteration is performed on the type of the camera application scene according to the sub-features corresponding to the hidden state node, and the recognition result generated by the previous hidden state node is transferred to the next hidden state node through the transfer matrix; Among them, the recognition result generated by the hidden state node arranged at the end is the recognition result of the multiple rounds of iterative recognition.

7. The method according to claim 5, characterized in that The multidimensional sub-features include: Behavioral characteristics of camera activity; Configuration information of the camera stream, including the size, number and encoding format of the camera stream; The data transfer method used by the codec and the process of building the graphics buffer source; Take a picture of a binary frame and request metadata information; Camera streaming use case setup.

8. The method according to claim 1, characterized in that The identifying the type of camera application scenario corresponding to the calling behavior includes: Identifying the calling behavior through a pre-trained scene classifier to determine the type of camera application scene corresponding to the calling behavior; The scene classifier is trained by capturing historical data generated by historical calling behaviors of the camera service by a third-party application.

9. The method according to claim 8, characterized in that The method further comprises: The scene classifier is updated according to the model update data sent by the cloud server.

10. The method according to claim 1, characterized in that The original image data includes: preview image data and / or captured image data.

11. An image data optimization device, characterized in that: The device comprises: an identification module, configured to identify, in response to a calling behavior of a third-party application for a camera service of an operating system, a type of a camera application scenario corresponding to the calling behavior; The optimization module is used to determine a preset optimization strategy corresponding to the identified type to optimize the raw image data generated by the camera service in response to the calling behavior.

12. An electronic device, characterized in that: include: processor; a memory for storing a computer program executable by the processor; The processor is configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images