Image processing apparatus, method, image pickup apparatus, and electronic device

Abstract

The application relates to the technical field of image processing, and provides an image processing device, method, camera device and electronic equipment. The device comprises a view unit, a view model unit, a first renderer, a second renderer and a filter drawing unit. The view unit comprises an interface main class, the interface main class comprises N display interface fragments, each display interface fragment comprises a self-defined control; the view model unit comprises a view model base class, the view model base class comprises M view models, and the M view models are bound to the N display interface fragments; the first renderer receives layout information of the display interface fragments through the self-defined control; the first renderer obtains an image to be displayed, and renders the image to be displayed based on the layout information to obtain rendering data of the image to be displayed; and the filter drawing unit receives the rendering data, and calls the second renderer to perform shading processing on the rendering data to obtain the image to be displayed. The device can improve the image processing efficiency.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing apparatus, method, camera device, and electronic device. Background Technology

[0002] Traditional camera software typically uses the Model-View-Controller (MVC) architecture and is developed in conjunction with Google's Camera Application Programming Interface (API). This architecture has the following problems:

[0003] In the MVC architecture, the controller layer interface and logic are coupled, which is detrimental to later maintenance. As business logic increases, the code in the controller layer will continue to grow, potentially exceeding 3000 lines of code in a single controller class, placing a significant burden on developers.

[0004] It is developed entirely using the Camera API provided by Google. Although it can control the camera stream, its scalability is limited and it cannot perform special effects and texture operations such as adding watermarks and beautification. Summary of the Invention

[0005] In view of this, embodiments of this application provide an image processing apparatus, method, camera device, and electronic device to solve the problem that cameras in the prior art are not flexible and convenient enough when performing image processing.

[0006] A first aspect of this application provides an image processing apparatus, comprising:

[0007] View unit, view model unit, first renderer, second renderer, and filter drawing unit;

[0008] The view unit includes a main interface class, which includes N display interface fragments. Each display interface fragment includes a custom control, where N is a positive integer.

[0009] The view model unit includes a view model base class, which includes M view models. The M view models are bound to N display interface fragments. The view models are used to handle the representation logic and state of the display interface fragments, where M is a positive integer less than or equal to N.

[0010] The first renderer receives layout information for displaying interface fragments through a custom control;

[0011] The first renderer obtains the image to be displayed;

[0012] The first renderer renders the image to be displayed based on the layout information, and obtains the rendering data of the image to be displayed;

[0013] The filter drawing unit receives the rendering data and calls the second renderer to perform color processing on the rendering data to obtain the image to be displayed.

[0014] A second aspect of this application provides an image processing method, including:

[0015] User input is obtained from the view unit;

[0016] The view model is determined based on user input, and then the display interface fragment corresponding to the view model is determined.

[0017] The layout information of the display interface fragments is sent to the first renderer via a custom control;

[0018] The first renderer obtains the image to be displayed;

[0019] The first renderer renders the image to be displayed based on the layout information, and obtains the rendering data of the image to be displayed;

[0020] The filter drawing unit receives the rendering data and calls the second renderer to perform color processing on the rendering data to obtain the image to be displayed.

[0021] A third aspect of this application provides a camera device, including the image processing device described above.

[0022] A fourth aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0023] The beneficial effects of this application embodiment compared with the prior art are as follows: The image processing device provided in this application embodiment sets up a view unit, a view model unit, a first renderer, a second renderer, and a filter drawing unit. The view unit provides display interface fragments, the view model unit provides corresponding view models, custom controls communicate with the first renderer, the first renderer sends the processed rendering data to the filter drawing unit, the filter drawing unit calls the second renderer to color the rendering data, and finally displays it. This achieves decoupling between the view and the view model, reduces the complexity of code processing for developers, and improves image processing efficiency. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram illustrating an application scenario of this application.

[0026] Figure 2 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application.

[0027] Figure 3 This is a schematic diagram of the structure of the view unit provided in the embodiment of this application.

[0028] Figure 4 This is a schematic diagram of the structure of the view model unit provided in the embodiment of this application.

[0029] Figure 5 This is a schematic diagram of the software framework of a camera that includes the image processing apparatus provided in the embodiments of this application.

[0030] Figure 6 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application.

[0031] Figure 7 This is a schematic flowchart of the image processing method improved in the embodiments of this application.

[0032] Figure 8 This is a schematic diagram of the camera device provided in the embodiments of this application.

[0033] Figure 9 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0034] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0035] An image processing apparatus and method according to embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0036] Figure 1This is a schematic diagram illustrating an application scenario according to an embodiment of this application. The application scenario may include terminal devices 1, 2, and 3, server 4, and network 5.

[0037] Terminal devices 1, 2, and 3 can be hardware or software. When terminal devices 1, 2, and 3 are hardware, they can be various electronic devices with displays and supporting communication with server 4, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 1, 2, and 3 are software, they can be installed in the electronic devices described above. Terminal devices 1, 2, and 3 can be implemented as multiple software programs or software modules, or as a single software program or software module; this application embodiment does not impose any limitations on this. Furthermore, terminal devices 1, 2, and 3 can include a camera device, and the system of terminal devices 1, 2, and 3 can have built-in camera software to process images captured by the camera device. Even further, terminal devices 1, 2, and 3 can have various applications installed, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0038] Server 4 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 4 can be a single server, a server cluster consisting of several servers, or a cloud computing service center; this embodiment of the application does not impose any limitations on these aspects.

[0039] It should be noted that server 4 can be either hardware or software. When server 4 is hardware, it can be various electronic devices that provide various services to terminal devices 1, 2, and 3. When server 4 is software, it can be multiple software programs or software modules that provide various services to terminal devices 1, 2, and 3, or it can be a single software program or software module that provides various services to terminal devices 1, 2, and 3. This application embodiment does not impose any limitations on this.

[0040] Network 5 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), and Infrared. This application embodiment does not limit this.

[0041] Users can establish a communication connection with server 4 via network 5 through terminal devices 1, 2, and 3 to receive or send information. Specifically, when users use the camera devices on terminal devices 1, 2, and 3 to capture images, the acquired camera stream data can be processed locally on terminal devices 1, 2, and 3, or sent to server 4 for processing via network 5. After the camera stream data is processed, the processed image can be displayed on the terminal interface.

[0042] It should be noted that the specific types, quantities, and combinations of terminal devices 1, 2, and 3, server 4, and network 5 can be adjusted according to the actual needs of the application scenario, and this application embodiment does not impose any restrictions on this.

[0043] As mentioned above, traditional camera software typically uses the MVC architecture. The control layer interface and logic in the MVC architecture are coupled, which is detrimental to later maintenance. Furthermore, traditional camera software is usually developed using Google's Camera API, which, while allowing control of the camera stream, has limited extensibility and cannot implement watermarking, beautification, or other special effects and texture manipulation.

[0044] In view of this, embodiments of this application provide an image processing apparatus, which sets up a view unit, a view model unit, a first renderer, a second renderer, and a filter drawing unit. The view unit provides a display interface fragment, the view model unit provides a corresponding view model, a custom control communicates with the first renderer, the first renderer sends the processed rendering data to the filter drawing unit, the filter drawing unit calls the second renderer to color the rendering data, and finally displays it. This achieves decoupling between the view and the view model, reduces the complexity of code processing for developers, and improves image processing efficiency.

[0045] Figure 2 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. Figure 2 As shown, the image processing apparatus includes:

[0046] View unit 201 includes a main interface class, which includes N display interface fragments, each of which includes a custom control.

[0047] Where N is a positive integer.

[0048] The view model unit 202 includes a view model base class, which includes M view models. The M view models are bound to N display interface fragments. The view models are used to handle the representation logic and state of the display interface fragments.

[0049] Where M is a positive integer less than or equal to N.

[0050] The first renderer 203 receives layout information of a display interface fragment through a custom control; the first renderer obtains an image to be displayed; the first renderer renders the image to be displayed based on the layout information to obtain rendering data of the image to be displayed.

[0051] The filter drawing unit 204 and the second renderer 205 receive rendering data and call the second renderer to perform color processing on the rendering data to obtain the image to be displayed.

[0052] In this embodiment, the image processing device can be a device within camera software. Unlike the MVC architecture used in traditional camera software, the image processing device provided in this embodiment is implemented based on a Model-View-ViewModel (MVVM) architecture. The MVVM architecture includes a model layer, a view layer, and a view model layer. The model layer encapsulates the business logic and data of the camera application, the view layer encapsulates the user interface and user interface logic, and the view model layer encapsulates representation logic and state. The view layer can bind to the view model layer, execute commands, and request corresponding actions from the view model layer. The view model layer can communicate with the model layer, informing it to update in response to user interface operations. The image processing device provided in this embodiment is mainly used to process images corresponding to the user interface; therefore, this image processing device primarily focuses on the view layer and the view model layer.

[0053] In this embodiment, the view layer may include view units, each including a main interface class MainActivity. MainActivity may include N display interface fragments (Fragments), each Fragment including a custom control MyGLSurfaceView. Specifically, MainActivity may include a homepage display interface fragment (HomeFragment) and at least one image display interface fragment (PhotoFragment). HomeFragment may be the default interface of the camera application, and PhotoFragment may be the interface that the camera application needs to present to the user in response to different user operations on the camera application.

[0054] In this embodiment, the view model layer may include view model units, each including a view model base class `BaseViewModel`. Each `BaseViewModel` includes M `FragmentViewModels`, which are bound to the aforementioned N `Fragments`. In one example, `BaseViewModel` may include a `HomeFragmentViewModel`, which is bound to a `HomeFragment` in `MainActivity`. In another example, `BaseViewModel` may also include at least one `PhotoFragmentViewModel`, each of which is bound to one or more `PhotoFragments` in `MainActivity`. In other words, a `FragmentViewModel` can be bound to one or more different `Fragments`.

[0055] In this embodiment, the FragmentViewModel can be used to process the representation logic and state of a Fragment. The representation logic and state of a Fragment can be other business logic and state data in the view besides those related to visual perception. By processing the representation logic and state of a Fragment at the FragmentViewModel, the cohesion of the image processing device can be improved.

[0056] In this embodiment, the first renderer MyRender can receive the layout information of the Fragment through the aforementioned custom control MyGLSurfaceView. Furthermore, MyRender can also obtain the image to be displayed, and render the image based on the obtained layout information to obtain the rendering data of the image to be displayed.

[0057] Specifically, MyRender can be integrated into MyGLSurfaceView to receive Fragment layout information. On the other hand, MyRender can also obtain images to be displayed via specified APIs, including the homepage image of the camera software and other images that need to be displayed on the user interface in response to user actions. MyRender senses changes and rendering of images through the specified APIs, and based on these changes and rendering, converts the image stream obtained from the camera device into frame-by-frame image data, i.e., textures, and then transmits the texture identifier and matrix data to the FilterDraw unit.

[0058] In this embodiment of the application, FilterDraw can receive the rendering data processed by MyRender and call the second renderer OpenGL to perform color processing on the rendering data to obtain the image to be displayed.

[0059] According to the technical solution provided in the embodiments of this application, by setting up a view unit, a view model unit, a first renderer, a second renderer, and a filter drawing unit, the view unit provides a display interface fragment, the view model unit provides a corresponding view model, the custom control communicates with the first renderer, the first renderer sends the processed rendering data to the filter drawing unit, the filter drawing unit calls the second renderer to color the rendering data, and finally displays it, thereby achieving decoupling between the view and the view model, reducing the complexity of code processing for developers, and improving image processing efficiency.

[0060] Figure 3 This is a schematic diagram of the structure of the view unit provided in an embodiment of this application. For example... Figure 3 As shown, this view unit includes a base class `BaseActivity` and a main class `MainActivity`, where `MainActivity` is a subclass of `BaseActivity`. `BaseActivity` encapsulates basic methods for displaying Fragments, while `MainActivity` encapsulates specific methods for displaying each Fragment within a Fragment. In one example, `BaseActivity` can encapsulate some commonly used methods, such as binding a ViewModel and encapsulating navigation methods from the Navigation utility library. `MainActivity` can be used to implement methods not implemented in `BaseActivity` and display the actual interface.

[0061] Furthermore, MainActivity can include the Navigation utility library, which is encapsulated within MainActivity. This allows MainActivity to use the configuration file within Navigation to navigate between different Fragments. In one example, the configuration file within Navigation could be navi_graf.xml.

[0062] Furthermore, MainActivity can also include one or more display interface fragments (Fragments), each configured with a custom control, MyGLSurfaceView. MyGLSurfaceView is an implementation class of GLSurfaceView, a control officially provided by Android. MyGLSurfaceView combines the functionality of OpenGL and SurfaceView, allowing for both OpenGL image rendering and SurfaceView display.

[0063] By adopting the technical solution of this application embodiment, common methods are encapsulated in BaseActivity and specific methods are encapsulated in MainActivity in the view unit, which can improve processing efficiency; at the same time, by configuring MyGLSurfaceView controls in each Fragment of MainActivity, image rendering and display can be realized conveniently and quickly, further improving processing efficiency.

[0064] Figure 4 This is a schematic diagram of the structure of the view model unit provided in an embodiment of this application. For example... Figure 4 As shown, this view model unit includes the view model base class `BaseViewModel` and the view model `ViewModel`. `BaseViewModel` is the base class for `ViewModel`, encapsulating the common methods of `ViewModel`. `ViewModel` is responsible for handling the code logic of the Fragment, including time-consuming tasks such as code logic processing and thread creation, all of which can be implemented in `ViewModel`.

[0065] In this embodiment of the application, by using ViewModel, the code of Fragment can be fully decoupled. Time-consuming tasks such as code logic processing and thread opening operations are implemented in this class through ViewModel, while Fragment is only responsible for processing the interface. This division of labor and layering can greatly decouple Fragment and reduce the amount of code in Fragment by a factor of two.

[0066] In this embodiment, MyRender can obtain the image to be displayed in the following ways: in response to the image to be displayed being a fixed page image, MyRender obtains a pre-stored fixed page image as the image to be displayed; in response to the image to be displayed being an image captured in real time by a camera device, MyRender obtains the data stream of the camera device through a preset application programming interface and converts the data stream into the image to be displayed.

[0067] In other words, MyRender can first determine whether the image to be displayed is a static page image or an image captured in real-time by the camera. For example, if the camera app has just been opened, or the user has performed a "return to home" operation on the camera app interface, the image to be displayed is the home page image of the camera app. In this case, MyRender can directly retrieve the pre-stored static page image from memory or cache as the image to be displayed. If the user has performed a "take a picture" operation on the camera app interface, the image to be displayed is an image captured in real-time by the camera. In this case, MyRender can obtain the data stream from the camera through a preset application programming interface (API) and convert the data stream into the image to be displayed. The preset API can be the Camera2 API.

[0068] In this embodiment of the application, the second renderer OpenGL can be called to perform shading processing on the rendering data to obtain the image to be displayed in the following ways: call OpenGL to perform vertex shading processing and fragment shading processing on the rendering data to obtain the image to be displayed; or call OpenGL to perform vertex shading processing and fragment shading processing on the rendering data, and add special effects processing during fragment shading processing to obtain the image to be displayed.

[0069] In other words, FilterDraw can call OpenGL to perform vertex and fragment shading on the rendered data to obtain the image to be displayed. Furthermore, when calling OpenGL to perform fragment shading on the rendered data, special effects can be added, such as adding filters or eye-enlarging effects, to obtain the image to be displayed. The vertex shader can be Camera_vertex.glsl, and the fragment shader can be Camera_fragment.glsl. Camera_vertex.glsl is responsible for determining the display position coordinates, while Camera_fragment.glsl is responsible for displaying the image on the screen.

[0070] As previously stated, the image processing apparatus provided in this application embodiment is implemented based on the MVVM architecture. Therefore, the image processing apparatus may further include a model unit. Furthermore, the layout information of the Fragment can be determined in the following manner: the view unit receives user input and sends the user input to the view model unit; the view model unit sends the user input to the model unit and receives the data and commands to be bound from the model unit; the view model unit determines the view model based on the data and commands to be bound; the view unit determines the corresponding display interface fragment based on the view model, and then determines the layout information of the display interface fragment.

[0071] In other words, the view unit can receive user input from the camera application interface and send the user input to the view model unit. The view model unit then sends the user input to the model unit. After receiving the user input, the model unit determines the data and commands that need to be bound in response to this user input operation and sends this data and commands to the view model unit. The view model unit determines the ViewModel based on the received data and commands. Since the ViewModel is bound to the Fragment, the view unit can determine the corresponding displayed Fragment based on the ViewModel, and thus determine the Fragment's layout information.

[0072] In this embodiment of the application, both the first renderer and the second renderer use the OpenGL library to implement image rendering and shading.

[0073] Figure 5 This is a schematic diagram of the software framework of a camera that includes the image processing apparatus provided in the embodiments of this application. For example... Figure 5 As shown, this camera is implemented based on the MVVM architecture, which consists of a Model layer (M), a View layer (V), and a View-Model layer (VM). The M layer encapsulates the application's business logic and data, the V layer encapsulates the user interface and user interface logic, and the VM layer encapsulates the presentation logic and state.

[0074] A View can be bound to a ViewModel and then execute commands to request an action from it. Conversely, the ViewModel can also communicate with the Model, telling it to update in response to the user interface. This makes building user interfaces for an application very easy. The easier it is to attach an interface to an application, the easier it is for a visual designer to create a beautiful interface. At the same time, the more loosely coupled the user interface and functionality become, the more testable the functionality becomes.

[0075] Furthermore, the camera's software architecture is developed using the Kotlin language. Traditional Android development uses Java, which was released 20 years ago. Although it has been updated, it must be compatible with previous Java versions, making it inflexible and limiting its use. Kotlin is the officially recommended language for Android, fully leveraging the language design expertise accumulated over the past 20 years since Java's inception. Secondly, it is packed with all the much-needed modern features that Java mobile developers have long desired, a programming language whose efficiency has been proven through large projects. Developing using the Kotlin language offers the following advantages: high efficiency and familiar development tools designed to improve developer productivity; Kotlin has a good compiler; seamless integration with existing infrastructure due to its compatibility with all Java frameworks and libraries, and easy integration with Marven and Gradle build systems; and enhanced runtime performance.

[0076] Furthermore, the camera's software architecture employs Navigation to manage Fragments. The Navigation architecture pattern, included in Google's Android Jetpack toolkit, is a new architectural pattern for managing Fragments. Navigation makes it easy to implement scenarios where an Activity manages multiple fragment display pages, which is much more convenient than the traditional FragmentManager approach.

[0077] Meanwhile, the camera's software architecture uses Camera2 for camera streaming, instead of Google's Camera1 API, because Camera1's methods are relatively limited and cannot effectively handle multi-stream issues when multiple cameras are streaming simultaneously. Google's latest Camera2, however, uses a pipeline approach to handle streaming, providing more flexible solutions for multi-camera problems. Furthermore, the camera's software architecture utilizes OpenGL, a tool library for 2D and 3D vector graphics rendering. A custom renderer is created using the GLSurface control, where frame data from the audio and video streams is converted into textures, which are then fed into OpenGL to achieve specific display effects such as enlarged eyes, filters, special effects, and beautification.

[0078] Figure 6 This is a schematic diagram of the structure of the image processing apparatus provided in an embodiment of this application. Figure 6As shown, the image processing device includes a portion contained in the interface BaseActivity, a portion contained in the view model BaseViewModel, a portion contained in the renderer MyRender, and a portion contained in the rendering display OpenGL.

[0079] The base class `BaseActivity` is the foundation of the UI, encapsulating commonly used methods such as binding the view model (`ViewModel`) and encapsulating navigation methods from the `Navigation` utility library. The main class `MainActivity` is a subclass of `BaseActivity`, responsible for implementing the unimplemented methods in `BaseActivity` and displaying the actual UI.

[0080] The main UI class MainActivity integrates a Navigation utility library. Navigation manages the navigation between various UI fragments (Fragments). The navigation between multiple Fragments is established through the navi_graf.xml configuration file.

[0081] The HomeFragment and PhotoFragment are UI fragments within MainActivity, responsible for displaying specific business interfaces.

[0082] The BaseViewModel class is the base class for ViewModel. It encapsulates the common methods of ViewModel. ViewModel is responsible for handling the code logic of Fragment. For example, time-consuming tasks such as code logic processing and thread creation are implemented in this class.

[0083] Using HomeFragmentViewModel here can fully decouple the HomeFragment code. By using HomeFragmentViewModel, code logic processing or time-consuming tasks such as threaded operations can be implemented in this class, while HomeFragment is only responsible for handling the interface. This division of labor and layering can greatly decouple HomeFragment, reducing the amount of code in HomeFragment by a factor of two.

[0084] Both HomeFragment and PhotoFragment contain a custom control, MyGLSurfaceView. MyGLSurfaceView is an implementation class of GLSurfaceView. The MyGLSurfaceView control is an official Android control that combines the functionality of OpenGL and SurfaceView, allowing for both OpenGL image rendering and SurfaceView display.

[0085] MyGLSurfaceView internally integrates the first renderer, MyRender. MyRender is responsible for rendering the image. It perceives changes in the image and draws it through a specified interface. It requires the Camera2 stream to be present, converting the image stream into frame-by-frame image data, i.e., textures, and then passing the texture identifier and matrix data to FilterDraw.

[0086] MyGLSurfaceView and FilterDraw are closely related. The FilterDraw class can be used to integrate OpenGL and prepare the parameters required by OpenGL.

[0087] OpenGL is a graphics card language. All effects are ultimately achieved through OpenGL, which is divided into vertex shaders and fragment shaders. The vertex shader can be `Camera_vertex.glsl`, and the fragment shader can be `Camera_fragment.glsl`. `Camera_vertex.glsl` is responsible for determining the display position coordinates; `Camera_fragment.glsl` is responsible for displaying the image on the screen. Features like zooming in and out and filters are typically implemented in the fragment shader.

[0088] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0089] The following are embodiments of the method of this application, which can be executed by the device embodiments of this application. For details not disclosed in the method embodiments of this application, please refer to the device embodiments of this application.

[0090] Figure 7 This is a schematic flowchart of the image processing method improved in the embodiments of this application. Figure 7 The method shown can be derived from Figure 2 The image processing device shown performs the operation. Figure 7 As shown, the image processing method includes the following steps:

[0091] In step S701, the self-view unit acquires user input.

[0092] In step S702, a view model is determined based on user input, and then a display interface segment corresponding to the view model is determined.

[0093] In step S703, the layout information of the display interface fragment is sent to the first renderer via a custom control.

[0094] In step S704, the first renderer acquires the image to be displayed.

[0095] In step S705, the first renderer renders the image to be displayed based on the layout information to obtain the rendering data of the image to be displayed.

[0096] In step S706, the filter drawing unit receives the rendering data and calls the second renderer to perform color processing on the rendering data to obtain the image to be displayed.

[0097] According to the technical solution provided in the embodiments of this application, by setting up a view unit, a view model unit, a first renderer, a second renderer, and a filter drawing unit, the view unit provides a display interface fragment, the view model unit provides a corresponding view model, the custom control communicates with the first renderer, the first renderer sends the processed rendering data to the filter drawing unit, the filter drawing unit calls the second renderer to color the rendering data, and finally displays it, thereby achieving decoupling between the view and the view model, reducing the complexity of code processing for developers, and improving image processing efficiency.

[0098] In this embodiment of the application, the first renderer acquiring the image to be displayed may include: in response to the image to be displayed being a fixed page image, the first renderer acquiring a pre-stored fixed page image as the image to be displayed; in response to the image to be displayed being an image captured in real time by a camera device, the first renderer acquiring the data stream of the camera device through a preset application programming interface and converting the data stream into the image to be displayed.

[0099] In this embodiment of the application, calling the second renderer to perform color processing on the rendering data to obtain the image to be displayed may include: calling the second renderer to perform vertex color processing and fragment color processing on the rendering data to obtain the image to be displayed; or calling the second renderer to perform vertex color processing and fragment color processing on the rendering data, and adding special effects processing during fragment color processing to obtain the image to be displayed.

[0100] In this embodiment of the application, the layout information of the display interface segment can be determined in the following manner: the view unit receives user input and sends the user input to the view model unit; the view model unit sends the user input to the model unit and receives data and commands to be bound from the model unit; the view model unit determines the view model based on the data and commands to be bound; the view unit determines the corresponding display interface segment based on the view model, and then determines the layout information of the display interface segment.

[0101] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0102] Figure 8 This is a schematic diagram of the camera device provided in an embodiment of this application. Figure 8 As shown, the camera device may include, for example: Figure 2 or Figure 6 The image processing apparatus provided in the illustrated embodiment.

[0103] Figure 9 This is a schematic diagram of the electronic device provided in an embodiment of this application. For example... Figure 9 As shown, the electronic device 9 of this embodiment includes a processor 901, a memory 902, and a computer program 903 stored in the memory 902 and executable on the processor 901. When the processor 901 executes the computer program 903, it implements the steps in the various method embodiments described above. Alternatively, when the processor 901 executes the computer program 903, it implements the functions of each module / unit in the various device embodiments described above.

[0104] Electronic device 9 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 9 may include, but is not limited to, processor 901 and memory 902. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 9 and does not constitute a limitation on electronic device 9. It may include more or fewer components than shown, or different components.

[0105] The processor 901 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0106] The memory 902 can be an internal storage unit of the electronic device, such as a hard disk or RAM of the electronic device 9. The memory 902 can also be an external storage device of the electronic device 9, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the electronic device 9. The memory 902 can also include both internal and external storage units of the electronic device 9. The memory 902 is used to store computer programs and other programs and data required by the electronic device.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0108] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0109] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An image processing apparatus, characterized in that, The image processing device includes: View unit, view model unit, first renderer, second renderer, and filter drawing unit; The view unit includes a main interface class, which includes N display interface fragments, each display interface fragment including a custom control, where N is a positive integer; The view model unit includes a view model base class, which includes M view models. The M view models are bound to the N display interface fragments. The view models are used to process the representation logic and state of the display interface fragments, where M is a positive integer less than or equal to N. The first renderer receives layout information for the display interface fragment through the custom control; The first renderer acquires the image to be displayed; The first renderer renders the image to be displayed based on the layout information to obtain the rendering data of the image to be displayed; The filter drawing unit receives the rendering data and calls the second renderer to perform color processing on the rendering data to obtain the image to be displayed.

2. The apparatus according to claim 1, characterized in that, The view unit also includes: Interface base class; The main interface class is a subclass of the base interface class. The base interface class encapsulates basic methods for displaying the display interface fragments, and the main interface class encapsulates methods for displaying each display interface fragment within the display interface fragments.

3. The apparatus according to claim 1, characterized in that, The view unit also includes: Navigator Tool Library; The navigator toolkit is encapsulated in the main interface class, which uses the configuration file in the navigator toolkit to enable navigation between different display interface segments.

4. The apparatus according to claim 1, characterized in that, The first renderer acquires the image to be displayed, including: In response to the fact that the image to be displayed is a fixed page image, the first renderer obtains the pre-stored fixed page image as the image to be displayed; In response to the image to be displayed being an image captured in real time by a camera device, the first renderer obtains the data stream from the camera device through a preset application programming interface and converts the data stream into the image to be displayed.

5. The apparatus according to claim 1, characterized in that, The step of calling the second renderer to perform color processing on the rendering data to obtain the image to be displayed includes: The second renderer is invoked to perform vertex and fragment shading processing on the rendering data to obtain the image to be displayed; or The second renderer is invoked to perform vertex shading and fragment shading on the rendering data, and special effects are added during the fragment shading process to obtain the image to be displayed.

6. The apparatus according to claim 2, characterized in that, The device further includes a model unit, and the layout information of the display interface segment is determined in the following manner: The view unit receives user input and sends the user input to the view model unit; The view model unit sends the user input to the model unit and receives the data and commands to be bound from the model unit; The view model unit determines the view model based on the data and commands that need to be bound; The view unit determines the corresponding display interface segment based on the view model, and then determines the layout information of the display interface segment.

7. The apparatus according to any one of claims 1 to 6, characterized in that, The first and second renderers use the OpenGL library for image rendering and coloring.

8. An image processing method, performed by the apparatus according to any one of claims 1 to 7, characterized in that, The method includes: User input is obtained from the view unit; Based on the user input, a view model is determined, and then the display interface segment corresponding to the view model is determined; The layout information of the display interface fragment is sent to the first renderer via a custom control; The first renderer acquires the image to be displayed; The first renderer renders the image to be displayed based on the layout information to obtain the rendering data of the image to be displayed; The filter drawing unit receives the rendering data and calls the second renderer to perform color processing on the rendering data to obtain the image to be displayed.

9. A camera device, characterized in that, The camera device includes the image processing device as described in any one of claims 1 to 7.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in claim 8.

Images