Method for controlling camera device and camera device

Abstract

The invention discloses a method for controlling a camera device and the camera device, the camera device comprises an application program layer, a UVC protocol layer, a firmware layer and a bottom layer driver, the method comprises the following steps: obtaining a trigger instruction for a camera application program, and in response to the trigger instruction, performing encryption coding on the camera application program through the firmware layer; decoding the camera shooting application program through the application program layer, acquiring a first image containing a target object under the condition that the camera shooting application program passes decoding and verification, and calling a deep learning positioning and recognition model in the firmware layer to process the first image to obtain moving and zooming parameters of the camera shooting device; and the bottom layer drive adjusts the camera device based on the movement of the camera device and the zoom parameter, so that the camera device shoots a second image containing the target object. According to the invention, the problem of poor image display effect caused by fixed camera mounting angle, user posture, illumination or shielding and the like can be solved.

Description

Technical Field

[0001] The present application relates to the field of camera technology, and in particular to a method for controlling a camera device and a camera device. Background Art

[0002] Camera, also known as computer camera, computer eye, electronic eye, etc., is a video input device that is widely used in video conferencing, telemedicine and real-time monitoring. After the camera captures the image through the lens, the photosensitive component circuit and control component in the camera process the image and convert it into a digital signal that can be recognized by the computer, and then input it into the computer through the parallel port or USB connection, and the image is restored by the software.

[0003] The applicant discovered during research that existing camera control schemes are prone to problems with poor display effects of captured images due to factors such as fixed camera installation angles, user posture, lighting or occlusion. Summary of the invention

[0004] The present application discloses a method for controlling a camera device and a camera device, so as to solve the problem of poor image display effect caused by reasons such as fixed camera installation angle, user posture, lighting or occlusion.

[0005] In a first aspect, the present application discloses a method for controlling a camera device, wherein the camera device includes an application layer, a UVC protocol layer, a firmware layer, and an underlying driver, wherein the camera application layer is equipped with a camera application; the UVC protocol layer is provided with an information access channel to implement encrypted encoding and decoding signal transmission between the firmware layer and the application layer through the information access channel; the firmware layer is provided with a deep learning positioning and recognition model, and the method includes:

[0006] Acquire a trigger instruction for the camera application, and in response to the trigger instruction, encrypt and encode the camera application through the firmware layer;

[0007] Decoding the camera application in the application layer, obtaining a first image containing a target object when the camera application is decoded, and invoking the deep learning positioning and recognition model in the firmware layer to process the first image to obtain movement and zoom parameters of the camera device;

[0008] The bottom layer driver adjusts the camera device based on the movement and zoom parameters of the camera device so that the camera device captures a second image containing the target object.

[0009] By implementing the embodiments of the present application, the application layer, UVC protocol layer, firmware layer and underlying driver carried in the camera device cooperate with each other, so as to achieve tracking between the camera device and the target object, so as to solve the problem of poor image display effect caused by fixed camera installation angle, user posture, lighting or occlusion, etc.

[0010] In a possible implementation, calling the deep learning positioning and recognition model in the firmware layer to process the first image to obtain movement and zoom parameters of the camera device includes:

[0011] Acquire the location information of the target object through the deep learning positioning and recognition model;

[0012] Performing a golden section on the first image based on the position information, calculating a golden section screen ratio, and projecting the target object into the golden section screen ratio to obtain movement and zoom parameters of the camera device.

[0013] In one possible implementation, the deep learning positioning and recognition model includes at least one of a hand detection model, a gesture recognition model, a portrait detection model, and an AI composition model.

[0014] In a possible implementation manner, the camera device captures a second image containing the target object, including:

[0015] The camera device changes the picture of the first image to the golden ratio of the AI ​​composition based on the movement and zoom parameters to obtain the second image.

[0016] In a possible implementation, the method further includes:

[0017] The second image is beautified by an image processing algorithm, and the beautified second image is output.

[0018] In a second aspect, the present application provides a camera device, which includes an application layer, a UVC protocol layer, a firmware layer, and an underlying driver, wherein the camera application layer is equipped with a camera application; the UVC protocol layer is provided with an information access channel to implement encrypted encoding and decoding signal transmission between the firmware layer and the application layer through the information access channel; the firmware layer is provided with a deep learning positioning and recognition model, and the device includes:

[0019] An instruction acquisition module, used for acquiring a trigger instruction for the camera application, and in response to the trigger instruction, encrypting and encoding the camera application through the firmware layer;

[0020] An image acquisition and image processing module, which decodes the camera application in the application layer, acquires a first image containing a target object when the camera application is decoded, and calls the deep learning positioning and recognition model in the firmware layer to process the first image to obtain movement and zoom parameters of the camera device;

[0021] The adjustment module is used to drive the bottom layer to adjust the camera device based on the movement and zoom parameters of the camera device so that the camera device can capture a second image containing the target object.

[0022] In a possible implementation, the image acquisition and image processing module is specifically used to:

[0023] Acquire the location information of the target object through the deep learning positioning and recognition model;

[0024] Performing a golden section on the first image based on the position information, calculating a golden section screen ratio, and projecting the target object into the golden section screen ratio to obtain movement and zoom parameters of the camera device.

[0025] In one possible implementation, the deep learning positioning and recognition model includes at least one of a hand detection model, a gesture recognition model, a portrait detection model, and an AI composition model.

[0026] In a possible implementation, the camera device further includes a shooting module, which is used to:

[0027] The picture of the first image is changed to the golden ratio of the AI ​​composition based on the movement and zoom parameters to obtain the second image.

[0028] In a possible implementation, the camera device further includes an image beautification module, which is used to:

[0029] The second image is beautified by an image processing algorithm, and the beautified second image is output. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments.

[0031] Figure 1 A schematic diagram of the structure of an exemplary electronic device provided in an embodiment of the present application;

[0032] Figure 2 A software and hardware structure design diagram of a camera device provided in an embodiment of the present application;

[0033] Figure 3A flowchart of a method for controlling a camera device provided in an embodiment of the present application;

[0034] Figure 4 A schematic structural diagram of a camera device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] First, refer to Figure 1 An exemplary electronic device 100 for implementing the present invention is described below. Figure 1 As shown, the electronic device 100 includes one or more processors 102, one or more storage devices 104, an input device 106, and an output device 108. The electronic device 100 may also include a data acquisition device 110 and / or an image acquisition device (also referred to as a camera device) 112, and these components are interconnected via a bus system 114 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as required.

[0037] The processor 102 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0038] The storage device 104 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM, hard disk, flash memory), etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may run the program instructions to implement the client functions (implemented by the processor) in the embodiments of the present application described below and / or other desired functions. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0039] The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0040] The output device 108 may output various information (eg, images and / or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

[0041] Exemplarily, the electronic device 100 may include a data acquisition device 110. The data acquisition device 110 may acquire sample images (including video frames) and corresponding annotation data, and store the acquired sample images and annotation data in the storage device 104 for use by other components. For example, the data acquisition device 110 may include one or more of a wired or wireless network interface, a universal serial bus (USB) interface, an optical disk drive, and the like.

[0042] Exemplarily, the electronic device 100 may include an image acquisition device 112. The image acquisition device 112 may acquire images (for example, may include video frames or images), and store the acquired images in the storage device 104 for use by other components. The image acquisition device 112 may be a camera, and it should be understood that the image acquisition device 112 is only an example, and the electronic device 100 may not include the image acquisition device 112. In this case, other devices with image acquisition capabilities may be used to acquire images to be processed, and the acquired images may be sent to the electronic device 100.

[0043] Exemplarily, the exemplary electronic device provided in the present application may be implemented on a device such as a personal computer or a remote server.

[0044] like Figure 2 As shown in the figure, a software and hardware structure design diagram of a camera device provided by an embodiment of the present application may include an application layer, a UVC protocol layer, a firmware layer, and an underlying driver. The application layer may include a camera application and a virtual camera application, and may be equipped with functions such as beauty and cutout; the UVC protocol layer may be provided with an information access channel to implement encrypted encoding and decoding signal transmission between the firmware layer and the application layer through the information access channel; for example, Figure 2As shown, the UVC protocol layer may include a video control processing unit (Video Control Processing Unit), a video control extension unit (Video Control Extension Unit) (Setting Data), a video control input terminal (Video Control lnput Terminal), an audio control input terminal (Audio Control lnput Terminal), etc.; the firmware layer is provided with a graphics interface, a hardware interface, and a deep learning positioning and recognition model. Exemplarily, the deep learning positioning and recognition model may include one or more of a hand detection model, a gesture recognition model, a portrait detection model, and an AI composition model.

[0045] by Figure 2 As an example, the camera device shown in FIG. Figure 3 A method for controlling a camera device proposed in this application is specifically described, which may include but is not limited to the following steps:

[0046] Step S301: Acquire a trigger instruction for the camera application, and in response to the trigger instruction, encrypt and encode the camera application through the firmware layer.

[0047] In actual applications, the trigger instruction initiated for the camera application may be a touch operation performed by the user on the camera application, and the touch operation may include but is not limited to single click, double click, long press, etc.

[0048] Step S302: decode the camera application in the application layer, obtain a first image containing a target object when the camera application is decoded, and call the deep learning positioning and recognition model in the firmware layer to process the first image to obtain movement and zoom parameters of the camera device.

[0049] In practical applications, the camera application is decoded and verified by the encoding unit and the decoding unit in the UVC protocol layer to verify the legitimacy of the identity of the camera device. When the camera application passes the decoding and verification, or includes a first image of a target object, the object can be any object, such as a person, a car, a building, etc. In the following description, the present application will be described by taking the target object as an example, but this is not a limitation of the present application.

[0050] In one embodiment, calling the deep learning positioning and recognition model in the firmware layer to process the first image to obtain the movement and zoom parameters of the camera device may include: first, obtaining the position information of the target object through the deep learning positioning and recognition model. Exemplarily, the deep learning positioning and recognition model may include one or more of a hand detection model, a gesture recognition model, a portrait detection model, and an AI composition model; then, performing a golden section on the first image based on the position information, calculating the golden section screen ratio, and projecting the target object into the golden section screen ratio to obtain the movement and zoom parameters of the camera device.

[0051] Step S303: The bottom-level driver adjusts the camera device based on the movement and zoom parameters of the camera device, so that the camera device captures a second image containing the target object.

[0052] In one embodiment, the underlying driver may adjust the camera device based on the acquired movement and zoom parameters of the camera device so that the camera device can track the object and capture a second image containing the target object based on the adjusted movement and zoom parameters.

[0053] In one embodiment, the underlying driver may adjust the camera device based on the acquired movement and zoom parameters of the camera device, and change the picture of the first image to the golden ratio of the AI ​​composition based on the movement and zoom parameters to obtain the second image.

[0054] By implementing the embodiments of the present application, the application layer, UVC protocol layer, firmware layer and underlying driver carried in the camera device cooperate with each other, so as to achieve tracking between the camera device and the target object, so as to solve the problem of poor image display effect caused by fixed camera installation angle, user posture, lighting or occlusion, etc.

[0055] According to another aspect of the present application, another method for controlling a camera device is provided. After acquiring the second image, the second image can be further processed, for example, the second image can be beautified by an image processing algorithm, and the beautified second image can be output. Exemplarily, the application layer uses a deep learning model to locate the face position and key part coordinates, and combines OPENGL rendering to fine-tune the overall picture, including but not limited to picture color, human skin quality, face contour fine-tuning, image quality improvement, etc., wherein based on the face coordinates, adjustments can be made to a certain part such as the eyes, nose, and mouth to achieve the purpose of beauty.

[0056] Combination of the above Figure 1-Figure 3The method for controlling the camera device according to the embodiment of the present application is described in detail. In order to facilitate better implementation of the above-mentioned scheme of the embodiment of the present application, correspondingly, related devices and equipment for cooperating in implementing the above-mentioned scheme are also provided below.

[0057] See also Figure 4 , is a schematic diagram of the structure of a camera device 40 provided in an embodiment of the present application, which may include:

[0058] The instruction acquisition module 400 is used to acquire a trigger instruction for the camera application, and in response to the trigger instruction, encrypt and encode the camera application through the firmware layer;

[0059] The image acquisition and image processing module 402 decodes the camera application in the application layer, acquires a first image containing a target object when the camera application is decoded, and calls the deep learning positioning and recognition model in the firmware layer to process the first image to obtain movement and zoom parameters of the camera device;

[0060] The adjustment module 404, the bottom layer driver adjusts the camera device based on the movement and zoom parameters of the camera device, so that the camera device captures a second image containing the target object.

[0061] In a possible implementation, the image acquisition and image processing module 402 is specifically configured to:

[0062] Acquire the location information of the target object through the deep learning positioning and recognition model;

[0063] Performing a golden section on the first image based on the position information, calculating a golden section screen ratio, and projecting the target object into the golden section screen ratio to obtain movement and zoom parameters of the camera device.

[0064] In one possible implementation, the deep learning positioning and recognition model includes at least one of a hand detection model, a gesture recognition model, a portrait detection model, and an AI composition model.

[0065] In a possible implementation, the camera device further includes a shooting module 406, which is used to:

[0066] The picture of the first image is changed to the golden ratio of the AI ​​composition based on the movement and zoom parameters to obtain the second image.

[0067] In a possible implementation, the camera device further includes an image beautification module 408, which is used to:

[0068] The second image is beautified by an image processing algorithm, and the beautified second image is output.

[0069] It should be noted that each device in the above system may also include other units. The specific implementation of each device and unit can refer to the relevant description in the above method embodiment, which will not be repeated here.

[0070] The embodiment of the present application also provides a computer storage medium, in which instructions are stored, and when the instructions are executed on a computer or a processor, the computer or the processor executes one or more steps of the method described in any of the above embodiments. If the components of the above-mentioned device are implemented in the form of software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer product is stored in a computer-readable storage medium.

[0071] The computer-readable storage medium may be an internal storage unit of the device described in the foregoing embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Further, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0072] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0073] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0074] The steps in the methods of each embodiment of the present application can be adjusted in sequence, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0075] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0076] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0077] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0078] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0079] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0080] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0081] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0082] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for controlling a camera device, characterized in that: The camera device includes an application layer, a UVC protocol layer, a firmware layer and a bottom driver, wherein the camera application layer is equipped with a camera application; the UVC protocol layer is provided with an information access channel to realize the encrypted encoding and decoding signal transmission between the firmware layer and the application layer through the information access channel; the firmware layer is provided with a deep learning positioning and recognition model, and the method includes: Acquire a trigger instruction for the camera application, and in response to the trigger instruction, encrypt and encode the camera application through the firmware layer; Decoding the camera application in the application layer, obtaining a first image containing a target object when the camera application is decoded, and invoking the deep learning positioning and recognition model in the firmware layer to process the first image to obtain movement and zoom parameters of the camera device; The bottom layer driver adjusts the camera device based on the movement and zoom parameters of the camera device so that the camera device captures a second image containing the target object.

2. The method according to claim 1, characterized in that The calling of the deep learning positioning and recognition model in the firmware layer to process the first image to obtain movement and zoom parameters of the camera device includes: Acquire the location information of the target object through the deep learning positioning and recognition model; Performing a golden section on the first image based on the position information, calculating a golden section screen ratio, and projecting the target object into the golden section screen ratio to obtain movement and zoom parameters of the camera device.

3. The method according to claim 1 or 2, characterized in that The deep learning positioning and recognition model includes at least one of a hand detection model, a gesture recognition model, a portrait detection model and an AI composition model.

4. The method according to claim 1, characterized in that The camera device captures a second image containing the target object, including: The camera device changes the picture of the first image to the golden ratio of the AI ​​composition based on the movement and zoom parameters to obtain the second image.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: The second image is beautified by an image processing algorithm, and the beautified second image is output.

6. A camera device, characterized in that: The camera device includes an application layer, a UVC protocol layer, a firmware layer and a bottom driver, wherein the camera application layer is equipped with a camera application; the UVC protocol layer is provided with an information access channel to realize the encrypted encoding and decoding signal transmission between the firmware layer and the application layer through the information access channel; the firmware layer is provided with a deep learning positioning and recognition model, and the device includes: An instruction acquisition module, used for acquiring a trigger instruction for the camera application, and in response to the trigger instruction, encrypting and encoding the camera application through the firmware layer; An image acquisition and image processing module, which decodes the camera application in the application layer, acquires a first image containing a target object when the camera application is decoded, and calls the deep learning positioning and recognition model in the firmware layer to process the first image to obtain movement and zoom parameters of the camera device; The adjustment module is used to drive the bottom layer to adjust the camera device based on the movement and zoom parameters of the camera device so that the camera device can capture a second image containing the target object.

7. The imaging device according to claim 6, wherein: The image acquisition and image processing module is specifically used for: Acquire the location information of the target object through the deep learning positioning and recognition model; Performing a golden section on the first image based on the position information, calculating a golden section screen ratio, and projecting the target object into the golden section screen ratio to obtain movement and zoom parameters of the camera device.

8. The imaging device according to claim 6 or 7, wherein: The deep learning positioning and recognition model includes at least one of a hand detection model, a gesture recognition model, a portrait detection model and an AI composition model.

9. The imaging device according to claim 6, wherein: The camera device also includes a shooting module, which is used to: The picture of the first image is changed to the golden ratio of the AI ​​composition based on the movement and zoom parameters to obtain the second image.

10. The camera device according to any one of claims 6 to 9, characterized in that: The camera device further includes an image beautification module, which is used for: The second image is beautified by an image processing algorithm, and the beautified second image is output.

Images