Method, device and storage medium for sharing image acquisition device

Abstract

The embodiments of the present disclosure provide a method, device and storage medium for sharing an image acquisition device. The method includes: when a first service occupies an image acquisition device and a second service requests to use the image acquisition device, the first service establishes a connection with the second service; the second service requests to obtain a video stream acquired by the first service through the image acquisition device; the first service sends compressed video stream data to the second service; after receiving the video stream data, the second service decompresses the image in the video stream, puts the decompressed bitmap into a pre-created frame buffer, and renders the next picture to be displayed in advance through the frame buffer; when the second service receives a shooting command sent by the user, the second service sends a command to the first service to stop obtaining the video stream of the first service, and saves and displays the last received image. The implementation method of the present disclosure can be used to realize the sharing of an image acquisition device by multiple services.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of Internet of Things, and in particular to a method, device and storage medium for sharing an image acquisition device. Background Art

[0002] With the rapid development of IoT technology, more and more services and technologies are being applied to real life. Accurate recommendations, collecting user information, and establishing user portraits have become mainstream methods. In particular, the use of cameras can obtain user facial information, analyze user interests through age, gender and other information, and recommend user products. However, due to the limitations of terminal cameras, only one service can occupy the camera at the same time. Summary of the invention

[0003] The embodiments of the present disclosure provide a method, a device, and a storage medium for sharing an image acquisition device, which can realize sharing of the image acquisition device by multiple services.

[0004] In one aspect, an embodiment of the present disclosure provides a method for sharing an image acquisition device, comprising:

[0005] When the first service occupies the image acquisition device and the second service requests to use the image acquisition device, the first service establishes a connection with the second service;

[0006] The second service requests to obtain the video stream acquired by the first service through the image acquisition device;

[0007] The first service sends compressed video stream data to the second service;

[0008] After receiving the video stream data, the second service decompresses the image in the video stream, puts the decompressed bitmap into a pre-created frame buffer, and renders the next image to be displayed in advance through the frame buffer;

[0009] When the second service receives the shooting command sent by the user, the second service sends a command to the first service to stop acquiring the first service video stream, and saves and displays the last received image.

[0010] On the other hand, an embodiment of the present disclosure further provides a display device, including a processor and a memory storing a computer program that can be run on the processor, wherein the processor implements the steps of the aforementioned method of sharing an image acquisition device when executing the program.

[0011] On the other hand, an embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program executable on a processor, wherein the computer program is used to implement the above-mentioned method of sharing an image acquisition device when executed by the processor.

[0012] The method of the embodiment of the present disclosure can be used to achieve the sharing of an image acquisition device by multiple services, and because of caching and rendering display at the same time, it is possible to avoid video freeze and slow rendering problems when playing large images into video streams continuously and quickly.

[0013] Of course, it is not necessary to achieve all the advantages described above at the same time for any product or method of the present disclosure. Other features and advantages of the present disclosure will be described in the subsequent description embodiments, and partly become apparent from the description embodiments, or be understood by practicing the present disclosure. The purposes and other advantages of the embodiments of the present disclosure can be achieved and obtained by the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to provide a further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure. The shapes and sizes of the components in the accompanying drawings do not reflect the actual proportions and are only intended to illustrate the contents of the present disclosure.

[0015] Figure 1 A flow chart of a method for sharing an image acquisition device according to an embodiment of the present disclosure;

[0016] Figure 2 A schematic diagram of a method for sharing an image acquisition device for an application example of the present disclosure;

[0017] Figure 3 A schematic diagram of a frame buffer according to an embodiment of the present disclosure;

[0018] Figure 4 A diagram of the process of capturing a video stream for an example application of the present disclosure;

[0019] Figure 5 Code diagram for starting service A in the background;

[0020] Figure 6 This is a diagram of the task manager after starting service A;

[0021] Figure 7 Schematic diagram of the photo taking function for service B;

[0022] Figure 8 It is a schematic diagram of the structure of a display device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0023] The present disclosure describes multiple embodiments, but the description is exemplary rather than restrictive, and it is obvious to those skilled in the art that there may be more embodiments and implementations within the scope of the embodiments described in the present disclosure. Although many possible feature combinations are shown in the drawings and discussed in the embodiments, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with any other feature or element in any other embodiment, or may replace any other feature or element in any other embodiment.

[0024] The present disclosure includes and contemplates combinations of features and elements known to those of ordinary skill in the art. The embodiments, features, and elements disclosed in the present disclosure may also be combined with any conventional features or elements to form a unique invention scheme defined by the claims. Any features or elements of any embodiment may also be combined with features or elements from other invention schemes to form another unique invention scheme defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in the present disclosure may be implemented individually or in any appropriate combination. Therefore, except for the limitations made according to the attached claims and their equivalents, the embodiments are not subject to other limitations. In addition, various modifications and changes may be made within the scope of protection of the attached claims.

[0025] In addition, when describing representative embodiments, the specification may have presented the method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of the steps described herein, the method or process should not be limited to the steps in the specific order described. As will be appreciated by those of ordinary skill in the art, other orders of steps are also possible. Therefore, the specific order of the steps set forth in the specification should not be interpreted as a limitation to the claims. In addition, the claims for the method and / or process should not be limited to the steps performed in the order written, and those skilled in the art can easily understand that these orders can be changed and still remain within the spirit and scope of the disclosed embodiments.

[0026] Unless otherwise defined, the technical terms or scientific terms used in this disclosure are the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. In this disclosure, "plurality" can mean two or more numbers. "Include" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0027] In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of some known functions and known components. The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures can refer to the general design.

[0028] For large-screen terminals, the camera service is generally unique. So when the camera is occupied by a service, other services cannot open the camera. But sometimes other services do need to use the camera function. For example, the smart screen at a bank branch will start a series of services in the background after the smart screen is turned on, such as face recognition service, voice recognition service or recommendation algorithm service, etc. These background services are collectively referred to as A services. The App in the smart screen is called B service. Assume that the scenario is that the A service occupies the camera after it is started, and the background has been using the camera to collect the user's facial information. At this time, the user opens the B service. There is a function in the B service that allows the user to take a photo for record. Because the A service occupies the camera, the B service here cannot open the camera and cannot complete the photo taking. It is similar to the process of using WeChat video, where you cannot take photos with the camera.

[0029] To this end, the present disclosure provides a method for sharing an image acquisition device, such as Figure 1 As shown, the following steps are included:

[0030] Step 10, when the first service occupies the image acquisition device and the second service requests to use the image acquisition device, the first service establishes a connection with the second service;

[0031] Optionally, the first service and the second service establishing a connection include: the first service and the second service establishing a long connection via Websocket. Websocket is a protocol for full-duplex communication over a single TCP connection. Websocket stipulates a communication specification, and through a handshake mechanism, a TCP-like connection can be established between the client and the server, thereby facilitating communication between the client and the server.

[0032] The establishment of a long connection facilitates the transmission of video streams between the first service and the second service.

[0033] Step 20, the second service requests to obtain the video stream collected by the first service through the image acquisition device;

[0034] Optionally, the first service may pre-set related commands, and the second service sends the commands pre-defined by the first service to the first service. For example, one or more of the following commands: a command to obtain a video stream, a command to stop obtaining a video stream, and a command to capture an image. The first service and the second service interact using pre-defined commands.

[0035] Step 30, the first service sends the compressed video stream data to the second service;

[0036] Step 40, after receiving the video stream data, the second service decompresses the image in the video stream, puts the decompressed bitmap into a pre-created frame buffer, and uses the frame buffer to render the next image to be displayed in advance;

[0037] Optionally, the decompression of images in the video stream can be performed by the controller (CPU). The CPU performs decompression and decodes the compressed images into a bitmap format, thereby sharing the work of the GPU, improving the processing speed, and avoiding video freezes.

[0038] Optionally, the second service may open a sub-thread, and the CPU may decode the compressed image into a bitmap through the sub-thread to avoid occupying the main thread and causing a freeze in the user interface (UI).

[0039] Optionally, the frame buffer is used to cache a first picture and a second picture. When the GPU renders and displays the first picture, the second picture is cached in the frame buffer. When the first picture is rendered and displayed, the second picture is updated to the first picture. When the GPU renders and displays the second picture, the third picture is cached in the frame buffer. The third picture is updated to the second picture, and so on. The pointer of the GPU processing image always points to the first picture. The pointer of the GPU processing image always points to the first picture.

[0040] The GPU rendering and displaying the first picture includes: the GPU using a system graphics drawing tool to draw the first picture on the screen.

[0041] Step 50: When the second service receives the shooting command sent by the user, the second service sends a command to the first service to stop acquiring the video stream of the first service, and saves and displays the last received image.

[0042] Through the method of this embodiment, multiple services can share an image acquisition device (such as a camera). The above only uses the example of the second service sharing the video stream of the first service. In other embodiments, other services (APPs) can also establish a connection with the first service to obtain the video stream captured by the first service through the image acquisition device.

[0043] In other embodiments, through the above method, the second service can save the acquired video stream to achieve acquisition of the video stream.

[0044] The method of the disclosed embodiment can be used to achieve that when multiple services share the image acquisition device, and because of the caching and rendering display, the problem of video freeze and slow rendering can be avoided when large pictures are continuously and quickly played into the video stream. When the image acquisition device is occupied by the first service, the second service can capture the video stream captured by the image acquisition device through the implementation method, and render the video stream smoothly and clearly to the screen.

[0045] The following is a detailed description of the above method through application examples. Figure 2 As shown, including:

[0046] Step 1: Service A opens WebSocket and establishes a long link;

[0047] Establishing a long connection through the Websocket protocol facilitates communication between service A and service B.

[0048] In this example, service A opens a socket long connection as the server, sets a preset dedicated IP and port number to wait for other clients to connect. A long connection means that multiple data packets can be sent continuously on a connection. Using a long connection facilitates the transmission of image data packets.

[0049] In addition, the command protocol is predefined, including the following commands: start capturing camera video stream command (Command: 1), stop capturing camera video stream command (Command: 2), and take a photo command (Command: 3). This is just an example, and in other embodiments, more commands can be defined according to business needs.

[0050] Step 2, service B connects to the pre-defined IP and port of service A, establishes a persistent connection, and accepts the heartbeat packet command (Command: 0 - HertPackage (heartbeat packet) is used to maintain the persistent connection);

[0051] Step 3: After establishing a long connection, service B sends a capture start command Command1, and service A continues to send video stream data through the channel established with service B.

[0052] The so-called video is actually a continuous collection of pictures. When 24 pictures are played continuously in 1 second, which is what we usually call 24 frames, the human eye perceives it as a video. The higher the frame rate, the smoother the video. By continuously sending pictures, the video stream received by service A is sent to service B.

[0053] This example is described by taking the B service sending the capture start command Command1 as an example. In other embodiments, the B service may send other command requests according to business requirements.

[0054] Step 4, service B receives the video stream data, and can first decompress the image, and then put the decompressed image into a pre-created frame buffer, where the image put into the frame buffer is a decompressed Bitmap;

[0055] The frame buffer is a storage area for key frames of the video. In this frame buffer, the B service can use the frame buffer to render the next picture to be displayed in advance.

[0056] The frame buffer can store at least two key frame images F1 and F2, and control the pointer of the key frame to point to the frame to be displayed. For example, the player's resource pointer (i.e., the pointer for GPU processing images) always points to the F1 frame. When the F1 frame is played, the frame buffer decodes and draws the F2 frame image in advance (decoding refers to the algorithm for decompressing the image into a bitmap). After the F1 frame is played, the F2 frame content is transferred to the F1 frame position, and the player continues to play F1. At the same time, the F2 frame position continues to buffer the next frame image as the new F2. Figure 3 As shown in the figure, since it takes time to render an image, you can set the frame buffer to render it in advance to prevent lag during display.

[0057] The process of rendering a picture to the screen includes: GPU decompresses PNG or JPG (two compression formats of pictures, PNG is a lossless compression format, JPG is a lossy compression format) pictures into Bitmap bitmaps, and then renders the Bitmap bitmaps to the screen. In theory, after the B service gets the PNG or JPG picture, it can directly use the Image component of the system to render it to the screen for display. However, in the actual practice process, there are problems such as video freezes and pictures are left blank. The main cause of these problems is the loading and replacement process of the Image component controlled by 30 frames / s, and each time the picture is replaced, it must go through the processing of the GPU (graphics processing unit). For this reason, in the present embodiment, the CPU completes the work of decompressing the PNG or JPG picture into the Bitmap bitmap, and the bitmap that the GPU can directly render is transmitted to the GPU, which shares part of the work of the GPU. The GPU is only responsible for rendering the bitmap to the screen, that is, responsible for converting the binary image information into screen pixels (pixel information). This can save the processing time of the GPU and improve the processing performance of the GPU.

[0058] For example, after receiving the video stream, service B can open a child thread, and the CPU can decode the compressed image (PNG / JPG) into a Bitmap bitmap and then put it into the frame buffer. The GPU processes the Bitmap in the frame buffer and can use the underlying Canvas canvas (the system's underlying drawing and rendering tool, i.e. the aforementioned system graphics drawing tool) to draw it to the screen, thereby displaying the image. By using a child thread to process the image decoding, the CPU and GPU can avoid the time spent processing the image occupying the UI (user interface) layer of the main thread (the main task of the program, including executing user interactions and UI updates), and can avoid problems such as video freezes and blank images due to untimely processing.

[0059] Step 5, service B receives the user's photo-taking instruction and sends a command (Command: 2) to service A to stop capturing the camera video stream data. At this time, the last received image is saved and displayed.

[0060] If you need to take photos later, restart the long link and repeat the above steps.

[0061] In the whole process, pre-decoding and pre-caching of the frame buffer greatly reduce the time it takes to render the image to the screen. The actual effect is very obvious, and the video can be played smoothly and photos can be taken.

[0062] The whole process of capturing video stream is as follows Figure 4 As shown, it includes establishing a long link between the two ends (server and client), and the client sends data instructions to control the server to send pictures and take pictures. After receiving the picture data, the client opens a subthread to pre-decode it first, and then pushes it into the frame buffer for rendering. The frame buffer renders the next frame of the picture in advance so that when the GPU processing image pointer (or display pointer) points to the picture, it can be quickly presented to the terminal.

[0063] Taking the smart screen of a financial outlet as an example, after the smart screen terminal is turned on, the background automatically turns on the artificial intelligence screen (AIScreen service, A service). Figure 5 , Figure 6 It can be seen that the AIScreen service integrates voice recognition service, voice wake-up service, sound collection service, camera service, face fusion service, and session management service. The A service can monitor the user's voice and face information in real time. The camera service of the A service has occupied the camera. Figure 7 This is the Time Gallery function in the Smart Screen application (Service B). This function requires taking pictures and processing pictures through the camera. However, since Service A occupies the camera, Service B cannot open the camera. Therefore, Service B needs to share the camera with Service A.

[0064] Depend on Figure 7It can be seen that the video stream of the camera can be obtained smoothly in service B. Here, the method of the above embodiment is applied. By establishing a WebSocket session with service A, service A is notified through the Command command to send the real-time picture data (30 frames / s) of the camera to service B. Service B achieves the effect of smoothly displaying the real-time video of the camera through a series of optimization algorithm processes of the above method.

[0065] In an exemplary embodiment, the present disclosure also provides a display device, which may include a processor and a memory, wherein the memory stores a computer program that can be run on the processor, wherein the processor implements the method of a multi-service shared image acquisition device in any of the above embodiments of the present disclosure when executing the computer program.

[0066] In an exemplary embodiment, Figure 8 FIG. 1 is a schematic diagram of the structure of a display device in an embodiment of the present disclosure. Figure 8 As shown, the device 60 includes: at least one processor 601; and at least one memory 602 and a bus 603 connected to the processor 601; wherein the processor 601 and the memory 602 communicate with each other through the bus 603; the processor 601 is used to call the program instructions in the memory 602 to execute the steps of the smart branch application management method in any of the above embodiments.

[0067] The processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a transistor logic device, etc., and the present disclosure does not limit this.

[0068] The memory may include a read-only memory (ROM) and a random access memory (RAM), and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0069] In addition to the data bus, the bus may also include a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 8 In the text file, various buses are marked as buses.

[0070] In the implementation process, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the processor or the instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0071] In an exemplary embodiment, the present disclosure also provides a non-volatile computer-readable storage medium on which is stored a computer program executable on a processor, and the computer program, when executed by the processor, implements the steps of the aforementioned method for sharing an image acquisition device.

[0072] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the functional modules / units mentioned in the above description are not divided equally; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0073] Although the embodiments disclosed in the present disclosure are as above, the contents described are only embodiments adopted to facilitate understanding of the present disclosure and are not intended to limit the present disclosure. Any technician in the field to which the present disclosure belongs can make any modifications and changes in the form and details of implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be subject to the scope defined in the attached claims.

Claims

1. A method for sharing an image acquisition device, characterized in that: include: When the first service occupies the image acquisition device and the second service requests to use the image acquisition device, the first service establishes a connection with the second service; The second service requests to obtain the video stream acquired by the first service through the image acquisition device; The first service sends compressed video stream data to the second service; After receiving the video stream data, the second service decompresses the image in the video stream, puts the decompressed bitmap into a pre-created frame buffer, and renders the next image to be displayed in advance through the frame buffer; When the second service receives a shooting command sent by the user, the second service sends a command to the first service to stop acquiring the first service video stream, and saves and displays the last received image.

2. The method according to claim 1, characterized in that The first service and the second service establish a connection including: the first service and the second service establish a long connection via Websocket.

3. The method according to claim 1, characterized in that The method also includes: the first service predefines commands, and the commands include one or more of the following commands: a command to obtain a video stream, a command to stop obtaining a video stream, and a command to capture an image; the first service interacts with the second service using the predefined commands.

4. The method according to claim 1, characterized in that: After receiving the video stream data, the second service decompresses the image in the video stream, including: the CPU decoding the compressed image in the video stream into a bitmap format.

5. The method according to claim 4, characterized in that The CPU decodes the compressed image in the video stream into a bitmap format, including: The second service opens a sub-thread, and the CPU decodes the compressed image in the video stream into a bitmap format through the sub-thread.

6. The method according to claim 1, characterized in that The frame buffer is used to cache the first image and the second image. When the GPU renders and displays the first image, the second image is cached in the frame buffer. When the first image is rendered and displayed, the second image is updated to the first image. When the GPU renders and displays the second image, the third image is cached in the frame buffer. The third image is updated to the second image, and so on. The pointer of the GPU processing the image always points to the first image.

7. The method according to claim 1, characterized in that The first service is the artificial intelligence screen service in the smart screen of financial outlets.

8. A display device comprising a processor and a memory storing a computer program executable on the processor, wherein: When the processor executes the program, the steps of the method for sharing an image acquisition device as claimed in any one of claims 1 to 7 are implemented.

9. The device according to claim 8, characterized in that The device is a smart screen for financial outlets.

10. A computer-readable storage medium, characterized in that: A computer program that can be run on a processor is stored thereon, and when the computer program is executed by the processor, the steps of the method for sharing an image acquisition device as claimed in any one of claims 1 to 7 are implemented.

Images