Image pickup data acquisition method and apparatus, augmented reality device, and storage medium

Abstract

The application discloses a camera data acquisition method and device, an augmented reality device and a storage medium, and relates to the technical field of augmented reality, and is applied to the augmented reality device, and comprises the following steps: receiving a camera access request; if the target camera is not in use when the camera access request is received, running a system camera support component, acquiring an image stream photographed by the target camera through the system camera support component, and caching the image stream to a preset address, and acquiring camera data of the target camera from the preset address; if the target camera is in use when the camera access request is received, creating a virtual camera support component, copying the preset address to the virtual camera support component as a first access interface, running the virtual camera support component, and acquiring the camera data of the target camera through the first access interface. The application realizes shared use of the camera of the augmented reality device.

Description

Technical Field

[0001] This application relates to the field of augmented reality technology, and in particular to a method, apparatus, augmented reality device, and storage medium for acquiring camera data. Background Technology

[0002] AR (Augmented Reality) devices can overlay real-world environments and virtual videos onto the same screen or space in real time for the wearer to view. Common AR devices include AR glasses or AR headsets.

[0003] However, current AR device camera systems only support one process using the same camera. That is, only one process is allowed to obtain the video data captured by the camera. For example, if an AR device uses a certain camera for gesture control, it cannot use the same camera for scanning QR codes, object tracking, etc. Other processes will be disconnected from the previous process through certain strategies to ensure that the same camera is used by only one process. Summary of the Invention

[0004] The main purpose of this application is to provide a method, apparatus, augmented reality device and storage medium for acquiring camera data, in order to solve the technical problem that the same camera in current AR devices can only be used by one process.

[0005] To achieve the above objectives, this application provides a method for acquiring camera data, applied to an augmented reality device, the method comprising the following steps:

[0006] Receive camera access requests and designate the requested camera as the target camera;

[0007] If the target camera is not in use when the camera access request is received, the system camera support component is run, the image stream captured by the target camera is obtained through the system camera support component, the image stream is cached to a preset address, and the camera data of the target camera is obtained from the preset address;

[0008] If the target camera is already in use when the camera access request is received, a virtual camera support component is created, the preset address is copied into the virtual camera support component as the first access interface, the virtual camera support component is run, and the camera data of the target camera is obtained through the first access interface.

[0009] In one embodiment, prior to the step of running the virtual camera support component, the method further includes:

[0010] Obtain the pre-configured streaming access interfaces of all camera configuration streams, use the streaming access interfaces of all camera configuration streams as the second access interface, and copy the second access interface into the virtual camera support component;

[0011] The step of obtaining the camera data of the target camera through the first access interface includes:

[0012] All camera configuration streams are obtained through the second access interface, and a first camera configuration stream is selected from all the camera configuration streams;

[0013] The cached image stream is accessed through the first access interface, and the camera data of the target camera is obtained from the cached image stream based on the first camera configuration stream.

[0014] In one embodiment, the step of selecting a first camera configuration stream from all the camera configuration streams includes:

[0015] Obtain the first configuration parameters encapsulated in the camera access request;

[0016] The camera configuration stream with the highest matching degree with the first configuration parameter among all the camera configuration streams is taken as the first camera configuration stream. The more configuration parameters in the camera configuration stream that are consistent with the first configuration parameter, the higher the matching degree between the camera configuration stream and the first configuration parameter.

[0017] In one embodiment, the step of obtaining the camera data of the target camera from the cached image stream based on the first camera configuration stream includes:

[0018] If the configuration parameters contained in the first camera configuration stream are all consistent with the first configuration parameters, then at least one frame of image is obtained from the cached image stream based on the first camera configuration stream as the camera data obtained from the target camera;

[0019] If the configuration parameters included in the first camera configuration stream are inconsistent with the first configuration parameters, then at least one frame of image is obtained from the cached image stream based on the first camera configuration stream, image conversion is performed on all obtained images, and the converted image is used as the camera data obtained from the target camera, wherein the image parameters of the converted image are the first configuration parameters.

[0020] In one embodiment, the step of acquiring the image stream captured by the target camera includes:

[0021] All configuration parameter groups are obtained based on the camera access request, wherein each configuration parameter group includes one or more configuration parameters;

[0022] Configure a camera configuration stream according to each of the configuration parameter groups, wherein each of the configuration parameter groups corresponds to one camera configuration stream;

[0023] Select one camera configuration stream from all the camera configuration streams, and use the selected camera configuration stream as the target camera configuration stream;

[0024] The image stream captured by the target camera is obtained based on the target camera configuration stream.

[0025] In one embodiment, the step of selecting a camera configuration stream from all the camera configuration streams and using the selected camera configuration stream as the target camera configuration stream includes:

[0026] The camera configuration stream obtained from all the camera configuration streams based on the configuration parameters encapsulated in the camera access request is taken as the target camera configuration stream.

[0027] In one embodiment, after the step of caching the image stream to a preset address, the method further includes:

[0028] The image stream is uploaded to a preset server, and the process returns to execute the step of receiving the camera access request.

[0029] After the step of designating the requested camera as the target camera, the method further includes:

[0030] The application requesting access to the target camera is determined based on the camera access request;

[0031] If the application requesting access to the target camera is a non-native application, then the camera data of the target camera is obtained from the image stream uploaded to the preset server.

[0032] Furthermore, to achieve the above objectives, this application also provides a camera data acquisition device, the camera data acquisition device comprising:

[0033] The receiving module is used to receive camera access requests and designate the requested camera as the target camera.

[0034] The sharing module is used to run the system camera support component if the target camera is not in use when the camera access request is received, obtain the image stream captured by the target camera through the system camera support component, cache the image stream to a preset address, and obtain the camera data of the target camera from the preset address;

[0035] The acquisition module is used to create a virtual camera support component if the target camera is already in use when the camera access request is received, copy the preset address as the first access interface into the virtual camera support component, run the virtual camera support component, and acquire the camera data of the target camera through the first access interface.

[0036] In addition, to achieve the above objectives, this application also provides an augmented reality device, which includes a microphone array and a processor, wherein the microphone array is electrically connected to the processor; the processor is used to execute the steps of the camera data acquisition method described above.

[0037] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium, and stores a program implementing the camera data acquisition method thereon. The program implementing the camera data acquisition method is executed by a processor to implement the steps of the camera data acquisition method as described above.

[0038] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the camera data acquisition method described above.

[0039] In this application, the augmented reality device receives a camera access request and designates the requested camera as the target camera. If the target camera is not in use when the camera access request is received, a system camera support component is run. The system camera support component acquires the image stream captured by the target camera and caches the image stream to a preset address. The camera data of the target camera is then obtained from the preset address. If the target camera is in use when the camera access request is received, a virtual camera support component is created. The preset address is copied into the virtual camera support component as a first access interface. The virtual camera support component is run, and the camera data of the target camera is obtained through the first access interface. Thus, in this embodiment, when the target camera is not in use, the image stream captured by the target camera is cached. Since the camera system of the AR device requires the camera support component to only provide the image stream of the same camera to one process, when subsequent processes access the target camera, this embodiment creates a virtual camera support component. The virtual camera support component obtains the camera data from the cached image stream, instead of using the system camera support component to obtain the image stream of the target camera. Therefore, the process that initially uses the target camera and the process that uses the target camera again each obtain the camera data based on an independent camera support component, realizing that the same camera of the AR device can be shared by multiple processes. For example, when the AR device plays the OST (Original Sound Track) based on the captured current scene or uses VST (Virtual Studio Technology) to edit and process the audio content in the AR experience based on the captured current scene, gesture control can be supported simultaneously. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the first embodiment of the camera data acquisition method of this application;

[0043] Figure 2 This is a simplified flowchart illustrating a camera data acquisition method according to an embodiment of this application;

[0044] Figure 3 This is a flowchart illustrating the fourth embodiment of the camera data acquisition method of this application;

[0045] Figure 4 This is a schematic diagram of the device structure of the camera data acquisition apparatus of this application;

[0046] Figure 5 This is a schematic diagram of the hardware operating environment of the camera data acquisition device in the embodiments of this application.

[0047] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0048] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] In AR devices, when one process is using the camera, other processes cannot access the camera at the same time. This is because the camera is a hardware resource, and the operating system usually does not allow two different processes to directly control the same hardware resource at the same time.

[0050] Based on this, the main solution of this application is as follows: Upon receiving a camera access request, the requested camera is designated as the target camera; if the target camera is not in use when the camera access request is received, a system camera support component is run, the image stream captured by the target camera is obtained through the system camera support component, and the image stream is cached to a preset address, and the camera data of the target camera is obtained from the preset address; if the target camera is in use when the camera access request is received, a virtual camera support component is created, the preset address is copied as a first access interface into the virtual camera support component, the virtual camera support component is run, and the camera data of the target camera is obtained through the first access interface.

[0051] This application caches the image stream captured by the target camera when it is not in use. Since the camera system of an AR device requires that the camera support component only provide the image stream from the same camera to one process at a time, this application creates a virtual camera support component to retrieve the camera data from the cached image stream when subsequent processes access the target camera. This virtual component retrieves the camera data from the cached image stream, instead of using the system camera support component. Thus, the process initially using the target camera and subsequent processes accessing the target camera each obtain their image data based on an independent camera support component, enabling multiple processes to share the same camera on the AR device. For example, when the AR device plays an OST (Original Sound Track) based on the captured current scene or uses VST (Virtual Studio Technology) to edit and process audio content in the AR experience based on the captured current scene, gesture control can be supported simultaneously.

[0052] It should be noted that the execution subject of this embodiment can be an augmented reality device capable of realizing the above functions, such as AR glasses, AR helmets, etc. For example, AR glasses are used as the execution subject to describe the various embodiments of this application.

[0053] Based on this, this application proposes a camera data acquisition method according to the first embodiment, please refer to... Figure 1 The camera data acquisition method includes steps S10 to S30:

[0054] Step S10: Receive a camera access request and designate the requested camera as the target camera.

[0055] It should be noted that the AR glasses can be AR glasses based on the Android camera system, but are not limited to AR glasses based on the Android camera system. For example, they can also be AR glasses based on Apple (iOS). This embodiment does not impose any specific limitations on this.

[0056] Understandably, AR glasses sometimes need to perform corresponding tasks based on the video data captured by the camera. For example, when AR glasses perform gesture control, they need to acquire the video data captured by the camera to recognize the user's gestures and then perform subsequent control based on the recognized gestures. When AR glasses play OSTs, in order to improve the user experience, they need to acquire the video data captured by the camera to recognize the current scene in which the user is located and then play the OST corresponding to that scene.

[0057] When an application in AR glasses needs to obtain camera data from a specific camera, that is, when it needs to use that camera, it can initiate a camera access request for that camera through the process. ， Specifically, the process can encapsulate the camera identifier of the camera in the camera access request to determine the target camera to be accessed in this request based on the camera identifier encapsulated in the camera access request.

[0058] It should be noted that after a process initiates a camera access request to the target camera, if the AR glasses receive the process's access request, the process can obtain the target camera's video data. In other words, the process starts using the target camera. During the use of the target camera, the process will not send any more camera access requests. It only needs to initiate a camera stop access request when the process no longer needs to use the target camera to terminate the process's continued acquisition of the target camera's video data, that is, to terminate the process's use of the target camera.

[0059] Step S20: If the target camera is not in use when the camera access request is received, the system camera support component is run, the image stream captured by the target camera is obtained through the system camera support component, the image stream is cached to a preset address, and the camera data of the target camera is obtained from the preset address;

[0060] For ease of subsequent explanation and clarification, if the target camera is not in use when the camera access request is received, the process that accesses the target camera will be recorded as the process that accesses the target camera for the first time.

[0061] The image stream is specifically composed of image frames captured by the target camera at a preset shooting frequency.

[0062] The camera data of the target camera is obtained from the image stream. Specifically, one frame, multiple frames, or all frames can be obtained from the image stream as the camera data of the target camera. It is easy to understand that different processes have different requirements for the number of image frames they need to acquire. For example, the gesture recognition process may need to acquire all image frames, while the photo-taking process may only need to acquire one image frame. Based on this, the corresponding image frame can be obtained from the image stream as the acquired camera data based on the characteristics of the process itself.

[0063] The preset address is specifically the address of the shared storage area, so that all processes can access this address and obtain the image stream cached at this preset address.

[0064] The camera system architecture consists of, in sequence, the application layer, the framework layer, the hardware abstraction layer (HAL), and the hardware layer. The HAL implements interfaces related to specific hardware, abstracting the implementation details of different camera hardware. This allows the Android system to access camera devices from different manufacturers in a unified way. Specifically, CameraProvider is a key system-level component of the HAL layer, responsible for managing and providing access interfaces to the camera hardware. It provides information and status of the camera devices to the upper-layer CameraManager service and handles operations such as opening, configuring, transmitting data streams, and closing the camera devices. Therefore, in this embodiment, the camera support component is specifically CameraProvider.

[0065] The system camera support component is specifically the original CameraProvider of the AR glasses' camera system.

[0066] Step S30: If the target camera is already in use when the camera access request is received, create a virtual camera support component, copy the preset address as the first access interface into the virtual camera support component, run the virtual camera support component, and obtain the camera data of the target camera through the first access interface.

[0067] For ease of subsequent explanation and clarification, if the target camera is already in use when the camera access request is received, then the process that accesses the target camera will be recorded as the process that accesses the target camera again.

[0068] In this embodiment, the virtual camera support component is specifically a virtual CameraProvider. The virtual CameraProvider indicates that it is not an actual CameraProvider, but rather provides an interface for data access.

[0069] It should be noted that for each process that accesses the target camera before or after its first access, a corresponding virtual camera support component is created for it. However, if a virtual camera support component already exists within the AR glasses for the process that initiated the camera access request, it will not be created again; the existing virtual camera support component will be used directly. For example, suppose process B initiates a camera access request to access the target camera. At this time, process A is already using the target camera, and no virtual camera component has been found for process B. In this case, a corresponding virtual camera component B will be created for process B. If at some point process B initiates a camera stop access request to end its use of the target camera, and at some point process B initiates a camera access request to access the target camera again, and it is found that a corresponding virtual camera component B has already been created for process B, then the virtual camera support component will not be created again; the existing virtual camera support component B will be used directly.

[0070] In this embodiment, the augmented reality device receives a camera access request and designates the requested camera as the target camera. If the target camera is not in use when the camera access request is received, the system camera support component is run, and the image stream captured by the target camera is obtained through the system camera support component. The image stream is then cached at a preset address, and the camera data of the target camera is obtained from the preset address. If the target camera is in use when the camera access request is received, a virtual camera support component is created, and the preset address is copied into the virtual camera support component as a first access interface. The virtual camera support component is then run, and the camera data of the target camera is obtained through the first access interface. Thus, in this embodiment, when the target camera is not in use, the image stream captured by the target camera is cached. Since the camera system of the AR device requires the camera support component to only provide the image stream of the same camera to one process, when subsequent processes access the target camera, this embodiment creates a virtual camera support component. The virtual camera support component obtains the camera data from the cached image stream, instead of using the system camera support component to obtain the image stream of the target camera. Therefore, the process that initially uses the target camera and subsequent processes that use the target camera obtain the camera data based on an independent camera support component, realizing that the same camera of the AR device can be shared by multiple processes. For example, when the AR device plays the OST (Original Sound Track) based on the captured current scene or uses VST (Virtual Studio Technology) to edit and process the audio content in the AR experience based on the captured current scene, gesture control can be supported simultaneously.

[0071] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description and will not be repeated hereafter. Furthermore, before the step of running the virtual camera support component, the method further includes:

[0072] Step B10: Obtain the pre-configured stream access interfaces of all camera configuration streams, use all the stream access interfaces of the camera configuration streams as the second access interface, and copy the second access interface into the virtual camera support component.

[0073] Specifically, all the pre-configured camera configuration streams can be the HAL layer pre-configured camera configuration streams (i.e., camera streams) when the target camera is accessed for the first time.

[0074] This second access interface can specifically configure the storage address of the stream for all cameras.

[0075] The step of obtaining the camera data of the target camera through the first access interface includes:

[0076] Step B20: Obtain all camera configuration streams through the second access interface, and select a first camera configuration stream from all the camera configuration streams;

[0077] The first camera configuration stream can be selected randomly from all camera configuration streams or selected based on preset rules from those that are not currently in use. The phrase "camera configuration stream not in use" indicates that the camera configuration stream is not currently being used by the camera system to acquire image streams.

[0078] Step B30: Access the cached image stream through the first access interface, and obtain the camera data of the target camera from the cached image stream based on the first camera configuration stream.

[0079] Understandably, in a camera system, it is necessary to obtain the image stream of the camera based on the camera configuration stream. Based on this, in this embodiment, camera data is obtained from the cached image stream based on the selected first camera configuration stream.

[0080] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. Based on this, the step of selecting a first camera configuration stream from all the camera configuration streams includes:

[0081] Step C10: Obtain the first configuration parameters encapsulated in the camera access request;

[0082] Step C20: Select the camera configuration stream with the highest matching degree with the first configuration parameter from all the camera configuration streams as the first camera configuration stream. The more configuration parameters in the camera configuration stream that are consistent with the first configuration parameter, the higher the matching degree between the camera configuration stream and the first configuration parameter.

[0083] Furthermore, if there are multiple camera configuration streams with the highest matching degree, then any one of the multiple camera configuration streams with the highest matching degree can be selected as the first camera configuration stream.

[0084] It should be noted that for a camera configuration stream, the more configuration parameters in the camera configuration stream that are consistent with the first configuration parameter, the higher the matching degree between the camera configuration stream and the first configuration parameter.

[0085] In one implementation, the total configuration parameters that are consistent with the first configuration parameter among all configuration parameters contained in the camera configuration stream can be used as the matching degree between the camera configuration stream and this first camera configuration parameter. For example, assuming the first configuration parameter is [Format A, Color Space A, Resolution A], and there are camera configuration streams A, B, and C, containing configuration parameters of [Format A, Color Space B, Resolution C], [Format A, Color Space A, Resolution B], and [Format A, Color Space A, Resolution A] respectively, then the matching degree between camera configuration stream A and the first configuration parameter is 1, the matching degree between camera configuration stream B and the first configuration parameter is 2, and the matching degree between camera configuration stream C and the first configuration parameter is 3. In this case, camera configuration stream C has the highest matching degree, and camera configuration stream C is selected as the first camera configuration stream.

[0086] As another implementation, a corresponding weight can be set for each configuration parameter, and a weighted summation is performed based on the weights to calculate the matching degree between the camera configuration stream and this first configuration parameter. For example, assuming the first configuration parameter is [format A, color space A, resolution A], and the weights corresponding to image format, color space, and resolution are 0.1, 0.7, and 0.2 respectively, and there are camera configuration streams A, B, and C, containing configuration parameters [format A, color space B, resolution C], [format A, color space A, resolution B], and [format A, color space A, resolution A] respectively, then the matching degree between camera configuration stream A and the first configuration parameter is 0.1, the matching degree between camera configuration stream B and the first configuration parameter is 0.8, and the matching degree between camera configuration stream C and the first configuration parameter is 1. In this case, camera configuration stream C has the highest matching degree, and camera configuration stream C is selected as the first camera configuration stream.

[0087] It should be noted that the above are only two feasible implementation methods for calculating the matching degree between the camera configuration stream and the first configuration parameter provided in this embodiment. This embodiment does not specifically limit the specific implementation method for calculating the matching degree between the camera configuration stream and the first configuration parameter.

[0088] In this embodiment, the camera configuration stream with the highest matching degree with the first configuration parameter is selected as the first camera configuration stream. It can be understood that the process usually encapsulates the data format it expects to obtain in the form of configuration parameters in the camera access request. In this way, the data format of the image stream obtained based on the first camera configuration stream can be as close as possible to the data format expected by the process accessing the target camera, that is, the matching degree between the obtained data format and the expected data format is maximized, so as to reduce the amount of subsequent data format conversion.

[0089] In one possible implementation, the step of obtaining the camera data of the target camera from the cached image stream based on the first camera configuration stream includes:

[0090] Step D10: If the configuration parameters contained in the first camera configuration stream are consistent with the first configuration parameters, then at least one frame of image is obtained from the cached image stream based on the first camera configuration stream as the camera data obtained from the target camera.

[0091] The configuration parameters in the first camera configuration stream are identical to those in the first configuration parameter; that is, the parameter values ​​are the same. For example, assuming a configuration parameter includes image format, color space, and resolution, then the configuration parameters in the first camera configuration stream are identical to those in the first configuration parameter. Specifically, the image format, color space, and resolution in the first camera configuration stream are identical to those in the first configuration parameter. Since the configuration parameters are completely identical, the image stream obtained from the first camera configuration stream is in the expected data format, and no data format conversion is required.

[0092] Step D20: If there are configuration parameters in the configuration parameters included in the first camera configuration stream that are inconsistent with the first configuration parameters, then at least one frame of image is obtained from the cached image stream based on the first camera configuration stream, image conversion is performed on all obtained images, and the converted image is used as the camera data obtained from the target camera, wherein the image parameters of the converted image are the first configuration parameters.

[0093] Based on the first camera configuration stream, at least one frame of image is obtained from the cached image stream. The specific number and types of frames obtained can be determined based on the process's own needs, and this implementation does not impose any specific restrictions on this.

[0094] All acquired images undergo image conversion. Specifically, conversion can be performed based on inconsistent configuration parameters. For example, suppose the first camera configuration stream contains configuration parameters of [image format A, color space A, resolution B], and the first configuration parameter is [image format A, color space A, resolution A]. Where the image format and color space are consistent, but the resolutions are inconsistent, the acquired images can be scaled to reduce resolution B to resolution A, thereby making the converted data format conform to expectations.

[0095] In this embodiment, when the configuration parameters in the first camera configuration stream are inconsistent with the first configuration parameters, the image is converted to convert the image parameters such as image format, color space, and resolution into the first configuration parameters, so that the data format of the acquired camera data conforms to the process expectation.

[0096] Based on the first, second, and / or third embodiments of this application, in the fourth embodiment of this application, the content that is the same as or similar to the above-described embodiments one, two, and three can be referred to the above description and will not be repeated hereafter. Based on this, refer to... Figure 3 As shown, the step of acquiring the image stream captured by the target camera includes:

[0097] Step E10: Obtain all configuration parameter groups according to the camera access request, wherein each configuration parameter group includes one or more configuration parameters;

[0098] Furthermore, when the configuration parameter group includes multiple configuration parameters, these multiple configuration parameters are specifically multiple configuration parameters of different parameter types. Specifically, the parameter types of the configuration parameters include, but are not limited to, image format, color space, and resolution. This embodiment does not impose specific restrictions on these, and the parameter types required when configuring the camera stream (i.e., camera configuration stream) in the actual camera system shall prevail.

[0099] Furthermore, based on different parameter types, the configuration parameters encapsulated in the camera access request and the configuration parameters supported by the AR glasses are permuted and combined to obtain all configuration parameter groups. For example, assuming each configuration parameter group includes configuration parameters of two types: image format and resolution, and the configuration parameters encapsulated in the camera access request are [Format A, Resolution A], for the image format parameter type, the AR glasses support configuration parameters of format B and format C; for the resolution parameter type, the AR glasses support configuration parameters of resolution A and resolution B. Then, the configuration parameter groups obtained by permuting and combining different parameter types are [Format A, Resolution A], [Format A, Resolution B], [Format B, Resolution A], [Format B, Resolution B], [Format C, Resolution A], and [Format C, Resolution B].

[0100] Step E20: Configure a camera configuration stream according to each of the configuration parameter groups, wherein each of the configuration parameter groups corresponds to a camera configuration stream;

[0101] The specific configuration method for the camera configuration stream in the configuration parameter group can use the existing camera configuration stream configuration method in the Android camera system HAL layer. This embodiment will not detail the camera configuration stream configuration process. It should be noted that after each camera configuration stream is configured, it includes the configuration parameters in the configuration parameter group and a generated pipeline. The pipeline is a channel in the HAL layer used to manage image data streams. It ensures that image data is transmitted efficiently and consistently, while handling possible data format conversions. The pipeline may include multiple processing stages, such as noise reduction, white balance adjustment, color correction, cropping, and scaling. These processing stages help improve image quality and meet the needs of specific applications; they can be understood as a channel for transmitting image streams. The configuration parameters in the configuration parameter group are used to indicate the processing target of the pipeline.

[0102] Step E30: Select a camera configuration stream from all the camera configuration streams and use the selected camera configuration stream as the target camera configuration stream;

[0103] When AR glasses acquire an image stream, they need to acquire it based on a camera configuration stream. Based on this, a target camera configuration stream can be randomly selected from all camera configuration streams or selected based on preset rules in order to successfully acquire the image stream.

[0104] Step E40: Obtain the image stream captured by the target camera based on the target camera configuration stream.

[0105] When acquiring an image stream based on a certain camera configuration stream, it can specifically be done by acquiring the image stream through the pipeline in this camera configuration stream. The image stream output by the pipeline has the data format of the configuration parameters. The actual acquired image stream can be understood as the stream data output after passing through the pipeline.

[0106] It is understandable that the HAL layer of the camera system can only configure the camera configuration stream once for the same camera. Based on this, when this embodiment receives the camera access request from the process accessing the target camera for the first time, it not only configures the camera configuration stream according to the configuration parameters encapsulated in the camera access request, but also configures all camera configuration streams according to the configuration parameters supported by the AR glasses. In this way, the camera configuration stream of subsequent processes accessing the target camera is pre-configured so that subsequent processes accessing the target camera can obtain image streams based on the pre-configured camera configuration streams. This allows the AR glasses to support different processes obtaining image streams based on different camera configuration streams, and the image streams obtained by different camera configuration streams have different formats, enabling the AR glasses to support the acquisition of multi-format image streams and improving the intelligence of camera sharing.

[0107] In one possible implementation, the step of selecting one camera configuration stream from all the camera configuration streams and using the selected camera configuration stream as the target camera configuration stream includes:

[0108] Step F10: Use the camera configuration streams obtained from all the camera configuration streams based on the configuration parameters encapsulated in the camera access request as the target camera configuration stream.

[0109] In this embodiment, the camera configuration stream obtained based on the configuration parameters encapsulated in the camera access request is used as the target camera configuration stream. It can be understood that the process usually encapsulates the data format it expects to obtain in the form of configuration parameters in the camera access request. In this way, the data format of the image stream obtained based on the target camera configuration stream can meet the expectations of the process accessing the target camera for the first time.

[0110] Based on the first, second, third, and / or fourth embodiments of this application, in the fifth embodiment of this application, the content that is the same as or similar to the above-described embodiments one, two, three, and four can be referred to the above description and will not be repeated hereafter. In addition, after the step of caching the image stream to a preset address, the method further includes:

[0111] Step G10: Upload the image stream to a preset server and return to the step of receiving the camera access request;

[0112] Upload the image stream to the preset server; specifically, you can copy the image stream to the preset server.

[0113] After the step of designating the requested camera as the target camera, the method further includes:

[0114] Step G20: Determine the application requesting access to the target camera based on the camera access request;

[0115] Step G30: If the application requesting access to the target camera is a non-native application, then obtain the camera data of the target camera from the image stream uploaded to the preset server.

[0116] Native applications refer to applications that can run directly on the current operating system and are specifically developed for a particular operating system, such as iOS or Android. Native applications are built using specific programming languages ​​for specific device platforms. Non-native applications specifically refer to applications that are distinct from native applications, such as user-defined applications and web applications. In this embodiment, non-native applications may specifically refer to non-Android system native applications.

[0117] In this embodiment, non-native applications obtain camera data from a preset server. This allows non-native applications to share and use the camera, and directly obtaining camera data from the preset server improves the acquisition speed, achieving low-latency acquisition of camera data.

[0118] For example, to help understand the technical concept or principle of camera data acquisition in combination with the above-described embodiments one, two, three, and four, please refer to... Figure 2 , Figure 2 A simplified flowchart of a method for acquiring camera data is provided below:

[0119] 1. Process A and process B access the same camera through the Android camera system's Camera API (Camera Application Programming Interface). Process A and process B are processes of the native Camera application on the Android platform.

[0120] 2. In the Camera server (system camera service) module, if a camera that is already in use is opened, since the camera system can only assign a camera identifier to one process for the same camera, a virtual camera identifier is generated and passed to the HAL layer. This allows the camera system to accept the access request from that process for further processing, instead of kicking out the process that is currently using the camera or directly rejecting its access request. The Camera server module is the native Camera server module of the Android camera system, including native Android camera system modules such as Camera service, CameraDeviceClient, Camera3Device, and CameraProviderManager.

[0121] 3. In the Camera provider module, modify the camera opening logic. When an unused camera is opened, configure all camera streams (including all image formats, color spaces, and resolutions, generating all pipelines) and send them to the Camera driver. Then, send all camera streams to the Camera driver and cache the retrieved image streams. When the same camera is opened a second time, create a virtual CameraProvider (…). Figure 2 The Camera fake Provider shown in the image subsequently achieves data sharing by retrieving camera data from the cached image stream.

[0122] 4. Share the image stream to the server ( Figure 2 The server-side stream shown in the image indicates that when a third-party algorithm in a non-native application needs to use the camera, it obtains camera data from the server, and the image stream is transmitted from the server to the client of the non-native application. Figure 2 (The client stream shown in the image).

[0123] It should be noted that the above specific embodiments are only used to understand this application and do not constitute a limitation on the camera data acquisition process of this application. Any simple modifications based on this technical concept are all within the protection scope of this application.

[0124] Furthermore, embodiments of this application also provide a camera data acquisition device, referring to... Figure 4 The camera data acquisition device includes:

[0125] The receiving module 10 is used to receive camera access requests and to take the requested camera as the target camera.

[0126] The sharing module 20 is used to run the system camera support component if the target camera is not in use when the camera access request is received, to obtain the image stream captured by the target camera through the system camera support component, to cache the image stream to a preset address, and to obtain the camera data of the target camera from the preset address;

[0127] The acquisition module 30 is used to create a virtual camera support component if the target camera is already in use when the camera access request is received, copy the preset address as the first access interface into the virtual camera support component, run the virtual camera support component, and acquire the camera data of the target camera through the first access interface.

[0128] In addition, such as Figure 5 As shown, the augmented reality device may also include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the augmented reality device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the augmented reality device to communicate wirelessly or wiredly with other devices to exchange data. While the figures show augmented reality devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0129] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0130] The augmented reality device provided in this application, employing the camera data acquisition method described in the above embodiments, can solve the technical problem of camera data acquisition. Compared with the prior art, the beneficial effects of the augmented reality device provided in this application are the same as those of the camera data acquisition method provided in the above embodiments, and other technical features of this augmented reality device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0131] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0132] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0133] In addition, to achieve the above objectives, this application also provides a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the camera data acquisition method in the above embodiments.

[0134] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0135] The aforementioned computer-readable storage medium may be included in the augmented reality device; or it may exist independently and not be assembled into the augmented reality device.

[0136] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an augmented reality device, cause the augmented reality device to: receive a camera access request and designate the requested camera as the target camera; if the target camera is not in use when the camera access request is received, run a system camera support component, acquire an image stream captured by the target camera through the system camera support component, cache the image stream to a preset address, and acquire the camera data of the target camera from the preset address; if the target camera is in use when the camera access request is received, create a virtual camera support component, copy the preset address as a first access interface into the virtual camera support component, run the virtual camera support component, and acquire the camera data of the target camera through the first access interface.

[0137] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0139] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0140] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described camera data acquisition method, thereby solving the technical problem of camera data acquisition. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the camera data acquisition method provided in the above embodiments, and will not be repeated here.

[0141] Furthermore, this application also proposes a computer program product, including a camera data acquisition program, which, when executed by a processor, implements the steps of the camera data acquisition method described above.

[0142] The specific implementation of the computer program product in this application is basically the same as the embodiments of the above-described camera data acquisition method, and will not be repeated here.

[0143] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0144] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0146] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for acquiring camera data, characterized in that, The camera data acquisition method, applied to augmented reality devices, includes the following steps: Receive camera access requests and designate the requested camera as the target camera; If the target camera is not in use when the camera access request is received, the system camera support component is run, and all configuration parameter groups are obtained according to the camera access request, wherein each configuration parameter group includes one or more configuration parameters; Configure a camera configuration stream according to each of the configuration parameter groups, wherein each of the configuration parameter groups corresponds to one camera configuration stream; Select one camera configuration stream from all the camera configuration streams, and use the selected camera configuration stream as the target camera configuration stream; Based on the target camera configuration stream, the image stream captured by the target camera is obtained, and the image stream is cached to a preset address. The camera data of the target camera is then obtained from the preset address. If the target camera is already in use when the camera access request is received, create a virtual camera support component and copy the preset address as the first access interface into the virtual camera support component; Obtain the pre-configured streaming access interfaces of all camera configuration streams, use the streaming access interfaces of all camera configuration streams as the second access interface, and copy the second access interface into the virtual camera support component; Run the virtual camera support component, obtain all camera configuration streams through the second access interface, and select a first camera configuration stream from all the camera configuration streams; The cached image stream is accessed through the first access interface, and the camera data of the target camera is obtained from the cached image stream based on the first camera configuration stream.

2. The method as described in claim 1, characterized in that, The step of selecting a first camera configuration stream from all the camera configuration streams includes: Obtain the first configuration parameters encapsulated in the camera access request; The camera configuration stream with the highest matching degree with the first configuration parameter among all the camera configuration streams is taken as the first camera configuration stream. The more configuration parameters in the camera configuration stream that are consistent with the first configuration parameter, the higher the matching degree between the camera configuration stream and the first configuration parameter.

3. The method as described in claim 2, characterized in that, The step of obtaining the target camera's video data from the cached image stream based on the first camera configuration stream includes: If the configuration parameters contained in the first camera configuration stream are all consistent with the first configuration parameters, then at least one frame of image is obtained from the cached image stream based on the first camera configuration stream as the video data obtained from the target camera; If the configuration parameters included in the first camera configuration stream are inconsistent with the first configuration parameters, then at least one frame of image is obtained from the cached image stream based on the first camera configuration stream, image conversion is performed on all obtained images, and the converted image is used as the camera data obtained from the target camera, wherein the image parameters of the converted image are the first configuration parameters.

4. The method as described in claim 1, characterized in that, The step of selecting a camera configuration stream from all the camera configuration streams and using the selected camera configuration stream as the target camera configuration stream includes: The camera configuration stream obtained from all the camera configuration streams based on the configuration parameters encapsulated in the camera access request is taken as the target camera configuration stream.

5. The method according to any one of claims 1 to 4, characterized in that, After the step of caching the image stream to a preset address, the method further includes: The image stream is uploaded to a preset server, and the process returns to execute the step of receiving the camera access request. After the step of designating the requested camera as the target camera, the method further includes: The application requesting access to the target camera is determined based on the camera access request; If the application requesting access to the target camera is a non-native application, then the camera data of the target camera is obtained from the image stream uploaded to the preset server.

6. A camera data acquisition device, characterized in that, The camera data acquisition device includes: The receiving module is used to receive camera access requests and designate the requested camera as the target camera. The sharing module is configured to, if the target camera is not in use when the camera access request is received, run the system camera support component and obtain all configuration parameter groups according to the camera access request, wherein each configuration parameter group includes one or more configuration parameters; configure a camera configuration stream according to each configuration parameter group, wherein each configuration parameter group corresponds to one camera configuration stream; select a camera configuration stream from all the camera configuration streams and use the selected camera configuration stream as the target camera configuration stream; obtain the image stream captured by the target camera based on the target camera configuration stream, cache the image stream to a preset address, and obtain the camera data of the target camera from the preset address; The acquisition module is used to, if the target camera is already in use when the camera access request is received, create a virtual camera support component, copy the preset address as the first access interface into the virtual camera support component; acquire the pre-configured stream access interfaces of all camera configuration streams, and copy the stream access interfaces of all camera configuration streams as the second access interface into the virtual camera support component. The acquisition module is further configured to run the virtual camera support component, acquire all camera configuration streams through the second access interface, select a first camera configuration stream from all the camera configuration streams, access the cached image stream through the first access interface, and acquire the camera data of the target camera from the cached image stream based on the first camera configuration stream.

7. An augmented reality device, characterized in that, The augmented reality device includes: a memory, a processor, and a camera data acquisition program stored in the memory and executable on the processor, wherein the camera data acquisition program, when executed by the processor, implements the steps of the camera data acquisition method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, and the computer-readable storage medium stores a program that implements the camera data acquisition method. The program that implements the camera data acquisition method is executed by a processor to implement the steps of the camera data acquisition method as described in any one of claims 1 to 5.

Images