Imaging data acquisition method and apparatus, and augmented reality device and storage medium

Abstract

The present application relates to the technical field of augmented reality. Disclosed are an imaging data acquisition method and apparatus, and an augmented reality device and a storage medium. The imaging data acquisition method is applied to an augmented reality device, and comprises: receiving a camera access request; if a target camera is not in use when the camera access request is received, running a system camera support component, acquiring, by means of the system camera support component, an image stream captured by the target camera, and caching the image stream to a preset address, and acquiring from the preset address imaging data of the target camera; and if the target camera is in use when the camera access request is received, creating a virtual camera support component, copying the preset address as a first access interface into the virtual camera support component, running the virtual camera support component, and acquiring the imaging data of the target camera via the first access interface. The present application enables shared camera use in augmented reality devices.

Description

Camera data acquisition method and device, augmented reality device, and storage medium

[0001] The present application claims priority to the Chinese patent application No. 202410840973.X, filed on June 26, 2024, and entitled "Camera data acquisition method and device, augmented reality device, and storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of augmented reality technology, and in particular to a camera data acquisition method and device, an augmented reality device, and a storage medium. BACKGROUND

[0003] An AR (Augmented Reality) device can superimpose a real environment and a virtual video in real time into the same picture or space for a wearer to view. Common AR devices include AR glasses or AR headsets, etc.

[0004] However, the camera system of the current AR device only supports the use of the same camera by one process, i.e., only one process is allowed to acquire camera data captured by the camera. For example, if a camera in an AR device is used for gesture control, the camera cannot be used for code scanning recognition, object tracking, etc. Other processes will disconnect the use of the previous process through certain strategies to ensure that the same camera is only used by one process. SUMMARY

[0005] The main purpose of the present application is to provide a camera data acquisition method and device, an augmented reality device, and a storage medium, which aims to solve the technical problem that the same camera of the current AR device can only be used by one process.

[0006] To achieve the above purpose, the present application provides a camera data acquisition method applied to an augmented reality device, which comprises the following steps:

[0007] Receiving a camera access request, and taking the requested camera as a target camera;

[0008] If the target camera is not in use when the camera access request is received, running a system camera support component, acquiring an image stream captured by the target camera through the system camera support component, and caching the image stream to a preset address, and acquiring camera data of the target camera from the preset address;

[0009] If the target camera is in use when the camera access request is received, a virtual camera support component is created, the preset address is copied into the virtual camera support component as a first access interface, the virtual camera support component is run, and camera data of the target camera is acquired through the first access interface.

[0010] In an embodiment, before the step of running the virtual camera support component, the method further includes:

[0011] A stream access interface of all preconfigured camera configuration streams is acquired, stream access interfaces of all the camera configuration streams are copied into the virtual camera support component as second access interfaces, and the second access interfaces are copied into the virtual camera support component;

[0012] The step of acquiring camera data of the target camera through the first access interface includes:

[0013] All camera configuration streams are acquired through the second access interfaces, and a first camera configuration stream is selected from all the camera configuration streams;

[0014] The cached image stream is accessed through the first access interface, and camera data of the target camera is acquired from the cached image stream based on the first camera configuration stream.

[0015] In an embodiment, the step of selecting a first camera configuration stream from all the camera configuration streams includes:

[0016] A first configuration parameter encapsulated in the camera access request is acquired;

[0017] A camera configuration stream with the highest matching degree to the first configuration parameter from all the camera configuration streams is selected as the first camera configuration stream, wherein the more configuration parameters consistent with the first configuration parameter in the camera configuration stream, the higher the matching degree of the camera configuration stream to the first configuration parameter.

[0018] In an embodiment, the step of acquiring camera data of the target camera from the cached image stream based on the first camera configuration stream includes:

[0019] If the configuration parameters contained in the first camera configuration stream are all consistent with the first configuration parameter, at least one frame of image is acquired from the cached image stream based on the first camera configuration stream as camera data acquired from the target camera;

[0020] If there is a configuration parameter inconsistent with the first configuration parameter in the configuration parameters included in the first camera configuration stream, at least one frame of image is acquired from the buffered image stream based on the first camera configuration stream, image conversion is performed on all acquired images, and the converted images are taken as camera data acquired from the target camera, wherein the image parameters of the converted images are the first configuration parameters.

[0021] In an embodiment, the step of acquiring the image stream captured by the target camera comprises:

[0022] According to the camera access request, all configuration parameter groups are acquired, wherein each configuration parameter group includes one or more configuration parameters;

[0023] According to each configuration parameter group, a camera configuration stream is configured, wherein each configuration parameter group corresponds to a camera configuration stream;

[0024] From all the camera configuration streams, a camera configuration stream is selected, and the selected camera configuration stream is taken as a target camera configuration stream;

[0025] Based on the target camera configuration stream, the image stream captured by the target camera is acquired.

[0026] In an embodiment, the step of selecting a camera configuration stream from all the camera configuration streams and taking the selected camera configuration stream as a target camera configuration stream comprises:

[0027] From all the camera configuration streams, a camera configuration stream configured based on the configuration parameters encapsulated in the camera access request is taken as a target camera configuration stream.

[0028] In an embodiment, after the step of buffering the image stream to a preset address, the method further comprises:

[0029] The image stream is uploaded to a preset server, and the step of receiving a camera access request is returned to be executed;

[0030] After the step of taking the camera requested to be accessed as a target camera, the method further comprises:

[0031] According to the camera access request, an application requesting to access the target camera is determined;

[0032] If the application requesting to access the target camera is a non-native application, camera data of the target camera is acquired from the image stream uploaded to the preset server.

[0033] In addition, to achieve the above-mentioned purposes, the present application also provides a camera data acquisition device, the camera data acquisition device comprises:

[0034] receive a camera access request, and take a camera requested in the camera access request as a target camera;

[0035] share the image stream to a preset address, and acquire camera data of the target camera from the preset address;

[0036] If the target camera is in use when the camera access request is received, create a virtual camera support component, copy the preset address as a first access interface into the virtual camera support component, run the virtual camera support component, and acquire the camera data of the target camera through the first access interface.

[0037] In addition, to achieve the above object, the present application also provides an augmented reality device, which comprises a microphone array and a processor, wherein the microphone array is electrically connected with the processor; and the processor is used to execute the steps of the camera data acquisition method.

[0038] In addition, to achieve the above object, the present application also provides a readable storage medium, which is a computer readable storage medium, and a program for implementing the camera data acquisition method is stored on the computer readable storage medium; the program for implementing the camera data acquisition method is executed by a processor to implement the steps of the camera data acquisition method.

[0039] The present application also provides a computer program product, which comprises a computer program; when the computer program is executed by a processor, the steps of the camera data acquisition method are implemented.

[0040] The augmented reality device receives a camera access request, and takes a camera requested to be accessed as a target camera; if the target camera is not in use when the camera access request is received, a system camera support component is run, an image stream captured by the target camera is acquired through the system camera support component, and the image stream is cached to a preset address; camera data of the target camera is acquired from the preset address; if the target camera is in use when the camera access request is received, a virtual camera support component is created, the preset address is copied into the virtual camera support component as a first access interface, the virtual camera support component is run, and the camera data of the target camera is acquired through the first access interface. In this way, when the target camera is not in use, the image stream captured by the target camera is cached, and since the camera system of the AR device requires that the camera support component can only provide the image stream of the same camera to one process, when a subsequent process accesses the target camera again, the virtual camera support component is created in the embodiment of the application to acquire the camera data of the target camera from the cached image stream, instead of using the system camera support component to acquire the image stream of the target camera, so that the initial process using the target camera and the subsequent process using the target camera again acquire the camera data based on an independent camera support component respectively, and the same camera of the AR device can be shared and used by multiple processes. For example, when the AR device plays an OST (Original Sound Track) according to a captured current scene or uses a VST (Virtual Studio Technology) to edit and process audio content in the AR experience based on the captured current scene, gesture control can be supported at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, those skilled in the art can obtain other drawings from these drawings without any creative effort.

[0043] FIG. 1 is a flowchart of a first embodiment of a camera data acquisition method according to the present application;

[0044] FIG. 2 is a schematic diagram of a brief flow of a camera data acquisition method according to an embodiment of the present application;

[0045] Fig. 3 is a flow diagram of a fourth embodiment of the camera data acquisition method of the present application;

[0046] Fig. 4 is a device structure diagram of the camera data acquisition device of the present application;

[0047] Fig. 5 is a device structure diagram of the hardware operating environment involved in the camera data acquisition device of the present application.

[0048] The purposes, features and advantages of the present application will be further described with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and understandable, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0050] In an AR device, when one process is using a camera, other processes cannot access the camera at the same time, because the camera is a hardware resource, and the operating system usually does not allow two different processes to directly control the same hardware resource at the same time.

[0051] Based on this, the main solution of the present application is: receiving a camera access request, taking the requested camera as a target camera; if the target camera is not in use when the camera access request is received, running a system camera support component, obtaining an image stream captured by the target camera through the system camera support component, and caching the image stream to a preset address, obtaining camera data of the target camera from the preset address; if the target camera is in use when the camera access request is received, creating a virtual camera support component, copying the preset address as a first access interface to the virtual camera support component, running the virtual camera support component, and obtaining camera data of the target camera through the first access interface.

[0052] The application caches the image stream captured by the target camera when the target camera is not in use. Since the camera system of the AR device requires that the camera support component can only provide the image stream of the same camera to one process, when a subsequent process accesses the target camera, the application creates a virtual camera support component to obtain the camera data of the target camera from the cached image stream, instead of using the system camera support component to obtain the image stream of the target camera. Thus, the initial process using the target camera and the subsequent process using the target camera obtain camera data based on an independent camera support component, respectively, so that the same camera of the AR device can be shared and used by multiple processes. For example, when the AR device plays an OST (Original Sound Track) according to the captured current scene or uses a VST (Virtual Studio Technology) to edit and process audio content in the AR experience based on the captured current scene, gesture control can be supported at the same time.

[0053] It should be noted that the execution subject of the embodiment can be an augmented reality device capable of realizing the above functions, such as AR glasses, AR headgear, etc. For example, the AR glasses are taken as the execution subject to describe the embodiments of the application.

[0054] Based on this, the application provides a camera data acquisition method of the first embodiment, please refer to FIG. 1, the camera data acquisition method includes steps S10-S30:

[0055] Step S10, receiving a camera access request, taking the camera requested to access as a target camera;

[0056] It should be noted that the AR glasses can be specifically AR glasses based on an Android camera system, but are not limited to AR glasses based on the Android camera system, such as AR glasses based on an IOS (apple) system. The embodiment does not specifically limit this.

[0057] It can be understood that sometimes the AR glasses need to perform corresponding tasks according to the camera data captured by the camera, such as gesture control of the AR glasses, which needs to obtain the camera data captured by the camera to recognize the gesture of the user according to the obtained camera data, and to perform subsequent control according to the recognized gesture; when the AR glasses play an OST, in order to improve the user experience, the camera data captured by the camera needs to be obtained to recognize the scene where the user is currently located according to the obtained camera data, and then the OST corresponding to the scene is played.

[0058] When an application (hereinafter referred to as an application) in the AR glasses needs to obtain the camera data of a certain camera, that is, needs to use the camera, the process can initiate a camera access request for the camera, specifically, the process can encapsulate the camera identifier of the camera in the camera access request, so as to determine the target camera accessed by the current request based on the camera identifier encapsulated in the camera access request.

[0059] It should be noted that after the process initiates the camera access request for the target camera, if the AR glasses receives the access request for the target camera from the process, the process can obtain the camera data of the target camera, that is, the process starts to use the target camera. During the use of the target camera, the process will not send a camera access request again, and only needs to initiate a camera stop access request when the process does not need to use the target camera, so as to terminate the process from obtaining the camera data of the target camera, that is, terminate the use of the target camera by the process.

[0060] If the target camera is not in use when the camera access request is received, the system camera support component is run, the image stream captured by the target camera is obtained through the system camera support component, and the image stream is cached to a preset address, and the camera data of the target camera is obtained from the preset address.

[0061] For the convenience of subsequent description and illustration, if the target camera is not in use when the camera access request is received, the process accessing the target camera is recorded as the process first accessing the target camera.

[0062] The image stream is specifically an image stream composed of image frames captured by the target camera based on a preset capture frequency.

[0063] The camera data of the target camera is obtained from the image stream. Specifically, one frame, multiple frames or all frames of images can be obtained from the image stream as the camera data of the target camera. It is easy to understand that different processes have different requirements for the image frames to be obtained, for example, a gesture recognition process may need to obtain all image frames, and a photographing process may only need to obtain one frame of image. Based on this, the corresponding image frames can be obtained from the image stream as the obtained camera data based on the characteristics of the process itself.

[0064] The preset address is specifically the address of a shared storage area, so that all processes can access the address and thus obtain the image stream cached in the preset address.

[0065] In the architecture of the camera system, in sequence, an application layer, a FrameWork layer, a hardware abstraction layer (HAL) and a hardware layer are included. The hardware abstraction layer (HAL) implements an interface related to specific hardware, which abstracts implementation details of different camera hardware, so that the Android system can access camera devices of different manufacturers in a unified manner. Specifically, CameraProvider (camera support) is a key system-level component of the HAL layer, which is responsible for managing and providing access interfaces of camera hardware. It provides information and status of camera devices to the upper-layer CameraManager (camera management) service, and processes operations such as opening, configuring, data streaming and closing of camera devices. Based on this, the camera support component in the embodiment is specifically CameraProvider.

[0066] The system camera support component is specifically the original CameraProvider of the camera system of the AR glasses.

[0067] In step S30, if the target camera is in use when the camera access request is received, a virtual camera support component is created, the preset address is copied into the virtual camera support component as a first access interface, the virtual camera support component is run, and camera data of the target camera is obtained through the first access interface.

[0068] For the convenience of subsequent description and illustration, if the target camera is in use when the camera access request is received, the process of accessing the target camera is recorded as the process of re-accessing the target camera.

[0069] In the embodiment, the virtual camera support component is specifically a virtual CameraProvider. The virtual CameraProvider represents that it is not an actual CameraProvider. The virtual CameraProvider provides an interface for data access.

[0070] It should be noted that for each process accessing the target camera for the first time, a corresponding virtual camera support component is created, but if a process initiating a camera access request is found to have a virtual camera support component created for it in the AR glasses, the virtual camera support component is not repeatedly created, and the existing virtual camera support component is directly used. For example, assuming that process B initiates a camera access request to access the target camera, at this time process A is already using the target camera, and no virtual camera component is found to be created for process B. A corresponding virtual camera component B is created for process B. At a certain time, process B initiates a camera access request to access the target camera again. At this time, it is found that a corresponding virtual camera component B has been created for process B before, so the virtual camera support component is not repeatedly created, and the existing virtual camera support component B is directly run.

[0071] In the embodiment, the augmented reality device receives a camera access request, and takes a camera requested to be accessed as a target camera. If the target camera is not in use when the camera access request is received, a system camera support component is run, an image stream captured by the target camera is acquired through the system camera support component, and the image stream is cached to a preset address. The camera data of the target camera is acquired from the preset address. If the target camera is in use when the camera access request is received, a virtual camera support component is created, the preset address is copied to the virtual camera support component as a first access interface, the virtual camera support component is run, and the camera data of the target camera is acquired through the first access interface. In this way, in the embodiment, when the target camera is not in use, the image stream captured by the target camera is cached. Since the camera system of the AR device requires that the camera support component can only provide the image stream of the same camera to one process, when a subsequent process accesses the target camera again, the virtual camera support component is created in the embodiment, and the camera data of the target camera is acquired from the cached image stream through the virtual camera support component, instead of using the system camera support component to acquire the image stream of the target camera. Therefore, the process initially using the target camera and the process using the target camera again are based on an independent camera support component to acquire the camera data, and the same camera of the AR device can be shared and used by multiple processes. For example, when the AR device plays an OST (Original Sound Track) according to the captured current scene or uses a VST (Virtual Studio Technology) to edit and process the audio content in the AR experience based on the captured current scene, gesture control can be simultaneously supported.

[0072] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above embodiment one can be referred to the above description, and the subsequent will not be described in detail. On this basis, before the step of running the virtual camera support component, the method further comprises:

[0073] Step B10, obtaining the stream access interface of all pre-configured camera configuration streams, taking the stream access interface of all the camera configuration streams as a second access interface, and copying the second access interface into the virtual camera support component;

[0074] The pre-configured all camera configuration streams can be all camera configuration streams (i.e. camera streams) pre-configured by the HAL layer when accessing the target camera for the first time.

[0075] The second access interface can be a storage address of all camera configuration streams.

[0076] The step of obtaining the camera data of the target camera through the first access interface comprises:

[0077] Step B20, obtaining all camera configuration streams through the second access interface, and selecting a first camera configuration stream from all the camera configuration streams;

[0078] One or more camera configuration streams that are not in use can be randomly selected from all camera configuration streams as the first camera configuration stream based on a preset rule. Wherein, the camera configuration stream not in use is used to indicate that the camera configuration stream is currently not used by the camera system to obtain the image stream.

[0079] Step B30, accessing the cached image stream through the first access interface, and obtaining the camera data of the target camera from the cached image stream based on the first camera configuration stream.

[0080] It can be understood that in the camera system, it is necessary to obtain the image stream of the camera based on the camera configuration stream, and based on this, in the present embodiment, the camera data is obtained from the cached image stream based on the selected first camera configuration stream.

[0081] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as the above embodiment one and embodiment two can be referred to the above description, and the subsequent will not be described in detail. On this basis, the step of selecting a first camera configuration stream from all the camera configuration streams comprises:

[0082] Step C10, obtaining the first configuration parameter encapsulated in the camera access request;

[0083] Step C20, selecting the camera configuration stream with the highest matching degree as the first camera configuration stream from all the camera configuration streams, wherein the more configuration parameters in the camera configuration stream are consistent with the first configuration parameters, the higher the matching degree of the camera configuration stream with the first configuration parameters.

[0084] Further, if there are multiple camera configuration streams with the highest matching degree, one of the multiple camera configuration streams with the highest matching degree can be selected as the first camera configuration stream.

[0085] It should be noted that for a camera configuration stream, the more configuration parameters in the camera configuration stream are consistent with the first configuration parameters, the higher the matching degree of the camera configuration stream with the first configuration parameters.

[0086] As one of the embodiments, the total configuration parameters in the camera configuration stream that are consistent with the first configuration parameters can be taken as the matching degree of the camera configuration stream with the first configuration parameters. For example, assuming that the first configuration parameters are

format A, color space A, resolution A

format A, color space B, resolution C

format A, color space A, resolution B

format A, color space A, resolution A

[0087] As another embodiment, a corresponding weight can be set for each configuration parameter, and the matching degree of the camera configuration stream with the first configuration parameters is calculated based on the weighted sum. For example, assuming that the first configuration parameters are

format A, color space A, resolution A

format A, color space B, resolution C

format A, color space A, resolution B

format A, color space A, resolution A

[0088] It should be noted that the above is only two possible implementation manners of the matching degree calculation between the camera configuration stream and the first configuration parameter provided by the embodiment, and the embodiment does not specifically limit the specific implementation manner of the matching degree calculation between the camera configuration stream and the first configuration parameter.

[0089] In the embodiment, the camera configuration stream with the highest matching degree with the first configuration parameter is selected as the first camera configuration stream. It can be understood that the process usually encapsulates the data format expected to be obtained in the form of a configuration parameter in a camera access request, so that the data format of the image stream obtained based on the first camera configuration stream is closest to the data format expected by the process accessing the target camera, that is, the matching degree between the obtained data format and the expected data format is maximized, so as to reduce the amount of subsequent data format conversion.

[0090] In a possible implementation manner, the step of obtaining the camera data of the target camera from the buffered image stream based on the first camera configuration stream comprises:

[0091] In step D10, if the configuration parameters included in the first camera configuration stream are consistent with the first configuration parameters, at least one frame of image is obtained from the buffered image stream based on the first camera configuration stream as the camera data obtained from the target camera.

[0092] The configuration parameters included in the first camera configuration stream are consistent with the first configuration parameters, that is, the parameter values of the two are consistent. For example, assuming that a configuration parameter includes image format, color space and resolution, the configuration parameters included in the first camera configuration stream are consistent with the first configuration parameters, that is, the image format in the first camera configuration stream is consistent with the image format in the first configuration parameters, the color space in the first camera configuration stream is consistent with the color space in the first configuration parameters, and the resolution in the first camera configuration stream is consistent with the resolution in the first configuration parameters. At this time, since the configuration parameters are completely consistent, the image stream obtained based on the first camera configuration stream is an expected data format image stream, and data format conversion is not required.

[0093] In step D20, if there is a configuration parameter inconsistent with the first configuration parameter in the configuration parameters included in the first camera configuration stream, at least one frame of image is obtained from the buffered image stream based on the first camera configuration stream, image conversion is performed on all the obtained images, and the converted image is taken as the camera data obtained from the target camera, wherein the image parameter of the converted image is the first configuration parameter.

[0094] The number of frames of images and the frames of images obtained from the buffered image stream based on the first camera configuration stream can be obtained based on the requirements of the process itself, and the embodiment does not make specific limitation.

[0095] The obtained all images are converted, and specifically, the conversion can be based on inconsistent configuration parameters. For example, it is assumed that the configuration parameters contained in the first camera configuration stream are

image format A, color space A, resolution B

image format A, color space A, resolution A

[0096] In the embodiment, when the configuration parameters in the first camera configuration stream are inconsistent with the first configuration parameters, the images are converted to convert the image format, color space, resolution and other image parameters of the images to the first configuration parameters, so that the data format of the obtained camera data meets the expectation of the process.

[0097] Based on the first embodiment, the second embodiment and / or the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as the above-mentioned first embodiment, the second embodiment and the third embodiment can be referred to the above introduction, and will not be repeated hereinafter. On this basis, referring to FIG. 3, the step of obtaining the image stream shot by the target camera comprises:

[0098] Step E10, obtaining all configuration parameter groups according to the camera access request, wherein each configuration parameter group comprises one or more configuration parameters;

[0099] Further, when the configuration parameter group comprises a plurality of configuration parameters, the plurality of configuration parameters are specifically a plurality of configuration parameters of different parameter types. Specifically, the parameter types of the configuration parameters include but are not limited to image format, color space and resolution, and the embodiment does not make specific limitation, and the parameter types required to be configured in the actual camera system configuration camera stream (i.e. camera configuration stream) are used as the standard.

[0100] Further, based on different parameter types, the configuration parameters encapsulated in the camera access request and the configuration parameters supported by the AR glasses are arranged and combined to obtain all configuration parameter groups. For example, assuming that each configuration parameter group includes configuration parameters of two parameter types of image format and resolution, the configuration parameters encapsulated in the camera access request are

format A, resolution A

format A, resolution A

format A, resolution B

format B, resolution A

format B, resolution B

format C, resolution A

format C, resolution B

[0101] Step E20, configuring a camera configuration stream according to each configuration parameter group, wherein each configuration parameter group corresponds to a camera configuration stream;

[0102] The specific configuration manner of the configuration parameter group configuring the camera configuration stream can use the existing camera configuration stream configuration manner of the Android camera system HAL layer, and the configuration process of the camera configuration stream is not described in detail in the embodiment. It should be noted that after each camera configuration stream is configured, the camera configuration stream includes the configuration parameters in the configuration parameter group and a generated pipeline, wherein the pipeline is a pipe in the HAL layer for managing image data flow, which ensures that image data is transmitted in an efficient and consistent manner, while processing possible data format conversion, the pipeline can include multiple processing stages such as denoising, white balance adjustment, color correction, cropping, scaling, etc., which helps to improve image quality and meet the needs of specific applications, and can be understood as a pipe for transmitting image flow. The configuration parameters in the configuration parameter group are used to indicate the processing target of the pipeline.

[0103] Step E30, selecting a camera configuration stream from all the camera configuration streams, and taking the selected camera configuration stream as a target camera configuration stream;

[0104] The AR glasses need to acquire an image stream based on a camera configuration stream, based on which, one target camera configuration stream can be randomly selected from all the camera configuration streams or selected based on a preset rule, so that the image stream can be successfully acquired.

[0105] Step E40, acquiring the image stream captured by the target camera based on the target camera configuration stream.

[0106] When an image stream is acquired based on a camera configuration stream, specifically, the image stream can be acquired through a pipeline in the camera configuration stream. The image stream acquired actually can be understood as the stream data output after the pipeline.

[0107] It can be understood that the HAL layer of the camera system can configure the camera configuration stream for the same camera only once. Based on this, when the camera access request of the process accessing the target camera for the first time is received, the camera configuration stream is configured based on the configuration parameters encapsulated in the camera access request, and all camera configuration streams are configured based on the configuration parameters supported by the AR glasses. In this way, the camera configuration streams of the processes accessing the target camera subsequently are pre-configured, so that the processes accessing the target camera subsequently acquire the image stream based on the pre-configured camera configuration stream, and the AR glasses can support different processes to acquire the image stream based on different camera configuration streams. The formats of the image streams acquired by different camera configuration streams are different, so that the AR glasses can support the acquisition of multiple formats of image streams, and the intelligence of the shared use of the camera is improved.

[0108] In a possible implementation, the step of selecting one camera configuration stream from all the camera configuration streams and taking the selected camera configuration stream as a target camera configuration stream comprises:

[0109] The step F10 of taking the camera configuration stream configured based on the configuration parameters encapsulated in the camera access request from all the camera configuration streams as a target camera configuration stream.

[0110] In the embodiment, the camera configuration stream configured based on the configuration parameters encapsulated in the camera access request is taken as a target camera configuration stream. It can be understood that the process usually encapsulates the data format expected to be acquired in the form of configuration parameters in the camera access request. In this way, the data format of the image stream acquired based on the target camera configuration stream can meet the expectation of the process accessing the target camera for the first time.

[0111] Based on the first embodiment, the second embodiment, the third embodiment and / or the fourth embodiment of the present application, the same or similar contents as those in the above-mentioned first embodiment, the second embodiment, the third embodiment and the fourth embodiment can be referred to the above description, and will not be described hereinafter. On this basis, after the step of buffering the image stream to a preset address, the method further comprises:

[0112] The step G10 of uploading the image stream to a preset server and returning to the step of receiving the camera access request;

[0113] The image stream is uploaded to a preset server, and specifically, the image stream can be copied to the preset server.

[0114] After the step of taking the camera requested to be accessed as the target camera, the method further comprises:

[0115] Step G20, determining an application requesting to access the target camera according to the camera access request;

[0116] Step G30, if the application requesting to access the target camera is a non-native application, obtaining camera data of the target camera from the image stream uploaded to the preset server.

[0117] The native application refers to an application program that can be directly run on the current operating system and is specially developed for a certain operating system, such as iOS, Android, etc. The native application is built for a specific device platform using a specific programming language. The non-native application specifically refers to other applications other than the native application, such as user-defined applications, web applications, etc. In this embodiment, the non-native application can be a non-Android system native application.

[0118] In this embodiment, the non-native application obtains the camera data from the preset server, which on the one hand enables the non-native application to also share the use of the camera, and on the other hand directly from the preset server can improve the speed of obtaining the camera data, realizing low-latency acquisition of the camera data.

[0119] By way of example, in order to help understand the technical concept or technical principle of the camera data acquisition after the combination of the above-mentioned embodiment one, embodiment two, embodiment three and embodiment four, please refer to FIG. 2, which provides a brief flowchart of a camera data acquisition method, as follows:

[0120] 1. Process A and process B access the same camera through the Camera API (Camera Application Programming Interface) of the Android camera system, wherein process A and process B are processes of the native Camera application of the Android platform.

[0121] 2、In the Camera server module, if the camera that has been used is opened, because the camera system can only give a camera identifier to one process for the same camera, a virtual camera identifier is generated, and the virtual camera identifier is transmitted to the HAL layer, so that the camera system can receive the access request of the process for subsequent processing, instead of kicking off the process that is using the camera or directly rejecting the access request of the process. The Camera server module is the native Camera server module of the Android camera system, and includes the Camera service, the CameraDeviceClient, the Camera3Device, the CameraProviderManger, and the like.

[0122] 3、In the Camera provider module, the opening Camera logic is modified, when a camera that is not occupied is opened, all camera streams (including all image formats, color spaces, and all resolutions, all pipelines are generated), and are sent to the Camera drvier, and all camera streams are sent to the Camera drvier, and the obtained image stream is cached. When the same camera is opened for the second time, a virtual CameraProvider (Camera fake Provider shown in FIG. 2) is created, and subsequent camera data is obtained from the cached image stream, so that data sharing is achieved.

[0123] 4、The image stream is shared to the server (server stream shown in FIG. 2), and when the three-party algorithm in the non-native application needs to use the camera, the camera data is obtained from the server, and the image stream is transmitted from the server to the client of the non-native application (client stream shown in FIG. 2).

[0124] It should be noted that the above specific embodiments are only used to understand the present application, and do not constitute a limitation on the camera data acquisition process of the present application, and more forms of simple transformation based on the technical concept are within the protection scope of the present application.

[0125] In addition, the embodiment of the present application also provides a camera data acquisition device, referring to FIG. 4, the camera data acquisition device comprises:

[0126] The receiving module 10 is configured to receive a camera access request, and take the camera requested to be accessed as a target camera.

[0127] The sharing module 20 is configured to, if the target camera is not in use when the camera access request is received, run a system camera support component, acquire an image stream captured by the target camera through the system camera support component, and cache the image stream to a preset address, and acquire camera data of the target camera from the preset address.

[0128] The acquiring module 30 is configured to, if the target camera is in use when the camera access request is received, create a virtual camera support component, copy the preset address as a first access interface to the virtual camera support component, run the virtual camera support component, and acquire the camera data of the target camera through the first access interface.

[0129] In addition, as shown in FIG. 5, the augmented reality device can further include a processing apparatus 1001 (for example, a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage apparatus 1003 to a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for operation of the augmented reality device are also stored. The processing apparatus 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input apparatus 1007 including, for example, a touch screen, a touch pad, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output apparatus 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the augmented reality device to communicate with other devices wirelessly or by wire to exchange data. Although the augmented reality device with various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0130] In particular, according to the embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments of the present application are executed.

[0131] The augmented reality device provided by the embodiments of the present application adopts the camera data acquisition method in the above embodiments, and can solve the technical problem of camera data acquisition. Compared with the prior art, the augmented reality device provided by the present application has the same beneficial effects as the camera data acquisition method provided by the above embodiments, and other technical features in the augmented reality device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0132] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0133] The above describes only the specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0134] In addition, to achieve the above-mentioned object, the embodiments of the present application also provide a readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the camera data acquisition method in the above embodiments.

[0135] The computer readable storage medium provided by the embodiments of the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0136] The above computer readable storage medium may be contained in an augmented reality device, or may exist separately and not be assembled into an augmented reality device.

[0137] The above computer readable storage medium carries one or more programs, when the one or more programs are executed by the augmented reality device, the augmented reality device receives a camera access request, and takes the requested camera as a target camera; if the target camera is not in use when the camera access request is received, a system camera support component is run, the image stream captured by the target camera is obtained through the system camera support component, and the image stream is cached to a preset address, the camera data of the target camera is obtained from the preset address; if the target camera is in use when the camera access request is received, a virtual camera support component is created, the preset address is copied to the virtual camera support component as a first access interface, the virtual camera support component is run, and the camera data of the target camera is obtained through the first access interface.

[0138] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0139] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0140] The modules involved in the embodiments of the present application can be implemented in software or hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0141] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above-mentioned camera data acquisition method, and can solve the technical problem of camera data acquisition. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the camera data acquisition method provided by the above-mentioned embodiments, which will not be described here.

[0142] In addition, the embodiment of the present application further provides a computer program product, comprising a camera data acquisition program, which, when executed by a processor, implements the steps of the camera data acquisition method.

[0143] The computer program product embodiment of the present application is basically the same as the above-mentioned camera data acquisition method embodiments, and will not be described here.

[0144] It should be noted that in this paper, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or system. Without more limitations, the element defined by the sentence "comprises a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0145] The above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by software and necessary general hardware platform, of course, also can be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software sensor, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a number of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner or network device) execute the methods described in various embodiments of the present application.

[0147] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A video data acquisition method, characterized by, The application is applied to an augmented reality device, and the camera data acquisition method comprises the following steps: receiving a camera access request, and taking a requested camera as a target camera; if the target camera is not in use when the camera access request is received, running a system camera support component, acquiring an image stream photographed by the target camera through the system camera support component, and caching the image stream to a preset address, and acquiring camera data of the target camera from the preset address; if the target camera is in use when the camera access request is received, creating a virtual camera support component, copying the preset address as a first access interface into the virtual camera support component, and running the virtual camera support component to acquire camera data of the target camera through the first access interface.

2. The method of claim 1, wherein, Before the step of running the virtual camera support component, the method further comprises: acquiring stream access interfaces of all preconfigured camera configuration streams, taking the stream access interfaces of all the camera configuration streams as second access interfaces, and copying the second access interfaces into the virtual camera support component; the step of acquiring camera data of the target camera through the first access interface comprises: acquiring all camera configuration streams through the second access interfaces, selecting a first camera configuration stream from all the camera configuration streams, accessing the cached image stream through the first access interface, and acquiring camera data of the target camera from the cached image stream based on the first camera configuration stream. the step of selecting a first camera configuration stream from all the camera configuration streams comprises:

3. The method of claim 2, wherein, acquiring a first configuration parameter encapsulated in the camera access request; taking a camera configuration stream with the highest matching degree to the first configuration parameter from all the camera configuration streams as the first camera configuration stream, wherein the more configuration parameters in the camera configuration stream are consistent with the first configuration parameter, the higher the matching degree of the camera configuration stream to the first configuration parameter is. the step of acquiring camera data of the target camera from the cached image stream based on the first camera configuration stream comprises:

4. The method of claim 3, wherein, if configuration parameters contained in the first camera configuration stream are all consistent with the first configuration parameter, acquiring at least one frame of image from the cached image stream based on the first camera configuration stream as camera data acquired from the target camera; if there is a configuration parameter inconsistent with the first configuration parameter in the configuration parameters contained in the first camera configuration stream, acquiring at least one frame of image from the cached image stream based on the first camera configuration stream, performing image conversion on all acquired images, and taking the converted image as camera data acquired from the target camera, wherein an image parameter of the converted image is the first configuration parameter. the step of acquiring an image stream photographed by the target camera comprises:

5. The method of claim 1, wherein, acquiring all configuration parameter groups according to the camera access request, wherein each configuration parameter group comprises one or more configuration parameters; ​ configuring a camera configuration stream according to each of the configuration parameter groups, wherein each of the configuration parameter groups corresponds to a camera configuration stream; selecting one camera configuration stream from all the camera configuration streams, and taking the selected camera configuration stream as a target camera configuration stream; obtaining an image stream captured by the target camera based on the target camera configuration stream.

6. The method of claim 5, wherein, The step of selecting one camera configuration stream from all the camera configuration streams, and taking the selected camera configuration stream as a target camera configuration stream, comprises: taking, as the target camera configuration stream, a camera configuration stream obtained based on the configuration parameter encapsulated in the camera access request from all the camera configuration streams.

7. The method according to any one of claims 1 to 6, characterized in that, After the step of caching the image stream to a preset address, the method further comprises: uploading the image stream to a preset server, and returning to the step of receiving the camera access request; After the step of taking the camera requested to be accessed as a target camera, the method further comprises: determining an application requesting to access the target camera according to the camera access request; if the application requesting to access the target camera is a non-native application, obtaining camera data of the target camera from the image stream uploaded to the preset server.

8. An image pickup data acquisition apparatus characterized by comprising: The camera data obtaining apparatus comprises: a receiving module configured to receive a camera access request, and take a camera requested to be accessed as a target camera; a sharing module configured to, if the target camera is not in use when the camera access request is received, run a system camera support component, obtain an image stream captured by the target camera through the system camera support component, and cache the image stream to a preset address, and obtain camera data of the target camera from the preset address; a obtaining module configured to, if the target camera is in use when the camera access request is received, create a virtual camera support component, copy the preset address as a first access interface to the virtual camera support component, run the virtual camera support component, and obtain camera data of the target camera through the first access interface.

9. An augmented reality device, characterized by The augmented reality device comprises a memory, a processor, and a camera data obtaining program stored in the memory and executable on the processor, and the camera data obtaining program, when executed by the processor, implements the steps of the camera data obtaining method according to any one of claims 1 to 7.

10. A readable storage medium, characterized by, The readable storage medium is a computer readable storage medium, and the computer readable storage medium stores a camera data obtaining program, and the camera data obtaining program, when executed by a processor, implements the steps of the camera data obtaining method according to any one of claims 1 to 7.

Images