Imaging data acquisition method and apparatus, and augmented reality device and storage medium

Abstract

The present application relates to the technical field of augmented reality. Disclosed are an imaging data acquisition method and apparatus, and an augmented reality device and a storage medium. The imaging data acquisition method is applied to an augmented reality device. The imaging data acquisition method comprises: receiving a camera access request; if a target camera is not in use when the camera access request is received, running a system camera support component, acquiring, by means of the system camera support component, an image stream captured by the target camera, acquiring imaging data of the target camera from the image stream, and copying the image stream to obtain a copy of the image stream; and if the target camera is in use when the camera access request is received, running the system camera support component, and acquiring the imaging data of the target camera from the copy of the image stream by means of the system camera support component. The present application enables shared camera use in augmented reality devices.

Description

Camera data acquisition method and device, augmented reality device, and storage medium

[0001] The present application claims priority to the Chinese patent application No. 202410841010.1, filed on June 26, 2024, and entitled "Camera data acquisition method and device, augmented reality device, and storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of augmented reality technology, and in particular to a camera data acquisition method and device, an augmented reality device, and a storage medium. BACKGROUND

[0003] An AR (Augmented Reality) device can superimpose a real environment and a virtual video in real time into the same picture or space for a wearer to view. Common AR devices include AR glasses or AR headsets, etc.

[0004] However, the camera system of the current AR device only supports the use of the same camera by one process, i.e., only one process is allowed to acquire camera data captured by the camera. For example, if a camera in an AR device is used for gesture control, the camera cannot be used for code scanning recognition, object tracking, etc. Other processes will disconnect the use of the previous process through certain strategies to ensure that the same camera is only used by one process. SUMMARY

[0005] The main purpose of the present application is to provide a camera data acquisition method and device, an augmented reality device, and a storage medium, which aims to solve the technical problem that the same camera of the current AR device can only be used by one process.

[0006] To achieve the above-mentioned purpose, the present application provides a camera data acquisition method, which is applied to an augmented reality device, and includes the following steps:

[0007] Receiving a camera access request, and taking the requested camera as a target camera;

[0008] If the target camera is not in use when the camera access request is received, running a system camera support component, acquiring an image stream captured by the target camera through the system camera support component, acquiring camera data of the target camera from the image stream, and copying the image stream to obtain an image stream copy;

[0009] If the target camera is in use when the camera access request is received, the system camera support component is run to obtain camera data of the target camera from the image stream copy via the system camera support component.

[0010] In an embodiment, the step of obtaining the image stream captured by the target camera comprises:

[0011] All configuration parameter groups are obtained according to the camera access request, wherein each configuration parameter group comprises one or more configuration parameters.

[0012] Camera configuration streams are configured according to each configuration parameter group, wherein each configuration parameter group corresponds to a camera configuration stream.

[0013] A camera configuration stream is selected from all camera configuration streams, and the selected camera configuration stream is taken as a target camera configuration stream.

[0014] The image stream captured by the target camera is obtained based on the target camera configuration stream.

[0015] In an embodiment, the step of selecting a camera configuration stream from all camera configuration streams and taking the selected camera configuration stream as a target camera configuration stream comprises:

[0016] The camera configuration stream obtained based on the configuration parameters encapsulated in the camera access request from all camera configuration streams is taken as a target camera configuration stream.

[0017] In an embodiment, the step of obtaining camera data of the target camera from the image stream copy comprises:

[0018] All camera configuration streams are obtained according to the camera access request, from all camera configuration streams, a first camera configuration stream is selected;

[0019] Camera data of the target camera is obtained from the image stream copy based on the first camera configuration stream.

[0020] In an embodiment, the step of selecting a first camera configuration stream from all camera configuration streams comprises:

[0021] A first configuration parameter encapsulated in the camera access request is obtained;

[0022] The camera configuration stream with the highest matching degree to the first configuration parameter from all camera configuration streams is taken as a first camera configuration stream, wherein the more configuration parameters in the camera configuration stream that are consistent with the first configuration parameter, the higher the matching degree of the camera configuration stream to the first configuration parameter.

[0023] In an embodiment, the step of obtaining the camera data of the target camera from the image stream copy based on the first camera configuration stream comprises:

[0024] if the configuration parameters included in the first camera configuration stream are consistent with the first configuration parameters, obtaining at least one frame of image from the image stream copy as the camera data obtained from the target camera based on the first camera configuration stream;

[0025] if there are configuration parameters inconsistent with the first configuration parameters in the configuration parameters included in the first camera configuration stream, obtaining at least one frame of image from the image stream copy based on the first camera configuration stream, performing image conversion on all obtained images, and taking the converted image as the camera data obtained from the target camera, wherein the image parameters of the converted image are the first configuration parameters.

[0026] In an embodiment, after the step of copying the image stream to obtain an image stream copy, the method further comprises:

[0027] uploading the image stream copy to a preset server, and returning to perform the step of receiving the camera access request;

[0028] After the step of taking the camera requested to be accessed as the target camera, the method further comprises:

[0029] determining the application requesting to access the target camera according to the camera access request;

[0030] if the application requesting to access the target camera is a non-native application, obtaining the camera data of the target camera from the image stream copy uploaded to the preset server.

[0031] In addition, to achieve the above-mentioned purposes, the present application also provides a camera data obtaining device, which comprises:

[0032] a receiving module 10, configured to receive a camera access request and take the camera requested to be accessed as a target camera;

[0033] a sharing module 20, configured to, if the target camera is not in use when the camera access request is received, run a system camera support component, obtain an image stream photographed by the target camera through the system camera support component, obtain camera data of the target camera from the image stream, and copy the image stream to obtain an image stream copy;

[0034] An acquisition module 30 is configured to acquire the camera data of the target camera from the image stream copy by running the system camera support component if the target camera is in use when the camera access request is received.

[0035] In addition, the present application also provides an augmented reality device to achieve the above-mentioned purposes, which comprises a microphone array and a processor, wherein the microphone array is electrically connected with the processor; and the processor is configured to execute the steps of the camera data acquisition method.

[0036] In addition, the present application also provides a readable storage medium to achieve the above-mentioned purposes, wherein the readable storage medium is a computer readable storage medium, and the computer readable storage medium stores a program for implementing the camera data acquisition method; and the program is executed by a processor to implement the steps of the camera data acquisition method.

[0037] The present application also provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the steps of the camera data acquisition method.

[0038] The augmented reality device receives a camera access request in the application, takes the camera requested to be accessed as a target camera, and runs a system camera support component if the target camera is not in use when the camera access request is received. The system camera support component is used to obtain an image stream captured by the target camera, obtain camera data of the target camera from the image stream, and copy the image stream to obtain an image stream copy. If the target camera is in use when the camera access request is received, the system camera support component is run to obtain camera data of the target camera from the image stream copy. In this way, when the target camera is not in use in the embodiment of the application, the image stream captured by the target camera is backed up, and when the target camera is used again in a subsequent process, the camera data of the target camera can be obtained from the backup image stream copy, instead of obtaining the image stream of the target camera again. Therefore, the process that initially uses the target camera uses the target camera based on the obtained image stream, and the process that uses the target camera again subsequently indirectly uses the target camera based on the backup image stream copy, instead of directly accessing or controlling the target camera. The backup image stream copy indirectly realizes the sharing use of the target camera by multiple processes, and the same camera of the AR device can be shared and used by multiple processes, for example, the AR device can simultaneously support gesture control when playing an OST (Original Sound Track) based on the captured current scene or editing and processing audio content in the AR experience based on the captured current scene using VST (Virtual Studio Technology). BRIEF DESCRIPTION OF DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and serve to explain the principles of the application together with the specification.

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0041] FIG. 1 is a flowchart of a first embodiment of the camera data acquisition method of the application;

[0042] FIG. 2 is a schematic diagram of a brief flow of the camera data acquisition method according to an embodiment of the application;

[0043] FIG. 3 is a flowchart of a third embodiment of the camera data acquisition method of the application;

[0044] Fig. 4 is a schematic diagram of the device structure of the camera data acquisition device of the present application;

[0045] Fig. 5 is a schematic diagram of the device structure of the hardware operating environment involved in the camera data acquisition device of the embodiment of the present application.

[0046] The object implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0047] In order to make the above object, features and advantages of the present application more apparent and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0048] In the AR device, when one process is using the camera, other processes cannot access the camera at the same time, because the camera is a hardware resource, and the operating system usually does not allow two different processes to directly control the same hardware resource at the same time.

[0049] Based on this, the main solution of the present application is: receiving a camera access request, taking the camera requested to be accessed as a target camera; if the target camera is not in use when the camera access request is received, running a system camera support component, obtaining an image stream captured by the target camera through the system camera support component, obtaining camera data of the target camera from the image stream, and copying the image stream to obtain an image stream copy; if the target camera is in use when the camera access request is received, running the system camera support component, and obtaining camera data of the target camera from the image stream copy through the system camera support component.

[0050] In the present application, when the target camera is not in use, the image stream captured by the target camera is backed up, and when the target camera is used again by a subsequent process, the camera data of the target camera can be obtained from the backed-up image stream copy, without obtaining the image stream of the target camera again, so that the process that initially uses the target camera uses the target camera based on the obtained image stream, and the process that uses the target camera again subsequently indirectly uses the target camera based on the backed-up image stream copy, rather than directly accessing or controlling the target camera, and multiple processes share the use of the target camera indirectly based on the backed-up image stream copy, so that the same camera of the AR device can be shared and used by multiple processes.

[0051] It should be noted that the execution subject of each embodiment of the camera data acquisition method is an augmented reality device capable of achieving the above functions, such as AR glasses, AR headgear, and the like. Exemplarily, the AR glasses are taken as the execution subject to describe each embodiment of the present application.

[0052] Based on this, the camera data acquisition method of the first embodiment is proposed. Please refer to FIG. 1. The camera data acquisition method includes steps S10-S30:

[0053] In step S10, a camera access request is received, and the camera requested to be accessed is taken as a target camera.

[0054] It should be noted that the AR glasses can be specifically AR glasses based on an Android camera system, but are not limited to AR glasses based on the Android camera system, such as AR glasses based on Apple (IOS). The present embodiment does not make specific limitations on this.

[0055] The AR glasses need to perform corresponding tasks according to the camera data photographed by the camera. For example, when the AR glasses perform gesture control, the camera data photographed by the camera needs to be acquired to recognize the user's gesture according to the acquired camera data, and subsequent control is performed according to the recognized gesture. When the AR glasses play an OST, in order to improve the user experience, the camera data photographed by the camera needs to be acquired to recognize the current scene in which the user is located according to the acquired camera data, and then the OST corresponding to the scene is played.

[0056] When an application (hereinafter referred to as an application) in the AR glasses needs to acquire the camera data of a certain camera, that is, needs to use the camera, a camera access request for the camera can be initiated by a process. Specifically, the process can encapsulate the camera identifier of the camera in the camera access request, so as to determine the target camera accessed by the present request based on the camera identifier encapsulated in the camera access request.

[0057] It should be noted that after the process initiates the camera access request for the target camera, if the AR glasses receives the access request for the target camera from the process, the process can acquire the camera data of the target camera, that is, the process starts to use the target camera. During the use of the target camera, the process will not send a camera access request again. Only when the process does not need to use the target camera, a camera stop access request is initiated to terminate the process from acquiring the camera data of the target camera, that is, to terminate the use of the target camera by the process.

[0058] In step S20, if the target camera is not in use when the camera access request is received, a system camera support component is run, the image stream captured by the target camera is obtained through the system camera support component, the camera data of the target camera is obtained from the image stream, and an image stream copy is obtained by copying the image stream.

[0059] For the sake of subsequent elaboration and description, if the target camera is not in use when the camera access request is received, the process of accessing the target camera is recorded as the first process of accessing the target camera.

[0060] The image stream specifically refers to an image stream composed of image frames captured by the target camera based on a preset capture frequency.

[0061] The camera data of the target camera is obtained from the image stream. Specifically, one frame, multiple frames, or all frames of images can be obtained from the image stream as the camera data of the target camera. It is easy to understand that different processes have different requirements for the image frames to be obtained, such as a gesture recognition process that may need to obtain all image frames, and a photographing process that may only need to obtain one frame of image. Based on this, the corresponding image frames can be obtained from the image stream as the obtained camera data based on the characteristics of the process itself.

[0062] Further, after obtaining the image stream copy, the access interface of the image stream copy (such as the storage address of the image stream copy) can be shared, so that subsequent processes accessing the target camera can access the image stream copy through the shared stream access interface.

[0063] It can be understood that in the architecture of the camera system of the AR glasses, there are in turn an application layer, a framework layer (FrameWork layer), a hardware abstraction layer (HAL), and a hardware layer. The hardware abstraction layer (HAL) implements interfaces related to specific hardware, which abstracts the implementation details of different camera hardware, enabling the camera system to access camera devices from different manufacturers in a unified manner. Specifically, CameraProvider (camera support) is a key system-level component of the HAL layer, responsible for managing and providing access interfaces for camera hardware. It provides information and status of camera devices to the CameraManager (camera management) service in the upper layer, and handles operations such as opening, configuring, data streaming, and closing of camera devices. Based on this, the system camera support component in the present embodiment is specifically CameraProvider.

[0064] In step S30, if the target camera is in use when the camera access request is received, the system camera support component is run, and the camera data of the target camera is obtained from the image stream copy through the system camera support component.

[0065] For the convenience of subsequent elaboration and description, if the target camera is in use when the camera access request is received, the process of accessing the target camera is recorded as the process of re-accessing the target camera.

[0066] The process of first accessing the target camera obtains camera data from the original image stream, while the process of subsequent re-accessing the target camera obtains camera data from the image stream copy. In this way, the process of subsequent re-accessing the target camera actually uses the target camera indirectly through the image stream copy, and the process of first accessing the target camera actually uses the target camera independently of the camera usage logic of the AR glasses.

[0067] All processes obtain camera data through the CameraProvider camera support component originally in the camera system. In this way, the modification amount of the camera system architecture is small, thereby reducing the implementation cost of shared use of the camera.

[0068] In the embodiment, the augmented reality device receives a camera access request, and takes the camera requested to be accessed as a target camera. If the target camera is not in use when the camera access request is received, a system camera support component is run, the image stream captured by the target camera is obtained through the system camera support component, camera data of the target camera is obtained from the image stream, and an image stream copy is obtained by copying the image stream. If the target camera is in use when the camera access request is received, the system camera support component is run, and camera data of the target camera is obtained from the image stream copy through the system camera support component. In this way, in the embodiment, when the target camera is not in use, a backup operation is performed on the image stream captured by the target camera. When the target camera is used again by a subsequent process, camera data of the camera can be obtained from the backup image stream copy, without obtaining the image stream of the target camera again. Thus, the process of initially using the target camera uses the target camera based on the obtained image stream, and the process of subsequently re-using the target camera indirectly uses the target camera based on the backup image stream copy, rather than directly accessing or controlling the target camera. The backup image stream copy indirectly enables multiple processes to share the use of the target camera, and enables the same camera of the AR device to be shared by multiple processes. For example, when the AR device plays an OST (Original Sound Track) based on the current scene captured or uses VST (Virtual Studio Technology) to edit and process audio content in the AR experience based on the current scene captured, gesture control can be simultaneously supported.

[0069] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above description, and the subsequent will not be repeated. On this basis, referring to FIG. 3, the step of obtaining the image stream captured by the target camera includes:

[0070] Step B10, obtaining all configuration parameter groups according to the camera access request, wherein each configuration parameter group includes one or more configuration parameters;

[0071] Further, when the configuration parameter group includes a plurality of configuration parameters, the plurality of configuration parameters are specifically a plurality of configuration parameters of different parameter types. Specifically, the parameter types of the configuration parameters include but are not limited to image format, color space and resolution, and the present embodiment does not make specific limitation thereto, and the parameter types required to be configured when configuring the camera stream (i.e. camera configuration stream) are subject to the actual camera system.

[0072] Further, based on different parameter types, the configuration parameters encapsulated in the camera access request and the configuration parameters supported by the AR glasses are arranged and combined to obtain all configuration parameter groups. For example, it is assumed that each configuration parameter group includes configuration parameters of two parameter types of image format and resolution, the configuration parameters encapsulated in the camera access request are

format A, resolution A

format A, resolution A

format A, resolution B

format B, resolution A

format B, resolution B

format C, resolution A

format C, resolution B

[0073] Step B20, configuring the camera configuration stream according to each configuration parameter group, wherein each configuration parameter group corresponds to a camera configuration stream;

[0074] The configuration parameter group configures the specific configuration mode of the camera configuration stream. The configuration mode of the camera configuration stream can use the existing camera configuration stream configuration mode of the Android camera system HAL layer. The configuration process of the camera configuration stream is not described in detail in the embodiment. It should be noted that after each camera configuration stream is configured, the camera configuration stream includes the configuration parameters in the configuration parameter group and a generated pipeline. The pipeline is a pipe in the HAL layer for managing image data streams, which ensures that image data is transmitted in an efficient and consistent manner, while processing possible data format conversion. The pipeline can include multiple processing stages, such as denoising, white balance adjustment, color correction, cropping, scaling, etc. These processing stages help improve image quality and meet the needs of specific applications. It can be understood as a pipe for transmitting image streams. The configuration parameters in the configuration parameter group are used to indicate the processing target of the pipeline.

[0075] Step B30, selecting one camera configuration stream from all the camera configuration streams, and taking the selected camera configuration stream as a target camera configuration stream;

[0076] When the AR glasses acquire an image stream, it needs to be acquired based on a camera configuration stream. Based on this, one or more target camera configuration streams can be randomly selected from all camera configuration streams based on a preset rule, so that the image stream can be successfully acquired.

[0077] Step B40, acquiring the image stream captured by the target camera based on the target camera configuration stream.

[0078] The image stream acquired based on the target camera configuration stream can be acquired through the pipeline in the target camera configuration stream. The output data format of the pipeline is the image stream of the configuration parameters. The actual acquired image stream can be understood as the stream data output after the pipeline.

[0079] It can be understood that the HAL layer of the camera system can only configure one camera configuration stream for the same camera. Based on this, when the camera access request of the process accessing the target camera for the first time is received, not only the configuration parameters encapsulated in the camera access request are used to configure the camera configuration stream, but also all the camera configuration streams are configured based on the configuration parameters supported by the AR glasses. In this way, the camera configuration stream of the process accessing the target camera is pre-configured, so that the process accessing the target camera can acquire the image stream based on the pre-configured camera configuration stream, and the AR glasses can support different processes to acquire image streams based on different camera configuration streams. The formats of the image streams acquired by different camera configuration streams are different, so that the AR glasses can support the acquisition of multiple formats of image streams, and the intelligence of the shared use of the camera is improved.

[0080] In a possible implementation, the step of selecting one camera configuration stream from all the camera configuration streams, and taking the selected camera configuration stream as a target camera configuration stream, comprises:

[0081] The step C10 of taking the camera configuration stream configured based on the configuration parameter encapsulated in the camera access request as a target camera configuration stream from all the camera configuration streams.

[0082] In the embodiment, the camera configuration stream configured based on the configuration parameter encapsulated in the camera access request is taken as the target camera configuration stream. It can be understood that a process usually encapsulates the data format expected to be obtained in the form of a configuration parameter in a camera access request. In this way, the data format of the image stream obtained based on the target camera configuration stream can meet the expectation of the process that first accesses the target camera.

[0083] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned first embodiment and second embodiment can be referred to the above description, and will not be described hereinafter. On this basis, the step of obtaining the camera data of the target camera from the image stream copy comprises:

[0084] The step D10 of obtaining all pre-configured camera configuration streams, and selecting one first camera configuration stream from all the camera configuration streams.

[0085] The all pre-configured camera configuration streams are specifically all the camera configuration streams configured when the target camera is first accessed.

[0086] One or more camera configuration streams not in use can be randomly selected from all the camera configuration streams or selected based on a preset rule as the first camera configuration stream. The camera configuration stream not in use is used to indicate that the camera configuration stream is not currently used by the camera system to obtain an image stream.

[0087] The step D20 of obtaining the camera data of the target camera from the image stream copy based on the first camera configuration stream.

[0088] It can be understood that in the camera system, the image stream of the camera needs to be obtained based on the camera configuration stream. Based on this, in the embodiment, the camera data is obtained from the image stream copy based on the selected first camera configuration stream. The actually obtained image stream is the stream data output by the first camera configuration stream from the image stream copy.

[0089] In a possible implementation, the step of selecting one first camera configuration stream from all the camera configuration streams comprises:

[0090] Step E10, obtaining the first configuration parameter encapsulated in the camera access request;

[0091] Step E20, selecting the camera configuration stream with the highest matching degree as the first camera configuration stream, wherein the more configuration parameters consistent with the first configuration parameter in the camera configuration stream, the higher the matching degree of the camera configuration stream with the first configuration parameter.

[0092] Further, if there are multiple camera configuration streams with the highest matching degree, one of the multiple camera configuration streams with the highest matching degree can be selected as the first camera configuration stream.

[0093] It should be noted that for a camera configuration stream, the more configuration parameters consistent with the first configuration parameter in the camera configuration stream, the higher the matching degree of the camera configuration stream with the first configuration parameter.

[0094] As one of the embodiments, the total configuration parameters consistent with the first configuration parameter in the camera configuration stream can be taken as the matching degree of the camera configuration stream with the first camera configuration parameter. For example, assuming that the first configuration parameter is

format A, color space A, resolution A

format A, color space B, resolution C

format A, color space A, resolution B

format A, color space A, resolution A

[0095] As another embodiment, a corresponding weight can be set for each configuration parameter, and the matching degree of the camera configuration stream with the first configuration parameter is calculated based on the weighted sum. For example, assuming that the first configuration parameter is

format A, color space A, resolution A

format A, color space B, resolution C

format A, color space A, resolution B

format A, color space A, resolution A

[0096] It should be noted that the above is only two possible implementation manners of the matching degree calculation between the camera configuration stream and the first configuration parameter provided by the embodiment, and the embodiment does not specifically limit the specific implementation manner of the matching degree calculation between the camera configuration stream and the first configuration parameter.

[0097] In the embodiment, the camera configuration stream with the highest matching degree with the first configuration parameter is selected as the first camera configuration stream. It can be understood that the process usually encapsulates the data format expected to be obtained in the form of a configuration parameter in a camera access request, so that the data format of the image stream obtained based on the first camera configuration stream is closest to the data format expected by the process accessing the target camera, that is, the matching degree between the obtained data format and the expected data format is maximized, so as to reduce the amount of subsequent data format conversion.

[0098] In a possible implementation manner, the step of obtaining the camera data of the target camera from the image stream copy based on the first camera configuration stream comprises:

[0099] In step F10, if the configuration parameters included in the first camera configuration stream are consistent with the first configuration parameters, at least one frame of image is obtained from the shared image stream copy based on the first camera configuration stream as the camera data obtained from the target camera.

[0100] The configuration parameters included in the first camera configuration stream are consistent with the first configuration parameters, that is, the parameter values of the two are consistent. For example, assuming that a configuration parameter includes image format, color space and resolution, the configuration parameters included in the first camera configuration stream are consistent with the first configuration parameters, that is, the image format in the first camera configuration stream is consistent with the image format in the first configuration parameters, the color space in the first camera configuration stream is consistent with the color space in the first configuration parameters, and the resolution in the first camera configuration stream is consistent with the resolution in the first configuration parameters. At this time, since the configuration parameters are completely consistent, the image stream obtained based on the first camera configuration stream is an image stream of the expected data format, and data format conversion is not required.

[0101] In step F20, if there is a configuration parameter in the configuration parameters included in the first camera configuration stream that is inconsistent with the first configuration parameters, at least one frame of image is obtained from the shared image stream copy based on the first camera configuration stream, image conversion is performed on all the obtained images, and the converted image is taken as the camera data obtained from the target camera, wherein the image parameter of the converted image is the first configuration parameter.

[0102] The number of frames of images and which frames of images are acquired based on the first camera configuration stream from the shared image stream copy can be acquired based on the needs of the process itself, and the embodiment does not make a specific limitation.

[0103] The acquired all images are subjected to image conversion. Specifically, the conversion can be performed based on inconsistent configuration parameters. For example, it is assumed that the configuration parameters contained in the first camera configuration stream are

image format A, color space A, resolution B

image format A, color space A, resolution A

[0104] In the embodiment, when the configuration parameters in the first camera configuration stream are inconsistent with the first configuration parameters, the images are subjected to image conversion to convert the image format, color space, resolution and other image parameters of the images to the first configuration parameters, so that the data format of the acquired camera data conforms to the expectation of the process.

[0105] Based on the first embodiment, the second embodiment and / or the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as the above-mentioned first embodiment, the second embodiment and the third embodiment can be referred to the above introduction, and will not be described hereinafter. On this basis, after the step of copying the image stream to obtain the image stream copy, the method further comprises:

[0106] Step G10, uploading the image stream copy to a preset server, and returning to execute the step of receiving the camera access request;

[0107] The image stream copy is uploaded to the preset server. Specifically, the image stream copy can be copied to the preset server, or the original image stream can be copied to the preset server, and the embodiment does not make a specific limitation.

[0108] After the step of taking the camera requested to be accessed as the target camera, the method further comprises:

[0109] Step G20, determining the application requesting to access the target camera according to the camera access request;

[0110] Step G30, if the application requesting to access the target camera is a non-native application, acquiring the camera data of the target camera from the image stream copy uploaded to the preset server.

[0111] The native application refers to an application program capable of directly running on a current operating system and specially developed for a certain operating system such as iOS, Android and the like, and the native application is constructed using a specific programming language for a specific device platform. The non-native application specifically refers to other applications other than the native application, such as a user-defined application, a web (World Wide Web) application and the like. In the embodiment, the non-native application specifically refers to an application program incapable of directly running on a current operating system of the AR glasses.

[0112] In the embodiment, the non-native application acquires the camera data from the preset server, which on one hand enables the non-native application to also share the use of the camera, and on the other hand directly from the preset server can improve the acquisition speed of the camera data, thereby realizing the low-latency acquisition of the camera data.

[0113] Exemplarily, in order to assist in understanding the technical concept or technical principle of the camera data acquisition after the combination of the above-mentioned embodiment one, embodiment two and embodiment three, please refer to FIG. 2, which provides a brief flowchart of a camera data acquisition method, and the details are as follows:

[0114] 1. Process A and process B access the same camera through the Camera API (Camera Application Programming Interface) of the Android camera system, wherein the process A and the process B are the processes of the native Camera application of the Android platform.

[0115] 2. In the Camera server (system camera service) module, if the camera that has been used is opened, since the camera identifier of the same camera of the camera system can only be given to one process, a virtual camera identifier is generated, and the virtual camera identifier is transmitted to the HAL layer, so that the camera system can receive the access request of the process for subsequent processing, instead of kicking off the process using the camera or directly rejecting the access request of the process. Wherein, the Camera server module is the native Camera server module of the Android camera system, including the Camera service (camera service), CameraDeviceClient (camera device client), Camera3Device (camera system device), CameraProviderManger (camera provider manager) and the like.

[0116] 3. In the Camera provider module, when opening a camera that is not in use, configure all camera streams (including all image formats, color spaces, and all resolutions, generate all pipelines), and send all camera streams to the Camera driver (camera driver), cache and copy the obtained image stream. When opening the same camera for the second time, mark the camera requested to be opened, obtain a copy of the copied image stream, and realize data sharing.

[0117] 4. Distribute the image frames in the image stream to the server (server stream shown in FIG. 2), and when a three-party algorithm in a non-native application needs to use a camera, obtain camera data from the server, and transmit the image stream from the server to the client of the non-native application (client stream shown in FIG. 2).

[0118] It should be noted that the above specific embodiments are only used to understand the present application, and do not constitute a limitation on the camera data acquisition process of the present application. Based on this technical concept, more forms of simple transformation are within the protection scope of the present application.

[0119] In addition, the embodiment of the present application also provides a camera data acquisition device, referring to FIG. 4, the camera data acquisition device comprises:

[0120] The receiving module 10 is configured to receive a camera access request and take a camera requested to be accessed as a target camera.

[0121] The sharing module 20 is configured to, if the target camera is not in use when the camera access request is received, run a system camera support component, obtain an image stream photographed by the target camera through the system camera support component, obtain camera data of the target camera from the image stream, and copy the image stream to obtain an image stream copy.

[0122] The obtaining module 30 is configured to, if the target camera is in use when the camera access request is received, run the system camera support component, and obtain the camera data of the target camera from the image stream copy through the system camera support component.

[0123] In addition, the embodiment of the present application also provides an augmented reality device, which comprises a memory, a processor, and a camera data acquisition program stored in the memory and executable on the processor. The camera data acquisition program is executed by the processor to implement the steps of the camera data acquisition method described above.

[0124] Further, as shown in FIG. 5, the augmented reality device can also include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the augmented reality device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the augmented reality device to communicate wirelessly or by wire with other devices to exchange data. Although the augmented reality device having various systems is shown in the figure, it should be understood that all of the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0125] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0126] The augmented reality device provided by the embodiments of the present disclosure adopts the camera data acquisition method in the above-mentioned embodiments, and can solve the technical problem of camera data acquisition. Compared with the prior art, the augmented reality device provided by the present disclosure has the same beneficial effects as the camera data acquisition method provided by the above-mentioned embodiments, and other technical features in the augmented reality device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0127] It should be understood that various aspects of the disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any appropriate manner in any one or more embodiments or examples.

[0128] The above description is merely illustrative of the application and not restrictive.

[0129] In addition, to achieve the above object, the embodiment of the present application further provides a readable storage medium having computer readable program instructions (i.e. computer programs) stored thereon, the computer readable program instructions being used to execute the camera data acquisition method in the above embodiment.

[0130] The computer readable storage medium provided by the embodiment of the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination of the above. In the embodiment, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any appropriate medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any appropriate combination of the above.

[0131] The above computer readable storage medium can be contained in an augmented reality device; or can exist separately and not be assembled into an augmented reality device.

[0132] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the augmented reality device, cause the augmented reality device to: receive a camera access request, take the camera requested to access as a target camera; if the target camera is not in use when the camera access request is received, run a system camera support component, acquire an image stream captured by the target camera through the system camera support component, acquire camera data of the target camera from the image stream, and copy the image stream to obtain an image stream copy; if the target camera is in use when the camera access request is received, run the system camera support component, and acquire the camera data of the target camera from the image stream copy through the system camera support component.

[0133] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0134] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0135] The modules described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0136] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the above-mentioned camera data acquisition method, and can solve the technical problem of camera data acquisition. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the camera data acquisition method provided by the above-mentioned embodiments, which will not be repeated here.

[0137] In addition, the embodiments of the present application also provide a computer program product, which includes a camera data acquisition program, and the camera data acquisition program is executed by a processor to implement the steps of the camera data acquisition method as described above.

[0138] The computer program product of the present application has substantially the same implementation as the above-mentioned camera data acquisition method, which will not be repeated here.

[0139] It should be noted that in this paper, the term "including", "containing" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or system including the element.

[0140] The serial numbers of the above-mentioned embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better implementation. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software sensor, which is stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc) and includes a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) execute the method described in each embodiment of the present application.

[0142] The above merely preferred embodiments of the present application and are not intended to limit the patent scope of the present application, any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for acquiring camera data, characterized in that, The camera data acquisition method, applied to augmented reality devices, includes the following steps: Receive camera access requests and designate the requested camera as the target camera; If the target camera is not in use when the camera access request is received, the system camera support component is run, the image stream captured by the target camera is obtained through the system camera support component, the camera data of the target camera is obtained from the image stream, and the image stream is copied to obtain an image stream copy; If the target camera is already in use when the camera access request is received, the system camera support component is run, and the camera data of the target camera is obtained from the image stream copy through the system camera support component.

2. The method as described in claim 1, characterized in that, The step of acquiring the image stream captured by the target camera includes: All configuration parameter groups are obtained based on the camera access request, wherein each configuration parameter group includes one or more configuration parameters; Configure a camera configuration stream according to each of the configuration parameter groups, wherein each configuration parameter group corresponds to one camera configuration stream; Select one camera configuration stream from all the camera configuration streams, and use the selected camera configuration stream as the target camera configuration stream; The image stream captured by the target camera is obtained based on the target camera configuration stream.

3. The method as described in claim 2, characterized in that, The step of selecting a camera configuration stream from all the camera configuration streams and using the selected camera configuration stream as the target camera configuration stream includes: The camera configuration stream obtained from all the camera configuration streams based on the configuration parameters encapsulated in the camera access request is taken as the target camera configuration stream.

4. The method as described in claim 1, characterized in that, The step of obtaining the camera data of the target camera from the image stream copy includes: Obtain all pre-configured camera configuration streams, and select a first camera configuration stream from all said camera configuration streams; Based on the first camera configuration stream, the camera data of the target camera is obtained from the image stream copy.

5. The method as described in claim 4, characterized in that, The step of selecting a first camera configuration stream from all the camera configuration streams includes: Obtain the first configuration parameters encapsulated in the camera access request; The camera configuration stream with the highest matching degree with the first configuration parameter among all the camera configuration streams is taken as the first camera configuration stream. The more configuration parameters in the camera configuration stream that are consistent with the first configuration parameter, the higher the matching degree between the camera configuration stream and the first configuration parameter.

6. The method as described in claim 5, characterized in that, The step of obtaining the camera data of the target camera from the image stream copy based on the first camera configuration stream includes: If the configuration parameters contained in the first camera configuration stream are all consistent with the first configuration parameters, then at least one frame of image is obtained from the image stream copy based on the first camera configuration stream as the camera data obtained from the target camera; If the configuration parameters included in the first camera configuration stream are inconsistent with the first configuration parameters, then at least one frame of image is obtained from the image stream copy based on the first camera configuration stream, image conversion is performed on all obtained images, and the converted image is used as the camera data obtained from the target camera, wherein the image parameters of the converted image are the first configuration parameters.

7. The method according to any one of claims 1 to 6, characterized in that, After the step of copying the image stream to obtain an image stream copy, the method further includes: Upload the image stream copy to the preset server and return to execute the step of receiving the camera access request; After the step of designating the requested camera as the target camera, the method further includes: The application requesting access to the target camera is determined based on the camera access request; If the application requesting access to the target camera is a non-native application, then the camera data of the target camera is obtained from the image stream copy uploaded to the preset server.

8. A camera data acquisition device, characterized in that, The camera data acquisition device includes: The receiving module 10 is used to receive camera access requests and to take the requested camera as the target camera. The sharing module 20 is used to run the system camera support component if the target camera is not in use when the camera access request is received, to obtain the image stream captured by the target camera through the system camera support component, to obtain the camera data of the target camera from the image stream, and to copy the image stream to obtain an image stream copy. The acquisition module 30 is used to run the system camera support component if the target camera is already in use when the camera access request is received, and to acquire the camera data of the target camera from the image stream copy through the system camera support component.

9. An augmented reality device, characterized in that, The augmented reality device includes: a memory, a processor, and a camera data acquisition program stored in the memory and executable on the processor, wherein the camera data acquisition program, when executed by the processor, implements the steps of the camera data acquisition method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, and the computer-readable storage medium stores a program that implements the camera data acquisition method. The program that implements the camera data acquisition method is executed by a processor to implement the steps of the camera data acquisition method as described in any one of claims 1 to 7.

Images