Data processing method and device for multi-camera XR equipment, equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device for multi-camera XR equipment, equipment and a storage medium. The method comprises the following steps: acquiring a hardware connection relationship between a plurality of camera sensors and an ISP (Internet Service Provider); dividing the plurality of camera sensors into a plurality of camera groups according to the hardware connection relation and business requirements, the camera sensors in each camera group having the same function, and the camera sensors in the camera groups being connected to the same aggregator or directly connected to the ISP; service processing is carried out according to the image data, obtained through processing of the ISP, of the multiple camera sensors, and the ISP is used for receiving sensor data sent by the camera sensors according to the connected camera sets and the parameters of the camera sensors in the camera sets and processing the sensor data to obtain the image data. According to the method, the cameras can be dynamically grouped according to the hardware connection relationship and the service requirements, and the cameras which have the same function and are connected to the same aggregator can be divided into one camera group, so that the ISP can perform efficient receiving and processing conveniently, and resource waste is avoided.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a data processing method, apparatus, device, and storage medium for a multi-camera XR device. Background Technology

[0002] Extended Reality (XR) refers to the use of computers to combine the real and virtual worlds, creating a virtual environment that allows for human-computer interaction and providing users with a seamless sense of immersion between the virtual and real worlds.

[0003] In existing technologies, XR devices typically include multiple cameras, such as depth cameras, RGB cameras, face-tracking cameras, and eye-tracking cameras. These cameras have different functions and attributes. Some cameras can be directly connected to an Image Signal Processor (ISP) via a physical channel, while others can have their data aggregated via a bridge or a Field Programmable Gate Array (FPGA) before being connected to the ISP via a physical channel. A single bridge or FPGA can connect multiple cameras.

[0004] In existing technologies, each camera is treated as an individual and its data is processed, which is inefficient and wastes resources. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and storage medium for multi-camera XR devices. The method dynamically groups cameras based on the hardware connection relationship between multiple camera sensors and the ISP and the service requirements. Cameras with the same function and connected to the same aggregator can be grouped into a camera group, which facilitates efficient reception and processing by the ISP and avoids resource waste.

[0006] In a first aspect, embodiments of this application provide a data processing method, apparatus, device, and storage medium for a multi-camera XR device, applied to a service XR device, wherein the XR device includes multiple camera sensors, at least one aggregator, and at least one image signal processor (ISP), and the method includes:

[0007] Obtain the hardware connection relationship between the multiple camera sensors and the ISP, wherein each aggregator aggregates at least two camera sensors and connects them to the corresponding ISP;

[0008] Based on the hardware connection relationship and business requirements, the multiple camera sensors are divided into multiple camera groups, and parameters are set for the camera sensors in the camera groups that need to be activated. The camera sensors in each camera group have the same function, and the camera sensors in the camera group are connected to the same aggregator or directly connected to the ISP.

[0009] The ISP processes image data from multiple camera sensors obtained through processing. The ISP receives sensor data sent by the camera sensors based on the parameters of the connected camera group and the camera sensors within the camera group, and processes the sensor data to obtain image data.

[0010] In some exemplary embodiments, the method further includes:

[0011] When it is determined that the first camera sensor is faulty, the first camera sensor is removed from the target camera group to which the first camera sensor belongs.

[0012] The process of performing service processing based on image data from multiple camera sensors obtained by the ISP includes:

[0013] When a backup camera sensor for the first camera sensor exists in the XR device, the image data of the backup camera sensor is acquired, and the image data of the backup camera sensor and the image data of the normally functioning camera sensor in the target camera group are processed to obtain the first processing result of the first service.

[0014] When there is no backup camera sensor for the first camera sensor in the XR device, a compensation algorithm is used to process the image data of the normally functioning camera sensor in the target camera group to obtain the second processing result of the first service.

[0015] In some exemplary embodiments, the backup camera sensor belongs to the target camera group, or the backup camera sensor does not belong to the target camera group.

[0016] In some exemplary embodiments, the backup camera sensor is a camera sensor whose field of view overlaps with that of the first camera sensor, or the backup camera sensor is a camera sensor that has the same function as the first camera sensor.

[0017] In some exemplary embodiments, the compensation algorithm is an image fusion algorithm or a deep learning model.

[0018] In some exemplary embodiments, the method further includes: outputting visual and / or auditory cue information, wherein the visual and / or auditory cue information is used to indicate a fault in the first camera sensor and to perform fault recovery processing.

[0019] In some exemplary embodiments, the method further includes: acquiring the temperature of the plurality of camera sensors; and when the temperature of at least one camera sensor is greater than a preset temperature threshold, reducing the frame rate of some camera sensors or turning off some camera sensors.

[0020] In some exemplary embodiments, reducing the frame rate of some camera sensors or turning off some camera sensors includes: reducing the frame rate of camera sensors whose temperature is greater than the temperature threshold; or, reducing the frame rate of low-priority camera sensors or turning off low-priority camera sensors according to the priority of the plurality of camera sensors; or, reducing the frame rate of non-critical camera sensors; or, reducing the frame rate of backup camera sensors of the primary camera sensor, or turning off backup camera sensors of the primary camera sensor.

[0021] In some exemplary embodiments, the method further includes displaying a power-saving prompt message, the prompt message indicating that the frame rate of the camera sensor has been reduced or the camera sensor has been turned off.

[0022] In some exemplary embodiments, the method further includes: when it is detected that the XR device has activated an energy-saving mode, alternately turning off the camera sensors with a primary / backup relationship.

[0023] In some exemplary embodiments, when the first camera group includes multiple camera sensors, the multiple camera sensors in the first camera group share the shared buffer corresponding to the first camera group in a time-division multiplexing manner.

[0024] In some exemplary embodiments, the method further includes: cyclically outputting camera synchronization signals to a plurality of camera sensors in the first camera group according to a fixed timing sequence; each camera sensor in the first camera group outputting sensor data after receiving the camera synchronization signal and storing the sensor data in the shared buffer; the ISP reading the sensor data sent by the camera sensors from the shared buffer corresponding to the first camera group according to the parameters of the connected camera group and the camera sensors in the camera group, and processing the read sensor data to obtain image data.

[0025] In some exemplary embodiments, the sensor data output by each camera sensor in the first camera group includes information about the camera sensor. The step of processing the read sensor data to obtain image data includes: determining the camera sensor corresponding to the sensor data based on the camera sensor information included in the read sensor data; and processing the sensor data based on the camera sensor corresponding to the sensor data.

[0026] In some exemplary embodiments, the shared buffer is a circular buffer, and the size of the shared buffer is determined according to the processing speed of the ISP. The size of the shared buffer at least ensures that the sensor data output later will not overwrite the sensor data that has been output but not processed by the ISP.

[0027] Secondly, embodiments of this application provide a data processing apparatus for a multi-camera XR device, the apparatus comprising:

[0028] The acquisition module is used to acquire the hardware connection relationship between multiple camera sensors of the XR device and the ISP. The XR device includes multiple camera sensors, at least one aggregator and at least one image signal processor (ISP). Each aggregator aggregates at least two camera sensors and connects them to the corresponding ISP.

[0029] The grouping module is used to divide the multiple camera sensors into multiple camera groups according to the hardware connection relationship and business requirements, and set parameters for the camera sensors in the camera groups that need to be activated. The camera sensors in each camera group have the same function, and the camera sensors in the camera group are connected to the same aggregator or directly connected to the ISP.

[0030] The service module is used to perform service processing based on the image data of the multiple camera sensors obtained by the ISP. The ISP is used to receive sensor data sent by the camera sensors according to the parameters of the connected camera group and the camera sensors in the camera group, and process the sensor data to obtain image data.

[0031] Thirdly, embodiments of this application provide an XR device, the XR device including: a camera system, a processor, and a memory, wherein the camera system and the memory are connected to the processor;

[0032] The camera system includes multiple camera sensors, at least one aggregator, and at least one image signal processor (ISP). Each aggregator connects at least two camera sensors to the corresponding ISP. The ISP is used to receive sensor data sent by the camera sensors according to the parameters of the connected camera group and the camera sensors within the camera group, and to process the sensor data to obtain image data.

[0033] The memory is used to store computer programs, and the processor is used to call and run the computer programs stored in the memory to perform the method described in the first aspect of this application.

[0034] Fourthly, embodiments of this application provide a computer-readable storage medium for storing a computer program that causes a computer to perform the method described in the first aspect above.

[0035] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0036] The data processing method, apparatus, device, and storage medium for multi-camera XR devices provided in this application embodiment acquire the hardware connection relationship between multiple camera sensors and an ISP. Each aggregator aggregates at least two camera sensors and connects them to the corresponding ISP. Based on the hardware connection relationship and service requirements, the multiple camera sensors are divided into multiple camera groups. The camera sensors within each camera group have identical functions and are connected to the same aggregator or directly to the ISP. Service processing is performed on the image data from the multiple camera sensors obtained by the ISP. The ISP receives sensor data sent by the camera sensors based on the connected camera groups and the parameters of the camera sensors within each group, and processes the sensor data to obtain image data. This method can dynamically group cameras according to hardware connection relationships and service requirements, enabling cameras with identical functions and connected to the same aggregator to be grouped into a single camera group. This facilitates efficient reception and processing by the ISP and avoids resource waste. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A schematic diagram showing the mounting positions of multiple cameras in an XR device;

[0039] Figure 2 This is a schematic diagram of the hardware connection of a camera system for an XR device to which this application embodiment applies;

[0040] Figure 3 This is a schematic diagram of a functional module of an XR device to which the embodiments of this application apply;

[0041] Figure 4 This is a flowchart of the data processing method for a multi-camera XR device provided in Embodiment 1 of this application;

[0042] Figure 5 This is a flowchart of a data processing method for a multi-camera XR device provided in Embodiment 2 of this application;

[0043] Figure 6 A schematic diagram of FoV overlap for a camera sensor;

[0044] Figure 7 This is a flowchart of a data processing method for a multi-camera XR device provided in Embodiment 3 of this application;

[0045] Figure 8 This is a flowchart of a data processing method for a multi-camera XR device provided in Embodiment 4 of this application;

[0046] Figure 9 A schematic diagram showing the use of independent buffers for multiple camera sensors within the first camera group;

[0047] Figure 10 A schematic diagram illustrating the use of a shared buffer for multiple camera sensors within the first camera group;

[0048] Figure 11 This is a schematic diagram of the data processing apparatus for a multi-camera XR device provided in Embodiment 5 of this application;

[0049] Figure 12 This is a schematic diagram of the structure of an XR device provided in Embodiment Six of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0052] This application provides a data processing method for a multi-camera XR device, applied in Extended Reality (XR). XR refers to all real and virtual combined environments and human-computer interactions generated by computer technology and wearable devices. XR includes various forms such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). To facilitate understanding of the embodiments of this application, before describing the various embodiments, some concepts in the XR scene involved in all embodiments of this application will be appropriately explained as follows:

[0053] 1) VR, a technology for creating and experiencing virtual worlds, defines and generates a virtual environment. It is a multi-source information (virtual reality mentioned in this article includes at least visual perception, and may also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.) that realizes the fusion of virtual environment, interactive three-dimensional dynamic visual scenes and simulation of physical behavior, immersing users in the simulated virtual reality environment, and enabling applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

[0054] 2) Virtual reality devices (VR devices) are terminals that realize virtual reality effects. They can usually be provided in the form of glasses, head-mounted displays (HMDs), or contact lenses to realize visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to these, and they can be further miniaturized or enlarged according to actual needs.

[0055] Optionally, the virtual reality devices described in the embodiments of this application may include, but are not limited to, the following types:

[0056] 2.1) PC-based virtual reality (PCVR) devices utilize a PC for calculations and data output related to virtual reality functions. External PC-based virtual reality devices use the data output from the PC to achieve virtual reality effects.

[0057] 2.2) Mobile virtual reality devices support setting up mobile terminals (such as smartphones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations for virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through the mobile terminal's APP.

[0058] 2.3) All-in-one virtual reality devices have processors for performing virtual functions, thus having independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal, and have a high degree of freedom of use.

[0059] 3) Virtual field of view: The area in the virtual environment that a user can perceive through the lens in a virtual reality device. The perceived area is represented by the field of view (FOV).

[0060] 4) AR (Augmented Reality) is a technology that calculates the camera's pose parameters in the real world (or 3D world, real world) in real time during image acquisition, and adds virtual elements to the captured images based on these parameters. Virtual elements include, but are not limited to, images, videos, and 3D models. The goal of AR technology is to overlay the virtual world onto the real world on a screen for interactive viewing.

[0061] 5) MR (Mixed Reality) is a simulated scene that integrates computer-created sensory input (e.g., virtual objects) with sensory input or its representation from a physical scene. In some MR scenes, the computer-created sensory input can adapt to changes in sensory input from the physical scene. Additionally, some electronic systems used to present MR scenes can monitor orientation and / or position relative to the physical scene to enable virtual objects to interact with real objects (i.e., physical elements from the physical scene or their representations). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0062] 6) A virtual scene is a virtual scene displayed (or provided) by an application while running on an electronic device. This virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional virtual scene, or a purely fictional virtual scene. A virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this application embodiment does not limit the dimension of the virtual scene. For example, a virtual scene may include the sky, land, ocean, etc., and the land may include environmental elements such as deserts and cities. Users can control virtual objects to move within this virtual scene.

[0063] 7) Virtual objects are objects that interact in a virtual scene. They are controlled by the user or a robot program (e.g., an AI-based robot program) and can remain still, move, and perform various behaviors in the virtual scene, such as various characters in a game.

[0064] The method of this application embodiment is applied to a multi-camera XR device, which includes multiple cameras, each with different functions and attributes.

[0065] To clearly illustrate the technical solution of this application, the application scenarios of this application are described below. It should be understood that the technical solution of this application can be applied to the following scenarios, but is not limited to them:

[0066] Figure 1 A schematic diagram showing the mounting positions of multiple cameras in an XR device, such as... Figure 1 As shown, the XR device includes: two red-green-blue (RGB) cameras (i.e., cameras 11 and 12), one depth camera (i.e., camera 13), and four 6-DOF (DoF) cameras (i.e., cameras 14, 15, 16, and 17).

[0067] RGB cameras are used to capture RGB images, which can then be displayed on the XR device's screen. RGB images can be used in the Video See-Through (VST) service of XR devices. The VST service uses the RGB camera of the XR device (also known as a head-mounted display) to capture a real-time view of the surrounding environment. The image captured by the RGB camera is processed using an anti-distortion algorithm, and then output to the XR device's screen to simulate a see-through view of the outside world. This allows users to see the external environment through the XR device's display. VST allows users to interact with the real world without removing the head-mounted display, and it is widely used in XR devices. Therefore, in some XR devices, the RGB camera is also referred to as a VST camera.

[0068] Depth cameras are used to acquire depth images of images. These depth images contain depth information for each pixel in the image, also known as depth values. Based on the depth values ​​of each camera point and the pixel coordinates of each pixel in the 2D image, the three-dimensional spatial coordinates of each pixel can be obtained. These three-dimensional coordinates can then be used to reconstruct real-world scenes, enabling applications such as scene modeling. Of course, depth cameras have other uses as well, which will not be listed here. Depth cameras include, but are not limited to, one or more of the following: structured-light devices, binocular cameras, monocular cameras, and Time-of-Flight (TOF) sensors.

[0069] A 6DoF camera is used to collect tracking data, which is then used to provide 6DoF positioning services for XR devices. In addition to detecting changes in the field of view caused by head rotation (three degrees of freedom of rotation), 6DoF XR devices can also detect changes in vertical, horizontal, and longitudinal displacements (three displacement-related degrees of freedom) caused by body movement.

[0070] The diagram shows four 6DoF cameras. Optionally, in some XR devices, cameras 14 and 15, located in the upper left and upper right corners, are used to acquire environmental images and perform 6DoF localization through environmental perception. Therefore, cameras 14 and 15 are also referred to as environment understanding cameras. Cameras 16 and 17, located in the lower left and lower right corners, are used to acquire images of the tracker and / or the hand controller. The tracker is localized using 6DoF based on the tracker's image, and the hand controller is localized using 6DoF based on the hand controller's image.

[0071] The tracking device can be worn on different parts of the body, such as the limbs, torso, shoulders, and waist, to track the body's posture. The hand controller can be a handle for tracking the upper limbs. Therefore, cameras 16 and 17 are also referred to as posture tracking cameras.

[0072] Optionally, the XR device also includes eye and face tracking cameras for capturing images of the eyes and face and tracking the face and eyes. Optionally, the XR device also includes a dedicated face tracking camera for capturing images of the face. The eye and face tracking cameras can be oriented towards the user's face (i.e., inward), and the face tracking camera can be oriented in the opposite direction to the user's face (i.e., outward).

[0073] Figure 2 This is a hardware connection diagram of a camera system for an XR device to which this application is applicable, as shown in the embodiments. Figure 2 As shown, the camera system of an XR device includes six cameras. Each camera may include a camera sensor, an image signal processor (ISP), and connection hardware connecting the camera sensor and the ISP. The six cameras correspond to six camera sensors: the camera sensor (AF) acquires light signals through a photosensitive element and converts the light signals into electrical signals. The electrical signals output by the camera sensor will be referred to as sensor data.

[0074] The connection hardware includes the Physical Layer Camera Serial Interface (CSIphy) and the Aggregator. CSI stands for Mobile Industry Processor Interface (MIPI). The CSIphy interface is widely used in the camera systems of mobile phones and XR devices for transmitting image data. Camera sensors can connect directly to the ISP via the CSIphy interface or via an aggregator.

[0075] An aggregator, also known as a combiner, receives signals from multiple camera sensors, stitches them together, and transmits the stitched image signal to the ISP via the CSIphy interface. Adding aggregators expands the number of camera sensors that can be connected to the CSIphy interface; without an aggregator, each CSIphy interface can only connect to one camera sensor. Aggregators can be bridged or use a Field Programmable Gate Array (FPGA).

[0076] The image sensor (ISP) is used to process the sensor data output from the front-end camera sensor to obtain a visual image, that is, an image that can be displayed on the device. The ISP largely determines the image quality of the camera.

[0077] Figure 2 The ISP includes 5 hardware nodes (IFE lite). CSIphy and IFE lite are the hardware on the application processor (AP) side that receive and process image MIPI data. IFE lite is used to perform pixel processing on the received sensor data.

[0078] Figure 2 Camera sensor F is connected to IFE Lite0 via CSIphy0. Camera sensors A, B, and E are connected to IFE Lite1, IFE Lite2, and IFE Lite3 via aggregator 1 and CSIphy1, respectively. Camera sensors C and D are connected to IFE Lite4 via aggregator 2 and CSIphy2. It is important to note that a single CSIphy can connect to one or more IFE Lites, depending on the sensor data being processed.

[0079] When an aggregator is introduced, processing data from different camera sensors connected to that aggregator becomes more complex. In this embodiment, multiple cameras on the XR device are grouped into multiple camera groups. The camera sensors within each group have identical functions, and these sensors are connected to the same aggregator or directly to the ISP. The ISP can efficiently receive and process the data from the camera sensors within each camera group. For example, Figure 2 In this design, camera sensors A and B are assigned to camera group 1, and camera sensors C and D are assigned to camera group 2.

[0080] In this embodiment, the camera sensors within each camera group have identical functions. These sensors are connected to the same aggregator or directly to the ISP, or each camera sensor within a camera group shares the same aggregator. This shared connectivity and connection facilitates data processing within the camera group. For example, multiple cameras within a camera group may be used for 6DoF positioning and connected to the same aggregator; or, a VST service may be associated with two camera groups, with multiple RGB cameras in camera group 1 used for VST services and connected to the same aggregator, and multiple depth cameras in camera group 2 used for VST services and connected to the same aggregator. Camera group 1 and camera group 2 can be connected to the same aggregator or to two different aggregators; or, multiple cameras within a camera group may be used for eye tracking and connected to the same aggregator.

[0081] In existing technologies, multiple camera sensors connected to an aggregator typically have different functions. When cameras with the same function are grouped together as a camera group, these cameras with identical functional attributes do not share the same aggregator or CSIphy. Therefore, multiple camera groups need to be activated upon startup, leading to resource waste. However, the method in this application embodiment involves camera sensors within some camera groups that have the same function and are connected to the same aggregator. In some scenarios, depending on business needs, only some functions of the camera system need to be activated. In this case, only the camera groups corresponding to those specific functions need to be activated, instead of activating all camera groups, thus saving resources.

[0082] Figure 3 This is a schematic diagram of a functional module of an XR device to which this application is applicable, as shown in the embodiments. Figure 3 As shown, the XR device includes the following modules: three camera groups, three ISPs, a statistics module, and a service module. The number of camera groups and ISPs can vary; this is just an example.

[0083] The business module groups the multiple camera sensors of the XR device and assigns these multiple camera sensors to the appropriate group.

[0084] The system divides the system into multiple camera groups and sets parameters for each camera sensor. The service module also provides the ISP with the grouping information of the multiple camera sensors and the parameter configuration of each camera sensor. The ISP receives the sensor data based on this grouping information and the parameters of each camera sensor, and processes the transmitter signal.

[0085] Each camera group outputs sensor data to the ISP according to the instructions of the service module. The ISP then sends the sensor data to the statistics module and the service module. The statistics module provides auxiliary information to the service module based on the functional overlap and field of view (FoV) overlap between the cameras, which is used by the service module for business processing. For example, the statistics module can infer the brightness of a faulty camera based on the FoV overlap between the cameras and provide this brightness information to the service module. The service module then performs business processing based on the brightness relationship of the faulty camera provided by the statistics module and other data.

[0086] Based on the sensor data sent by the ISP and the auxiliary information provided by the statistics module, the business module decides how to use the sensor data in the next step.

[0087] After introducing the application scenarios of the embodiments of this application, the data processing method of a multi-camera XR device provided by the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0088] Figure 4 This is a flowchart of a data processing method for a multi-camera XR device provided in Embodiment 1 of this application. The execution subject of this embodiment is an XR device or a head-mounted device. Figure 1 and Figure 3 The method of the embodiments of this application will be described. Figure 4 As shown, the method in this embodiment includes the following steps.

[0089] S101. Obtain the hardware connection relationship between multiple camera sensors and the ISP, wherein each aggregator aggregates at least two camera sensors and connects them to the corresponding ISP.

[0090] The XR device includes multiple camera sensors, at least one aggregator, and at least one ISP. The multiple camera sensors are connected to the ISP via the aggregator and CSI interface.

[0091] After the XR device is powered on, the service module can obtain the hardware connection relationships between multiple camera sensors and the ISP from the camera's driver module. Each aggregator aggregates at least two camera sensors and connects them to the corresponding ISP. (Refer to...) Figure 2 and Figure 3Each aggregator connects to at least two camera sensors, and each aggregator may connect to one or more ISPs. In XR devices, there may also be camera sensors directly connected to the ISP. Sensors directly connected to the ISP refer to different aggregators where the camera sensors connect to the ISP via a CSI interface, for example... Figure 2 The camera sensor F in the middle.

[0092] In this embodiment, camera sensors with the same characteristics are connected to the same aggregator as much as possible. This ensures that the camera sensors in the grouped camera system have the same function and are connected to the same aggregator, making the grouping of the camera system more reasonable and avoiding cameras with the same function belonging to different camera groups.

[0093] S102. Based on the hardware connection relationship and business requirements, divide multiple camera sensors into multiple camera groups, and set parameters for the camera sensors in the camera groups that need to be activated. The camera sensors in each camera group have the same function, and the camera sensors in the camera group are connected to the same aggregator or directly connected to the ISP.

[0094] The hardware connection between the camera sensors and the ISP is fixed, but the service module can dynamically group the camera sensors. Based on this hardware connection and the current scenario's service requirements, the service module divides multiple camera sensors into multiple camera groups. These service requirements are dynamic and can change with the scenario; for example, the service requirements in a VST scenario differ from those in a game scenario. When the service scenario changes, the multiple camera sensors can be regrouped according to the service requirements and the hardware connection.

[0095] This business requirement can describe the services needed for the business. For example, the business requirement indicates that the required services include 6DoF tracking and VST services. Optionally, the business requirement can also include minimum service requirements. It can be understood that a business may be associated with one or more camera groups. For example, a 6DoF tracking service is associated with only one 6DoF camera group; a VST service requires both RGB and depth cameras, so it is associated with one RGB camera group and one depth camera group.

[0096] When the service module groups cameras based on hardware connectivity, it assigns multiple camera sensors connected to the same aggregator to one or more camera groups, but does not group camera sensors from multiple aggregators into a single camera group. The camera sensors within each group have identical functions, and they are connected to the same aggregator or directly to the ISP. In this embodiment, camera sensors directly connected to the ISP can be treated as a separate camera group. Figure 2 The camera sensor F in the middle.

[0097] In this embodiment, a camera group may also include only one camera, for example... Figure 3 Camera sensor 5 in the image belongs to camera group 3, or Figure 2 The camera sensor E and camera sensor F in the system each form a separate camera group.

[0098] Optionally, in other embodiments of this application, Figure 2 Camera sensors E and F can also be treated as independent cameras, without further dividing them into camera groups.

[0099] In one implementation, the business module can decide which camera groups to activate based on business requirements. The module can activate all camera groups or only some, for example, if the business requirement indicates that eye-tracking service is not needed, then the eye-tracking cameras can be disabled. In another implementation, the business module can activate all camera groups by default. Yet another implementation allows the business module to decide which cameras within certain camera groups to activate based on business requirements. For example, camera group 1 may include four 6DoF cameras for 6DoF tracking service, but if the accuracy requirements of the 6DoF tracking service are not high, then only two 6DoF cameras can be activated.

[0100] Optionally, the business module can set the parameters of the camera sensors in the camera group according to business needs. The parameters of the camera sensors include, but are not limited to, field of view, resolution, and sensitivity.

[0101] To ensure that the camera sensors within a group have the same functionality and are connected to the same aggregator, the hardware connection of the camera system should be designed to connect camera sensors with the same functionality to the same aggregator as much as possible.

[0102] In one alternative implementation, the camera sensors connected to each aggregator have the same function. In this implementation, all cameras in the camera system are grouped according to function. When there are multiple cameras with a certain function, the camera sensors for that function are connected to the same aggregator. Since the camera sensors connected to each aggregator have the same function, after grouping, the camera sensors connected to each aggregator can be grouped together, thereby ensuring that the camera sensors within a camera group have the same function, and that the camera sensors within a camera group are connected to the same aggregator or directly connected to the ISP.

[0103] In another alternative implementation, some aggregators connect to camera sensors with the same function, while other aggregators can connect to multiple camera sensors with the same function. For example, aggregator 1 connects to two camera sensors with the same function. The number of camera sensors corresponding to each function can be multiple. For example, function 1 corresponds to two camera sensors, function 2 corresponds to two camera sensors, or function 1 corresponds to two camera sensors and function 2 corresponds to one camera sensor.

[0104] S103. Based on the image data from multiple cameras obtained through ISP processing.

[0105] The service module sends the parameters of the camera groups and the camera sensors within the cameras to each ISP. Each ISP receives the sensor data sent by the camera sensors within the connected camera groups based on the parameters of the connected camera groups and the camera sensors within the camera groups, and processes the sensor data to obtain image data.

[0106] Each ISP sends the processed image data to the service module. The service module performs service processing based on the image data from the multiple camera sensors. The services processed by the service module include 6DoF tracking service, VST service, depth acquisition service, eye tracking service, etc. The service module aggregates the image data from each camera sensor into the corresponding service and processes it according to the function of each camera sensor to obtain the processing results of each service.

[0107] In this embodiment, the hardware connection relationships between multiple camera sensors and the ISP are obtained. Each aggregator aggregates at least two camera sensors and connects them to the corresponding ISP. Based on these hardware connection relationships and service requirements, the multiple camera sensors are divided into multiple camera groups. The camera sensors within each camera group have the same function, and they are connected to the same aggregator or directly to the ISP. Service processing is performed on the image data from the multiple camera sensors obtained by the ISP. The ISP receives sensor data sent by the camera sensors based on the connected camera groups and the parameters of the camera sensors within each group, and processes the sensor data to obtain image data. This method can dynamically group cameras according to hardware connection relationships and service requirements, enabling cameras with the same function and connected to the same aggregator to be grouped into a single camera group. This facilitates efficient reception and processing by the ISP and avoids resource waste.

[0108] In XR devices, the camera is a key component for environmental perception and user interaction. A single point of failure, such as camera damage, can lead to system performance degradation and affect user experience. Embodiment 2 of this application provides a data processing method for multi-camera XR devices, which can maintain or quickly restore the user experience through the coordinated work of algorithms and hardware when a critical camera fails. Figure 5 This is a flowchart of the data processing method for a multi-camera XR device provided in Embodiment 2 of this application, as shown below. Figure 5 As shown, the method provided in this embodiment includes the following steps.

[0109] S201. Obtain the hardware connection relationship between multiple camera sensors and the ISP.

[0110] Each aggregator combines at least two camera sensors and connects them to the corresponding ISP.

[0111] S202. Based on the hardware connection relationship and business requirements, divide the multiple camera sensors into multiple camera groups, and set parameters for the camera sensors in the camera groups that need to be activated.

[0112] In this system, the camera sensors within each camera group have the same function. The camera sensors within a camera group are connected to the same aggregator or directly to the ISP. The camera sensors connected to an aggregator are divided into one or more camera groups.

[0113] The specific implementation of steps S201-S202 is described in the relevant description of steps S101-S102 in Embodiment 1, and will not be repeated here.

[0114] S203. When it is determined that the first camera sensor is faulty, the first camera sensor is removed from the target camera group to which the first camera sensor belongs.

[0115] The camera driver module of the XR device can detect whether each camera sensor is faulty. If a camera fails to start multiple times or reports an error, then the camera sensor is determined to be faulty. In this embodiment, the faulty camera sensor is referred to as the first camera sensor, and the camera group to which the first camera sensor belongs is referred to as the target camera group. In this embodiment, the fault of the first camera sensor can be understood as a fault in the camera corresponding to the first camera sensor. It may be a fault in the first camera sensor itself, or it may be a fault in the CSI interface or ISP connected to the first camera sensor.

[0116] When a fault is detected in the first camera sensor, the camera driver module reports the fault to the XR system, the XR system reports the fault to the service module, and when the service module determines that the first camera sensor is faulty, it removes the first camera sensor from the target camera group. By removing the first camera sensor from the target camera group, the ISP no longer receives and processes the data from the first camera sensor, and correspondingly, the service module also no longer receives and processes the data from the first camera sensor.

[0117] S204. Determine whether a backup camera sensor for the first camera sensor exists in the XR device.

[0118] If a backup camera sensor for the first camera sensor exists in the XR device, proceed to step S205; if a backup camera sensor for the first camera sensor does not exist in the XR device, proceed to step S206.

[0119] The backup camera sensor may or may not belong to the target camera group, meaning that the backup camera sensor and the first camera sensor belong to different camera groups.

[0120] Optionally, the backup camera sensor is a camera sensor that has FoV overlap with the first camera sensor, or the backup camera sensor is a camera sensor that has the same function as the first camera sensor.

[0121] Figure 6 A schematic diagram illustrating the FoV overlap of multiple camera sensors, as shown below. Figure 6 As shown in the figure, there are two camera groups: the first camera group and the second camera group. The first camera group includes two RGB cameras (represented by black circles in the figure), and the second camera group includes four 6DoF tracking cameras (represented by white circles in the figure). The area within the solid circle in the figure represents the FoV of the RGB cameras, and the area within the dashed circle represents the FoV of the 6DoF tracking cameras.

[0122] Figure 6 In the diagram, the FoV values ​​of the two RGB cameras overlap, as do the FoV values ​​of the two 6DoF tracking cameras on the left and the two 6DoF tracking cameras on the right. The RGB camera on the left can serve as a backup for the two 6DoF tracking cameras on the left, and vice versa. The two 6DoF tracking cameras on the left and right can be backups for each other. In some scenarios, only the two lower 6DoF tracking cameras can be activated for 6DoF tracking services, with the two upper 6DoF tracking cameras serving as backups.

[0123] S205. Obtain image data from the backup camera sensor, process the image data from the backup camera sensor and the image data from the normally functioning camera sensor in the target camera group, and obtain the first processing result of the first service.

[0124] The first service functions identically to the target camera group. When the camera sensor within the target camera group is a 6DoF tracking camera, the first service is a 6DoF tracking service. When the camera sensor within the target camera group is an RGB camera, the first service can be a VST service. When the camera sensor within the target camera group is a depth camera, the first service can be an image depth acquisition service.

[0125] Before the first camera sensor malfunctions, the processing result of the first service is obtained by the business module based on the image data of the first camera sensor and the image data of the normally functioning camera sensors in the target camera group.

[0126] In the event of a failure of the first camera sensor, and the existence of a backup camera sensor, the service module reads the image data of the backup camera sensor from the ISP corresponding to the backup camera sensor, processes the image data of the backup camera sensor and the image data of the normally functioning camera sensors in the target camera group, and obtains the first processing result of the first service.

[0127] by Figure 6 Taking the illustrated camera as an example, assuming the 6DoF tracking camera in the lower left corner malfunctions, the RGB camera on the left can be used as a replacement. The business module performs 6DoF tracking based on the image data from the RGB camera on the left and the image data from the other three normally functioning 6DoF tracking cameras. Optionally, the overlapping area of ​​the FoV values ​​of the RGB camera on the left and the image data from the RGB camera on the left can be determined. Image data within the overlapping FoV area is then selected from the image data of the RGB camera on the left, and 6DoF tracking is performed based on the selected image data within the overlapping FoV area and the image data from the other three normally functioning 6DoF tracking cameras.

[0128] S206. The image data of the normally functioning camera sensors in the target camera group are processed using a compensation algorithm to obtain the second processing result of the first service.

[0129] When the first camera sensor fails, if the normally functioning camera sensors within the target camera group can meet the minimum service requirements, the image data from the normally functioning camera sensors within the target camera group can be used to process the first service. For example, Figure 6 In the camera setup shown, assuming the 6DoF tracking camera in the lower left corner malfunctions, the image data from the remaining three 6DoF tracking cameras can be used for 6DoF tracking.

[0130] In this embodiment, some adaptive configuration operations need to be performed in the business module. For example, some backup compensation algorithms are provided. When a camera sensor failure occurs, the compensation algorithm can be used to process the image data of the normally functioning camera sensor.

[0131] Optionally, the compensation algorithm can be an image fusion algorithm or a deep learning model. The image fusion algorithm uses advanced image fusion technology to synthesize high-quality images from data from multiple cameras, maintaining image continuity even if some cameras fail.

[0132] Deep learning models can predict missing data to reduce the impact of a single camera failure on the overall experience. For example, under normal circumstances, 6DoF tracking is performed using image data from four 6DoF tracking cameras. A deep learning model can be pre-trained using image data from three 6DoF tracking cameras. This deep learning model can predict the 6DoF tracking result based on the image data from the three 6DoF tracking cameras. When one of the 6DoF tracking cameras fails, the business module uses this deep learning model and image data from the three normally functioning 6DoF tracking cameras to predict the 6DoF tracking result.

[0133] S207. Output visual and / or auditory cue information, which is used to indicate a fault in the first camera sensor and to perform fault recovery processing.

[0134] This step is optional. By adding a feedback mechanism to the user interface, when a camera malfunction is detected and fault recovery is performed, the user is given appropriate visual and auditory feedback, which helps the user to understand the operating status of the XR device in real time, take appropriate measures, and bring a better user experience.

[0135] The visual cues can be text, or text plus images; the auditory cues can be voice, or voice plus alarm signals.

[0136] The fault tolerance mechanism provided in steps S204-S207 can be applied to the following two application scenarios:

[0137] Application Scenario 1: Emergency Switching. In the event of a critical camera failure, the system can seamlessly switch to a backup camera to maintain the continuity of the XR experience.

[0138] Application Scenario 2: Predictive maintenance. By monitoring camera performance, potential camera hardware failures can be predicted, and preparations for switching can be made in advance.

[0139] It is understandable that, compared to before the first camera sensor malfunction, the accuracy or precision of the first and second processing results of the first service may be slightly lower than the normal processing results. However, this will not interrupt the first service. Therefore, the above-mentioned fault tolerance mechanism still has the following advantages:

[0140] (1) High reliability, even in the event of partial camera hardware failure, the system can still operate stably.

[0141] (2) Reduce experience interruptions caused by camera malfunctions and improve user satisfaction.

[0142] (3) Easy to maintain, automatic switching and fault prediction mechanisms reduce the need for manual maintenance.

[0143] The fault-tolerance mechanism provided in this embodiment significantly improves the stability and reliability of the XR system in the face of camera hardware failures, ensuring that users can obtain the same or similar high-quality experience even under less than ideal conditions. At the same time, it reduces system maintenance costs and enhances the device's market competitiveness.

[0144] XR devices generate heat during operation, especially during high-performance image processing and real-time rendering. Sustained increases in device temperature can lead to performance degradation and even affect device safety and user health. Embodiment 3 of this application provides a data processing method for multi-camera XR devices, which intelligently adjusts the camera's frame rate and on / off state to reduce device power consumption and optimize battery life. Figure 7 This is a flowchart of the data processing method for a multi-camera XR device provided in Embodiment 3 of this application, as follows: Figure 7 As shown, the method provided in this embodiment includes the following steps.

[0145] S301. Obtain the hardware connection relationship between multiple camera sensors and the ISP.

[0146] Each aggregator combines at least two camera sensors and connects them to the corresponding ISP.

[0147] S302. Based on the hardware connection relationship and business requirements, divide the multiple camera sensors into multiple camera groups, and set parameters for the camera sensors in the camera groups that need to be activated.

[0148] In this system, the camera sensors within each camera group have the same function. The camera sensors within a camera group are connected to the same aggregator or directly to the ISP. The camera sensors connected to an aggregator are divided into one or more camera groups.

[0149] S303. Perform service processing based on the image data from multiple camera sensors obtained by the ISP. The ISP is used to receive sensor data sent by the camera sensors according to the parameters of the connected camera group and the camera sensors in the camera group, and process the sensor data to obtain image data.

[0150] The specific implementation of steps S301-S303 is described in the relevant description of steps S101-S103 in Embodiment 1, and will not be repeated here.

[0151] S304: Obtain the temperature of multiple camera sensors.

[0152] In this embodiment, a temperature sensor can be installed on each camera sensor. The temperature sensor is used to detect the temperature of the camera sensor and can transmit the temperature of the camera sensor to the business module or the control module.

[0153] S305. When the temperature of at least one camera sensor is greater than a preset temperature threshold, reduce the frame rate of some camera sensors or turn off some camera sensors.

[0154] The business module or control module can set a temperature threshold. When the temperature of one or more camera sensors exceeds this threshold, the frame rate of some camera sensors can be reduced or some camera sensors can be shut down. By reducing the frame rate of some camera sensors, the load on the camera sensors and ISP can be reduced, thereby reducing heat generation.

[0155] Optionally, the business module or control module may reduce the frame rate of some camera sensors or turn off some camera sensors using the following strategies:

[0156] (1) Reduce the frame rate of camera sensors with temperatures above the temperature threshold.

[0157] The functions and installation locations of the camera sensors in an XR device differ (affecting heat dissipation), resulting in varying temperatures for each sensor. At any given time, some sensors may have temperatures exceeding a certain threshold, while others may have temperatures below it. To prevent the temperature of a camera sensor exceeding the threshold from continuing to rise, its frame rate can be reduced.

[0158] (2) Based on the priority of multiple camera sensors, reduce the frame rate of low-priority camera sensors or turn off low-priority camera sensors.

[0159] The priorities of multiple camera sensors are pre-configured. The business module can adjust the priorities of multiple camera sensors, reduce the frame rate of low-priority camera sensors or turn them off. By reducing the frame rate of low-priority camera sensors or turning them off, the device's limitations can be reduced while minimizing the impact on the main business of the XR device.

[0160] (3) Reduce the frame rate of non-critical camera sensors.

[0161] In different scenarios, non-critical camera sensors may differ. By reducing the frame rate of non-critical camera sensors, the impact on the main functions of XR devices can be minimized. For example, in some scenarios, the depth camera is used as a non-critical camera, and its frame rate can be reduced. The normal frame rate of a depth camera may be 30 frames per second, but in some scenarios, the frame rate of the depth camera can be reduced to 1 frame per second.

[0162] (4) Reduce the frame rate of the backup camera sensor of the primary camera sensor, or turn off the backup camera sensor of the primary camera sensor.

[0163] For example, Figure 6 The second camera group shown contains four 6DoF tracking cameras. The minimum system requirement is two 6DoF tracking cameras. For example, the two upper 6DoF tracking cameras can be used as the primary cameras for 6DoF tracking, and the two lower 6DoF tracking cameras can be used as backup cameras. When the temperature of a camera sensor is higher than the temperature threshold, the service module will actively reduce the frame rate of the two backup camera sensors or turn off the two backup camera sensors.

[0164] or, Figure 6 The first camera group shown contains two RGB cameras. The minimum system requirement is one 6DoF tracking camera. For example, the RGB camera on the right can be used as the primary camera for the VST service, and the RGB camera on the left can be used as a backup camera. When the temperature of a camera sensor is higher than the temperature threshold, the service module will actively turn off the backup RGB camera or reduce the frame rate of the backup RGB camera.

[0165] S306. Display power-saving prompt information, which is used to indicate that the frame rate of the camera sensor has been reduced or the camera sensor has been turned off.

[0166] This step is optional. When some camera sensors are turned off or the frame rate of some camera sensors is reduced, feedback energy-saving prompts are provided through the user interface to inform the user of the current energy-saving measures so that the user can understand the current operating status of the XR device.

[0167] S307. When it is detected that the XR device has enabled power saving mode, the camera sensors with a master-slave relationship are alternately turned off.

[0168] Users can activate a power-saving mode, also known as a long-lasting mode, as needed. For example, the user can activate power-saving mode when the XR device's battery is low, or the XR device can automatically activate power-saving mode when its battery level falls below a certain threshold. When the service module detects that the XR device has activated power-saving mode, it will alternately shut down the camera sensors with a primary / backup relationship.

[0169] For example, Figure 6 The second camera group shown contains four 6DoF tracking cameras. The two upper 6DoF tracking cameras serve as the primary cameras for 6DoF tracking, while the two lower 6DoF tracking cameras serve as backup cameras. When the XR device is detected to be in power-saving mode, the service module alternately shuts down the two primary cameras and the two backup cameras. For example, the service module can first shut down the two backup cameras, then after a preset time, turn on the two backup cameras and shut down the two primary cameras, then after another preset time, turn on the two primary cameras and shut down the two backup cameras, and so on, to achieve alternating shutdown.

[0170] By alternately shutting down the camera sensors with a master-slave relationship, the main camera sensor takes turns generating heat, resulting in a better temperature for the entire device.

[0171] Optionally, after alternately turning off camera sensors with a primary / backup relationship, a power-saving prompt message is displayed to indicate that the camera sensor has been turned off.

[0172] It should be clarified that the thermal management strategy described in steps S304-S305 and the energy-saving strategy described in S306 can be executed simultaneously. In some embodiments, only the thermal management strategy may be executed, without the energy-saving strategy; in other embodiments, the energy-saving strategy may be executed, without the thermal management strategy.

[0173] In this embodiment, when the frame rate of the camera sensor is reduced or some camera sensors are turned off, the business module will use some optimization algorithms to optimize the situation of reducing the frame rate of the camera sensor or turning off some camera sensors, so as to ensure that the user experience is not greatly affected.

[0174] In the case of shutting down some camera sensors, the fault tolerance mechanism after camera sensor failure in Example 2 can be used for handling. The shut-down camera sensors can be regarded as faulty camera sensors, which will not be elaborated here.

[0175] When the frame rate of the camera sensor is reduced, image data of the missing frame can be obtained by interpolation and then processed using the interpolated image data. Alternatively, image data before the missing frame can be used to obtain the predicted image data of the missing frame and then processed based on the predicted image.

[0176] The thermal management and energy-saving strategies provided in this embodiment have the following advantages: (1) Automatic adjustment: The system can automatically adjust the frame rate of the camera sensor or turn off the camera sensor according to the real-time temperature of the camera sensor without user intervention; (2) Energy consumption balance: While maintaining acceptable image quality, it effectively reduces power consumption; (3) Improved user experience: The system maintains the continuity of visual experience through intelligent optimization algorithms and reduces the impact caused by energy-saving measures or thermal management strategies.

[0177] In this embodiment, the thermal management strategy significantly improves the stability and safety of the XR device in high-temperature environments, while the energy-saving strategy optimizes the device's battery life, enhancing the overall user experience. Furthermore, these thermal management and energy-saving strategies can help extend the device's lifespan and reduce maintenance and replacement costs caused by overheating.

[0178] In multi-camera systems, each camera sensor typically outputs sensor data independently, requiring the system to allocate a separate buffer in memory for each camera. As the number of cameras increases, the required memory resources also increase, potentially leading to inefficient memory usage. To address this technical problem, Embodiment 4 of this application provides a data processing method for a multi-camera XR device, employing a memory optimization strategy to save memory resources. Figure 8 This is a flowchart of the data processing method for a multi-camera XR device provided in Embodiment 4 of this application, as follows: Figure 8 As shown, the method provided in this embodiment includes the following steps.

[0179] S401: Obtain the hardware connection relationship between multiple camera sensors and the ISP.

[0180] Each aggregator combines at least two camera sensors and connects them to the corresponding ISP.

[0181] S402. Based on the hardware connection relationship and business requirements, divide the multiple camera sensors into multiple camera groups, and set parameters for the camera sensors in the camera groups that need to be activated.

[0182] In this system, the camera sensors within each camera group have the same function. The camera sensors within a camera group are connected to the same aggregator or directly to the ISP. The camera sensors connected to an aggregator are divided into one or more camera groups.

[0183] The specific implementation of steps S401-S402 is described in the relevant description of steps S101-S102 in Embodiment 1, and will not be repeated here.

[0184] S403. The camera synchronization signal is cyclically output to multiple camera sensors in the first camera group according to a fixed timing sequence, wherein the multiple camera sensors in the first camera group share the shared buffer corresponding to the first camera group through time-division multiplexing.

[0185] In XR devices, some camera groups include multiple camera sensors, while some camera groups include only one camera sensor. In this embodiment, the first camera group is any one of the camera groups that includes multiple camera sensors.

[0186] In this embodiment, the system allocates a shared buffer in memory for each first camera group. Multiple camera sensors in the first camera group share the shared buffer through time-division multiplexing. That is, the time when multiple camera sensors in the first camera group output sensor data should be staggered. The staggered time of output of sensor data by multiple camera sensors in the first camera group is a prerequisite for realizing buffer sharing.

[0187] For example, in this embodiment, the timing of the output sensor data of multiple camera sensors in the first camera group is staggered by using a camera synchronization signal. In one implementation, a microcontroller unit (MCU) outputs the camera synchronization signal in a fixed timing sequence, and each camera sensor in the first camera group outputs sensor data only after receiving the camera synchronization signal.

[0188] Assume the first camera group includes two camera sensors: Camera Sensor 1 and Camera Sensor 2. Starting from a given moment, the MCU first sends a camera synchronization signal to Camera Sensor 1. After Camera Sensor 1 stores its sensor data in the buffer, the MCU then sends a camera synchronization signal to Camera Sensor 2. After Camera Sensor 2 stores its sensor data in the buffer, the MCU then sends a camera synchronization signal to Camera Sensor 1, and so on, alternately sending camera synchronization signals to both Camera Sensor 1. Since the frequency or period at which each camera sensor acquires and outputs sensor data is fixed, the MCU can cyclically output camera synchronization signals to multiple camera sensors within the first camera group according to a fixed timing sequence.

[0189] In another implementation, each camera sensor sends a camera synchronization signal to the next camera sensor. The order in which multiple camera sensors in the first camera group send camera synchronization signals can be predetermined. Taking a first camera group consisting of two camera sensors as an example, camera sensor 1 sends the camera synchronization signal first and starts outputting sensor data first. After storing the sensor data in the buffer, camera sensor 1 sends the camera synchronization signal to camera sensor 2. After receiving the camera synchronization signal, camera sensor 2 starts outputting sensor data. After storing the sensor data in the buffer, camera sensor 2 sends the camera synchronization signal to camera sensor 1. If the camera synchronization signal is sent cyclically, the time of outputting sensor data by multiple camera sensors in the first camera group is staggered.

[0190] S404. Each camera sensor in the first camera group outputs sensor data after receiving the camera synchronization signal and stores the sensor data in the shared buffer.

[0191] Each camera sensor in the first camera group receives the camera synchronization signal cyclically and outputs sensor data, thereby enabling all camera sensors in the first camera group to share the shared buffer, which reduces the memory footprint of the camera system and improves system performance.

[0192] Optionally, in other embodiments of this application, each camera sensor within the first camera group may also employ an independent buffer.

[0193] Figure 9A schematic diagram illustrating the use of independent buffers for multiple camera sensors within the first camera group, as shown below. Figure 9 As shown, the first camera group includes two camera sensors. The system allocates a first buffer for camera sensor 1 and a second buffer for camera sensor 2. The sensor data of camera sensor 1 occupies the first buffer exclusively, and the sensor data of camera sensor 2 occupies the second buffer exclusively. In this memory allocation method, each camera sensor outputs sensor data independently. Therefore, multiple camera sensors in the first camera group can output sensor data simultaneously.

[0194] Figure 10 A schematic diagram illustrating the use of a shared buffer for multiple camera sensors within the first camera group, as shown below. Figure 9 As shown, the first camera group includes two camera sensors. The system allocates a shared buffer for the first camera group. Camera sensor 1 and camera sensor 2 output sensor data in a staggered time sequence through the camera synchronization signal (i.e., the sync signal in the figure), thereby realizing the time-division multiplexing of the shared buffer.

[0195] In this embodiment, an efficient buffer management algorithm is required so that sensor data output at different times can take turns using the same shared buffer. Optionally, the shared buffer can be a circular buffer, also known as a circular buffer. One characteristic of a circular buffer is that when a data element is used, the remaining data elements do not need to move their storage locations, and the storage space in the circular buffer can be written to and read from in a circular manner.

[0196] When this shared buffer is time-division multiplexed, it is necessary to ensure that the output sensor data is not overwritten by the subsequently output sensor data before it is processed by the ISP. Therefore, an appropriate size needs to be set for this shared buffer. Optionally, the size of the shared buffer is determined based on the ISP's processing speed. When the ISP's processing speed is fast, the shared buffer can be smaller; when the ISP's processing speed is slow, the shared buffer can be larger. The size of the shared buffer must at least ensure that the subsequently output sensor data does not overwrite the sensor data that has already been output but not yet processed by the ISP.

[0197] When using a shared buffer, since the timing of the sensor data output by each camera sensor in the first camera group needs to be staggered, the shared buffer is more suitable for cameras with less stringent timing requirements.

[0198] S405, the ISP reads sensor data sent by the camera sensor from the buffer corresponding to the first camera group based on the parameters of the connected camera group and the camera sensor within the camera group, and processes the read sensor data to obtain image data.

[0199] When a shared buffer is used, the ISP reads sensor data from multiple camera sensors from the shared buffer. The ISP needs to distinguish the sensor data from multiple camera sensors. In one implementation, the sensor data output by each camera sensor in the first camera group includes camera sensor information, which can be the camera sensor's identifier. The ISP determines the camera sensor corresponding to the sensor data based on the camera sensor information included in the read sensor data, and processes the sensor data according to the corresponding camera sensor.

[0200] The memory optimization strategy provided in this embodiment is applicable to XR devices with limited memory resources. This memory optimization strategy can significantly reduce memory usage and improve system performance by using time-sharing multiplexing of the buffer.

[0201] It should be clarified that the fault tolerance mechanism provided in Embodiment 2, the thermal management strategy and energy-saving strategy provided in Embodiment 3, and the memory optimization strategy provided in Embodiment 4 can be applied individually in XR devices, or they can be combined in any way and applied to XR devices.

[0202] To facilitate better implementation of the data processing method for multi-camera XR devices according to the embodiments of this application, the embodiments of this application also provide a data processing apparatus for multi-camera XR devices. Figure 11 This is a schematic diagram of the data processing apparatus for a multi-camera XR device provided in Embodiment 5 of this application, as shown below. Figure 11 As shown, the data processing device 200 of the multi-camera XR device may include:

[0203] The acquisition module 21 is used to acquire the hardware connection relationship between multiple camera sensors of the XR device and the ISP. The XR device includes multiple camera sensors, at least one aggregator and at least one image signal processor (ISP). Each aggregator aggregates at least two camera sensors and connects them to the corresponding ISP.

[0204] Grouping module 22 is used to divide the multiple camera sensors into multiple camera groups according to the hardware connection relationship and business requirements, and set parameters for the camera sensors in the camera groups that need to be activated. The camera sensors in each camera group have the same function, and the camera sensors in the camera group are connected to the same aggregator or directly connected to the ISP.

[0205] The service module 23 is used to perform service processing based on the image data of the multiple camera sensors obtained by the ISP. The ISP is used to receive sensor data sent by the camera sensors according to the parameters of the connected camera group and the camera sensors in the camera group, and process the sensor data to obtain image data.

[0206] In some implementations, the business module 23 is further used for:

[0207] When it is determined that the first camera sensor is faulty, the first camera sensor is removed from the target camera group to which the first camera sensor belongs.

[0208] When a backup camera sensor for the first camera sensor exists in the XR device, the image data of the backup camera sensor is acquired, and the image data of the backup camera sensor and the image data of the normally functioning camera sensor in the target camera group are processed to obtain the first processing result of the first service.

[0209] When there is no backup camera sensor for the first camera sensor in the XR device, a compensation algorithm is used to process the image data of the normally functioning camera sensor in the target camera group to obtain the second processing result of the first service.

[0210] In some implementations, the backup camera sensor belongs to the target camera group, or the backup camera sensor does not belong to the target camera group.

[0211] In some implementations, the backup camera sensor is a camera sensor whose field of view overlaps with that of the first camera sensor, or the backup camera sensor is a camera sensor that has the same function as the first camera sensor.

[0212] In some implementations, the compensation algorithm is an image fusion algorithm or a deep learning model.

[0213] In some implementations, the device further includes an output module for outputting visual and / or auditory cue information, wherein the visual and / or auditory cue information is used to indicate a fault in the first camera sensor and to perform fault recovery processing.

[0214] In some implementations, the service module 23 is further configured to: acquire the temperature of the plurality of camera sensors; and when the temperature of at least one camera sensor is greater than a preset temperature threshold, reduce the frame rate of some camera sensors or turn off some camera sensors.

[0215] In some implementations, the business module 23 is specifically used for:

[0216] Reduce the frame rate of camera sensors whose temperature exceeds the stated temperature threshold;

[0217] Alternatively, based on the priority of the plurality of camera sensors, the frame rate of low-priority camera sensors may be reduced or low-priority camera sensors may be turned off.

[0218] Alternatively, reduce the frame rate of non-critical camera sensors;

[0219] Alternatively, reduce the frame rate of the backup camera sensor of the primary camera sensor, or turn off the backup camera sensor of the primary camera sensor.

[0220] In some implementations, the device further includes a display module for displaying energy-saving prompts, the prompts indicating that the frame rate of the camera sensor has been reduced or that the camera sensor has been turned off.

[0221] In some implementations, the service module 23 is also used to: when the XR device is detected to have enabled power-saving mode, alternately turn off the camera sensors with a master-slave relationship.

[0222] In some implementations, when the first camera group includes multiple camera sensors, the multiple camera sensors in the first camera group share the shared buffer corresponding to the first camera group through time-division multiplexing.

[0223] In some implementations, the device further includes a control module;

[0224] The control module is used to cyclically output camera synchronization signals to multiple camera sensors in the first camera group according to a fixed timing sequence.

[0225] Each camera sensor in the first camera group outputs sensor data after receiving the camera synchronization signal and stores the sensor data in the shared buffer;

[0226] The ISP reads sensor data sent by the camera sensors from the shared buffer corresponding to the first camera group based on the parameters of the connected camera group and the camera sensors within the camera group, and processes the read sensor data to obtain image data.

[0227] In some implementations, the sensor data output by each camera sensor in the first camera group includes information about the camera sensor, and the processing of the read sensor data to obtain image data includes:

[0228] Based on the camera sensor information included in the read sensor data, determine the camera sensor corresponding to the sensor data;

[0229] The sensor data is processed based on the camera sensor corresponding to the sensor data.

[0230] In some implementations, the shared buffer is a circular buffer, and the size of the shared buffer is determined according to the processing speed of the ISP. The size of the shared buffer ensures that the sensor data output later will not overwrite the sensor data that has been output but not processed by the ISP.

[0231] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here.

[0232] The apparatus 200 of this application embodiment has been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that this functional module can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the method disclosed in this application embodiment can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0233] This application also provides an XR device. Figure 12 This is a schematic diagram of the structure of an XR device provided in Embodiment Six of this application, as shown below. Figure 12 As shown, the XR device 300 may include a memory 31, a processor 32, and a camera system 33, wherein the camera system 33 and the memory 31 are connected to the processor 32.

[0234] The camera system 33 includes multiple camera sensors, at least one aggregator, and at least one ISP. Each aggregator aggregates at least two camera sensors and connects them to the corresponding ISP. The ISP is used to receive sensor data sent by the camera sensors according to the parameters of the connected camera group and the camera sensors in the camera group, and to process the sensor data to obtain image data.

[0235] The memory 31 is used to store computer programs and transfer the program code to the processor 32. In other words, the processor 32 can call and run the computer program from the memory 31 to implement the data processing method for the multi-camera XR device provided in this application embodiment.

[0236] For example, the processor 32 can be used to execute the above-described method embodiments according to instructions in the computer program. In some embodiments of this application, the processor 32 may include, but is not limited to:

[0237] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0238] In some embodiments of this application, the memory 31 includes, but is not limited to:

[0239] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0240] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to perform the method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in an XR device.

[0241] like Figure 12 As shown, the XR also includes a transceiver 34, a display screen 35, etc., and the processor 32 is electrically connected to the transceiver 34 and the display screen 35 respectively.

[0242] The processor 32 can control the transceiver 34 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 33 may include a transmitter and a receiver. The transceiver 34 may further include antennas, and the number of antennas may be one or more.

[0243] The display screen 35 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The display screen 35 can be a touch screen, which may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the computer device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 32. It can also receive and execute commands from the processor 32. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 32 to determine the type of touch event. Subsequently, the processor 32 provides corresponding visual output on the display panel based on the type of touch event.

[0244] Understandable, although Figure 12As not shown in the diagram, the XR device 300 may also include an audio module, a Wi-Fi module, a positioning module, a Bluetooth module, etc., which will not be described in detail here.

[0245] It should be understood that the various components in the XR device are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0246] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0247] This application also provides a computer program product comprising a computer program stored in a computer-readable storage medium. The processor of an XR device reads the computer program from the computer-readable storage medium and executes the computer program, causing the XR device to perform the corresponding flow of the method provided in the embodiments of this application. For brevity, further details are omitted here.

[0248] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0249] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0250] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method for a multi-camera XR device, characterized in that, Extended Reality (XR) devices include multiple camera sensors, at least one aggregator, and at least one image signal processor (ISP), and the method includes: Obtain the hardware connection relationship between the multiple camera sensors and the ISP, wherein each aggregator aggregates at least two camera sensors and connects them to the corresponding ISP; Based on the hardware connection relationship and business requirements, the multiple camera sensors are divided into multiple camera groups, and parameters are set for the camera sensors in the camera groups that need to be activated. The camera sensors in each camera group have the same function, and the camera sensors in the camera group are connected to the same aggregator or directly connected to the ISP. The ISP processes image data from multiple camera sensors obtained through processing. The ISP receives sensor data sent by the camera sensors based on the parameters of the connected camera group and the camera sensors within the camera group, and processes the sensor data to obtain image data.

2. The method according to claim 1, characterized in that, Also includes: When it is determined that the first camera sensor is faulty, the first camera sensor is removed from the target camera group to which the first camera sensor belongs. The process of performing service processing based on image data from multiple camera sensors obtained by the ISP includes: When a backup camera sensor for the first camera sensor exists in the XR device, the image data of the backup camera sensor is acquired, and the image data of the backup camera sensor and the image data of the normally functioning camera sensor in the target camera group are processed to obtain the first processing result of the first service. When there is no backup camera sensor for the first camera sensor in the XR device, a compensation algorithm is used to process the image data of the normally functioning camera sensor in the target camera group to obtain the second processing result of the first service.

3. The method according to claim 2, characterized in that, The backup camera sensor belongs to the target camera group, or the backup camera sensor does not belong to the target camera group.

4. The method according to claim 2, characterized in that, The backup camera sensor is a camera sensor whose field of view overlaps with that of the first camera sensor, or the backup camera sensor is a camera sensor with the same function as the first camera sensor.

5. The method according to claim 2, characterized in that, The compensation algorithm is an image fusion algorithm or a deep learning model.

6. The method according to claim 2, characterized in that, The method further includes: The system outputs visual and / or auditory cues, which are used to indicate a fault in the first camera sensor and to perform fault recovery processing.

7. The method according to any one of claims 1-6, characterized in that, Also includes: The temperatures of the multiple camera sensors are obtained; When the temperature of at least one camera sensor exceeds a preset temperature threshold, reduce the frame rate of some camera sensors or shut down some camera sensors.

8. The method according to claim 7, characterized in that, The reduction of the frame rate of some camera sensors or the shutdown of some camera sensors includes: Reduce the frame rate of camera sensors whose temperature exceeds the stated temperature threshold; Alternatively, based on the priority of the plurality of camera sensors, the frame rate of low-priority camera sensors may be reduced or low-priority camera sensors may be turned off. Alternatively, reduce the frame rate of non-critical camera sensors; Alternatively, reduce the frame rate of the backup camera sensor of the primary camera sensor, or turn off the backup camera sensor of the primary camera sensor.

9. The method according to any one of claims 1-6, characterized in that, Also includes: The system displays an energy-saving warning message, which indicates that the frame rate of the camera sensor has been reduced or the camera sensor has been turned off.

10. The method according to any one of claims 1-6, characterized in that, Also includes: When the XR device is detected to have entered power-saving mode, the camera sensors with a primary / backup relationship are alternately turned off.

11. The method according to any one of claims 1-6, characterized in that, When the first camera group includes multiple camera sensors, the multiple camera sensors in the first camera group share the shared buffer corresponding to the first camera group through time-division multiplexing.

12. The method according to claim 11, characterized in that, The method further includes: Camera synchronization signals are cyclically output to multiple camera sensors within the first camera group according to a fixed timing sequence. Each camera sensor in the first camera group outputs sensor data after receiving the camera synchronization signal and stores the sensor data in the shared buffer; The ISP reads sensor data sent by the camera sensors from the shared buffer corresponding to the first camera group based on the parameters of the connected camera group and the camera sensors within the camera group, and processes the read sensor data to obtain image data.

13. The method according to claim 12, characterized in that, The sensor data output by each camera sensor in the first camera group includes information about the camera sensor. The process of processing the read sensor data to obtain image data includes: Based on the camera sensor information included in the read sensor data, determine the camera sensor corresponding to the sensor data; The sensor data is processed based on the camera sensor corresponding to the sensor data.

14. The method according to claim 12 or 13, characterized in that, The shared buffer is a circular buffer, and its size is determined based on the processing speed of the ISP. The size of the shared buffer ensures that the sensor data output later will not overwrite the sensor data that has already been output but not processed by the ISP.

15. A data processing apparatus for a multi-camera XR device, characterized in that, The device includes: The acquisition module is used to acquire the hardware connection relationship between multiple camera sensors of the XR device and the ISP. The XR device includes multiple camera sensors, at least one aggregator and at least one image signal processor (ISP). Each aggregator aggregates at least two camera sensors and connects them to the corresponding ISP. The grouping module is used to divide the multiple camera sensors into multiple camera groups according to the hardware connection relationship and business requirements, and set parameters for the camera sensors in the camera groups that need to be activated. The camera sensors in each camera group have the same function, and the camera sensors in the camera group are connected to the same aggregator or directly connected to the ISP. The service module is used to perform service processing based on the image data of the multiple camera sensors obtained by the ISP. The ISP is used to receive sensor data sent by the camera sensors according to the parameters of the connected camera group and the camera sensors in the camera group, and process the sensor data to obtain image data.

16. An XR device, characterized in that, include: A camera system, a processor, and a memory, wherein the camera system and the memory are connected to the processor; The camera system includes multiple camera sensors, at least one aggregator, and at least one image signal processor (ISP). Each aggregator connects at least two camera sensors to the corresponding ISP. The ISP is used to receive sensor data sent by the camera sensors according to the parameters of the connected camera group and the camera sensors within the camera group, and to process the sensor data to obtain image data. The memory is used to store computer programs, and the processor is used to call and run the computer programs stored in the memory to perform the method according to any one of claims 1 to 14.

17. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 14.

Images