Generation and manipulation of computer vision-based extended reality applications

Abstract

A method for facilitating generation and / or operation of computer vision-based extended reality (XR) applications. For example, the method includes generating an application suite of computer vision-based XR applications, initializing the computer vision-based XR applications, and operating the computer vision-based XR applications. The method can be performed using an XR device and / or an external computing system.

Description

Technical Field

[0001] The present invention relates to the generation and operation of computer vision-based extended reality (XR) applications. Background Art

[0002] Extended reality (XR) technology is generally associated with immersive technologies that can extend reality. Common types of extended reality (XR) technology include augmented reality (AR), virtual reality (VR), and mixed reality (MR).

[0003] Some existing XR applications use computer vision-based techniques. However, to date, the development and generation of computer vision-based XR applications for XR devices remains challenging because significant effort, resources, and / or expertise may be required to properly integrate computer vision-based techniques with the XR applications. In some cases, the operation of computer vision-based XR applications (e.g., multiple applications on multiple XR devices) may be difficult to manage and may require significant computing resources. Summary of the invention

[0004] An object of some embodiments of the present invention is to provide a tool for facilitating the development and generation of computer vision-based XR applications (e.g., AR applications, VR applications, and / or MR applications). An object of some embodiments of the present invention is to provide a method for facilitating the initialization of computer vision-based XR applications. An object of some embodiments of the present invention is to provide a method for facilitating the operation of computer vision-based XR applications (at an XR device and / or an associated external computing system). An object of some embodiments of the present invention is to provide a method for supporting multiple instances of a computer vision-based XR application at an external computing system associated with multiple XR devices. An object of some embodiments of the present invention is to provide a method for controlling and / or managing the collaboration between XR devices and associated external computing systems and their performance. Some embodiments of the present invention may include one or more additional or alternative objects.

[0005] In a first aspect, a method for facilitating generation and / or operation of computer vision-based extended reality (XR) applications is provided. The computer vision-based XR applications can be arranged to: assist navigation (e.g., assist a user of an XR device to traverse, manipulate, or perform path finding in a physical environment), assist operation (e.g., instruct, guide, or teach a user of an XR device to perform one or more actions or tasks in a physical environment), remote assistance support (e.g., provide assistance to a user of an XR device, such as through screen sharing), assist recording (e.g., capture and optionally share data detected in a physical environment).

[0006] In some implementations of the first aspect, the method includes generating an application suite of computer vision-based XR applications. The generating can be performed at a computing system.

[0007] Optionally, the generating includes: receiving data comprising at least user experience and user interface design data for a computer vision-based XR application and computer vision-based XR instruction data for the computer vision-based XR application; and processing the data to generate an application suite for the computer vision-based XR application.

[0008] User experience and user interface design data may include audio and / or visual data arranged to be presented by the XR device. The audio and / or visual data may be associated with one or more user interfaces (UIs) (for visual data, graphical user interfaces GUIs) arranged to be displayed by the XR device or presented to the user of the XR device. The visual data may include images, text, virtual objects, etc. associated with one or more GUIs, and may correspond to menus, instructions, information, commands, guidance, etc., which are arranged to be displayed to the user of the XR device. The audio data may include sounds, noises, music, voices, etc. to be played by the XR device to be presented to the user of the XR device. The visual and / or audio data may also include: data associated with one or more attributes of the visual data (e.g., rendering attributes, UI design-related parameters (e.g., font size, color, transparency), etc.), data associated with one or more attributes of the audio data (e.g., synthesis attributes, sound processing attributes, etc.).

[0009] The computer vision-based XR instruction data may include data associated with computer vision-based object detection (e.g., data of a target object to be detected, a target object detection algorithm / criteria / logic, etc.), data associated with a computer vision-based object detection response (algorithm / logic associated with a response provided when the computer vision detects a target object), etc. The data associated with the computer vision-based object detection response may include data associated with one or more conditions associated with computer vision-based object detection for affecting the appearance of a user interface, such as: data associated with the order of user interfaces; data associated with one or more conditions for activating one or more user interfaces (audio and / or visual), data associated with one or more conditions for switching from one user interface to another user interface (audio and / or visual), data associated with one or more conditions for deactivating one or more user interfaces (audio and / or visual), etc. Each user interface may correspond to a corresponding state.

[0010] Optionally, the application suite is arranged to be deployed at the XR device and / or the external computing system for operating the computer vision-based XR application. In some examples, the application suite is fully deployed at the XR device for operating the computer vision-based XR application. In some examples, the application suite is fully deployed at the external computing system for operating the computer vision-based XR application. In some examples, the application suite is partially deployed at the XR device and partially deployed at the external computing system for operating the computer vision-based XR application.

[0011] Optionally, the received and / or processed data also includes data associated with a storage location storing data associated with the computer vision-based XR application. The storage location may be a storage location at an external computing system.

[0012] Optionally, the application suite includes at least: specification data of the computer vision-based XR application, a state machine of the computer vision-based XR application, and a computer vision-based processing algorithm of the computer vision-based XR application.

[0013] For example, the specification data may include data (e.g., audio and / or visual) associated with different guidance and / or instructions that the computer vision-based XR application is arranged to provide. The data structure of the specification data may include a plurality of guidance identifiers (each of which corresponds to a respective guidance that the computer vision-based XR application is arranged to provide to a user of the XR device), each guidance identifier being linked to associated visual and / or audio data.

[0014] For example, the state machine may include data associated with user interface sequences (audio and / or visual) corresponding to different instructions and / or commands, as well as data for affecting the appearance of the user interface (e.g., generating a new user interface, changing an existing user interface, deactivating / deleting an existing user interface, etc.). The data for affecting the appearance of the user interface may include a target object identifier corresponding to a target object identified by computer vision-based processing. The target object identifier may be linked to a guidance identifier so that corresponding guidance may be triggered when the target object is detected. The data structure of the state machine may include multiple states, each of which includes a target object identifier, a corresponding guidance identifier, and information about the next state (if any).

[0015] For example, a computer vision-based processing algorithm may include data associated with computer vision-based target object detection. The data associated with computer vision-based target object detection may include data of the target object and associated detection criteria. The data structure of the computer vision-based processing algorithm may include multiple object detections, where each object detection includes a target object identifier and a corresponding detection criterion.

[0016] Optionally, specification data for the computer vision-based XR application is arranged to be deployed at an XR device for operating the computer vision-based XR application.

[0017] Optionally, the state machine of the computer vision-based XR application is arranged to be deployed at an external computing system for operating the computer vision-based XR application.

[0018] Optionally, the computer vision-based processing algorithm of the computer vision-based XR application is arranged to be deployed at an external computing system for operating the computer vision-based XR application.

[0019] Optionally, the received and / or processed data also includes one or more of the following: data transmission control parameters for operating the computer vision-based XR application, and data associated with one or more characteristics of the XR device for operating the computer vision-based XR application. The data transmission control parameters may be data transmission control parameters between the XR device and the external computing system, for example, related to data transmission from the XR device to the external computing system. The data transmission control parameters may include, for example, the number of image frames per transmission from the XR device to the external computing system, the maximum image width for images transmitted from the XR device to the external computing system, the minimum image width for images transmitted from the XR device to the external computing system, the minimum frame rate for images transmitted from the XR device to the external computing system, the maximum frame rate for images transmitted from the XR device to the external computing system, and the like. The one or more characteristics of the XR device may be related to the hardware limitations of the XR device, and may include the brand of the XR device, the model of the XR device, the processing power of the XR device, the communication capabilities of the XR device, the available memory storage of the XR device, the power capacity of the XR device, and the like. Optionally, the application suite also includes a data transmission strategy for operating the computer vision-based XR application. The data transfer policy is arranged to regulate data communication between the XR device and the external computing system (in particular, data communication from the XR device to the external computing system).

[0020] In some implementations of the first aspect, the method includes initializing a computer vision-based XR application.

[0021] Optionally, initialization includes performing the following at the XR device: transmitting an identifier associated with the computer vision-based XR application to the external computing system using the address of the external computing system; and in response to the transmission of the identifier, receiving access information for accessing an instance of the computer vision-based XR application from the external computing system. The instance corresponds to a virtual machine. The instance is arranged to be operated by the external computing system and may include a state machine engine for the computer vision-based XR application and / or a computer vision-based processing engine for the computer vision-based XR application. Optionally, the instance for the computer vision-based XR application is generated at the external computing system in response to the external computing system receiving the identifier from the XR device. Optionally, initialization also includes performing the following at the XR device: receiving or retrieving an identifier associated with the computer vision-based XR application and the address of the external computing system from a computing system (which may or may not be an external computing system).

[0022] Optionally, initialization includes performing the following at an external computing system: receiving an identifier associated with a computer vision-based XR application, and in response, generating an instance for the computer vision-based XR application based on the identifier. The instance corresponds to a virtual machine. The instance is arranged to be operated by the external computing system and may include a state machine engine for the computer vision-based XR application and / or a computer vision-based processing engine for the computer vision-based XR application. Optionally, initialization also includes performing the following at the external computing system: transmitting access information for accessing the instance of the computer vision-based XR application to the XR device. Optionally, the generation includes determining or obtaining computing resource settings for the instance of the computer vision-based XR application based on the identifier (e.g., computing resource settings that can be used for generation), and the instance is generated based on the determined or obtained computing resource settings.

[0023] In some embodiments of the first aspect, the method includes operating a computer vision-based XR application.

[0024] Optionally, the operation includes performing, at the XR device, accessing the instance at the external computing system using the address of the external computing system and the received access information. For example, the access includes transferring data from the XR device to the instance at the external computing system (using the address of the external computing system and the received access information).

[0025] Optionally, the operation includes performing the following at the XR device: acquiring an image, transmitting the image (i.e., image data) to an external computing system for performing computer vision-based processing of the image at an instance, receiving data associated with a processing result of the computer vision-based image processing, and processing the received data associated with the processing result to implement a response at the XR device. The above process can be repeated for different images captured by the XR device. The transmission of the image to the external computing system can include transmitting one or more images (captured by the image sensor of the XR device) at a time.

[0026] Optionally, acquiring the image at the XR device includes capturing an image of an environment (in which the XR device is disposed) (ie, in a field of view of the image sensor) using an image sensor of the XR device.

[0027] Optionally, acquiring an image at the XR device includes receiving an image (i.e., image data) associated with screen content of a screen mirroring or screen sharing device. The screen content may be any content being displayed or arranged to be displayed on at least a portion of a screen of the screen mirroring or screen sharing device. The screen mirroring or screen sharing device may be a device in a field of view of an image sensor of the XR device (without being imaged using the image sensor of the XR device).

[0028] Optionally, the data associated with the processing result includes data associated with a response to be implemented at the XR device.

[0029] Optionally, the computer vision-based processing is arranged to detect one or more target objects in the image, and the data associated with the processing result includes data associated with a target object detection result in response to detecting the one or more target objects in the image. The detection of the one or more target objects in the image can be based on the presence or one or more attributes (e.g., size, orientation, color, etc.) of the target object in the image.

[0030] Optionally, the data associated with the target object detection results includes data for implementing a response at the XR device in response to detecting one or more target objects in the image.

[0031] Optionally, the response includes a visual and / or an audio response.

[0032] Optionally, the visual and / or audio responses are associated with XR-based instructions or guidance information to be provided by the XR device to a user of the XR device.

[0033] Optionally, the XR-based instructions or guidance information is arranged to assist or facilitate a user in performing an action or task (e.g., installation (software / hardware), education, training, maintenance, inspection, troubleshooting, etc.).

[0034] Optionally, the visual response includes generating, changing or terminating a graphical user interface arranged to be provided by the XR device. Optionally, the audio response includes generating, changing or terminating a sound (e.g., noise, speech, etc.) arranged to be provided by the XR device.

[0035] Optionally, the transmission of the image is performed according to a data transmission strategy, which is adapted to the communication quality or quality changes between the XR device and the external computing system and / or the processing performance or processing performance changes of the external computing system.

[0036] Optionally, the data transmission strategy defines one or more of the following transmission control parameters: the number of image frames transmitted each time from the XR device to the external computing system, the maximum image width of images transmitted from the XR device to the external computing system, the minimum image width of images transmitted from the XR device to the external computing system, the minimum frame rate of images transmitted from the XR device to the external computing system, the maximum frame rate of images transmitted from the XR device to the external computing system, etc.

[0037] Optionally, the data transmission strategy is arranged to affect the resolution of the image and / or the frame rate associated with the image based on the quality or changes in quality of communication between the XR device and the external computing system and / or the processing performance or changes in processing performance of the external computing system.

[0038] Optionally, the quality of communication or a change in quality is related to one or more of: the latency (instantaneous, cumulative, average, etc., optionally weighted) of one or more communication links between the XR device and the external computing system, the speed (instantaneous, cumulative, average, etc., optionally weighted) of one or more communication links between the XR device and the external computing system, the bandwidth (instantaneous, cumulative, average, etc., optionally weighted) of one or more communication links between the XR device and the external computing system, the packet loss rate or number of packet losses (instantaneous, cumulative, average, etc., optionally weighted) between the XR device and the external computing system.

[0039] Optionally, the quality of the external computing system or the change in processing performance is related to one or more of: the processing speed of the external computing system (instantaneous, cumulative, average, etc., optionally weighted), the processing throughput of the external computing system (instantaneous, cumulative, average, etc., optionally weighted), etc.

[0040] Optionally, the communication quality and / or processing performance of the external computing system is determined based on one or more of: the number of images transmitted from the XR device over a period of time, the number of images received at the external computing system over a period of time, the amount of data associated with the number of processing results generated by the external computing system over a period of time, the amount of data associated with the number of processing results transmitted from the external computing system over a period of time, and the amount of data associated with the number of processing results received at the XR device over a period of time.

[0041] Optionally, the operations further include, prior to transmitting the image: determining, in substantially real time, a quality or change in quality of a communication link between the XR device and the external computing system and / or a processing performance or change in processing performance of the external computing system, and determining a resolution of the image for transmission to the external computing system and / or a frame rate associated with the image based on the determined quality or change in quality and / or processing performance or change in processing performance. The transmission of the image may then be based on the determined resolution and / or frame rate. These two determination steps may be performed at the XR device, at the external computing system, or at both the XR device and the external computing system.

[0042] Optionally, if the quality of the communication link is at a first quality level, the transmission includes transmitting the image at a first resolution and / or a first frame rate, and if the quality of the communication link is at a second quality level lower than the first quality level, the transmission includes transmitting the image using a second resolution lower than the first resolution and / or a second frame rate lower than the first frame rate.

[0043] Optionally, if the quality or change in quality of the communication and / or the processing performance or change in processing performance of the external computing system is determined to be degraded, then: the frame rate associated with the image for transmission to the external computing system is reduced (if the frame rate has not reached a minimum frame rate), the image resolution for transmission to the external computing system is reduced (if the frame rate has reached the minimum frame rate and the resolution has not reached the minimum resolution), and error information is provided at the XR device (if the frame rate has reached the minimum frame rate and the resolution has reached the minimum resolution).

[0044] Optionally, if the quality or change in quality of the communication and / or the processing performance or change in processing performance of the external computing system is determined to have increased, then: the resolution of the image for transmission to the external computing system is increased (if the resolution has not reached a maximum resolution), and the frame rate associated with the image for transmission to the external computing system is increased (if the resolution has reached a maximum resolution and the frame rate has not reached a maximum frame rate).

[0045] Optionally, the operations include performing at the external computing system: receiving an image, processing the image using computer vision-based processing at the instance, and transmitting data associated with a processing result of the computer vision-based processing of the image to the XR device to facilitate implementing a response at the XR device.

[0046] Optionally, the data associated with the processing result includes data associated with a response to be implemented at the XR device.

[0047] Optionally, the computer vision-based processing is arranged to detect one or more target objects in the image, and the data associated with the processing result includes data associated with a target object detection result in response to detecting the one or more target objects in the image. The detection of the one or more target objects in the image can be based on the presence or one or more attributes (e.g., size, orientation, color, etc.) of the target object in the image.

[0048] Optionally, the data associated with the target object detection results includes data for implementing a response at the XR device in response to detecting one or more target objects in the image.

[0049] Optionally, the response includes a visual and / or an audio response.

[0050] Optionally, the visual and / or audio responses are associated with XR-based instructions or guidance information to be provided by the XR device to a user of the XR device.

[0051] Optionally, the XR-based instructions or guidance information is arranged to assist or facilitate a user in performing an action or task (e.g., installation (software / hardware), education, training, maintenance, inspection, troubleshooting, etc.).

[0052] Optionally, the visual response includes generating, changing or terminating a graphical user interface arranged to be provided by the XR device, and / or the audio response includes generating, changing or terminating a sound (e.g., noise, speech, etc.) arranged to be provided by the XR device.

[0053] Optionally, the operations include performing the following at the XR device: receiving a prompt signal from the external computing system, and in response to receiving the prompt signal, transmitting a keep-alive signal to the external computing system to maintain the operation of the instance in the external computing system.

[0054] Optionally, the operation includes performing the following at the external computing system: transmitting a prompt signal to the XR device, determining whether a corresponding keep-alive signal is received from the XR device, and then: if it is determined that the corresponding keep-alive signal is received, maintaining the operation of the instance at the external computing system, and if it is determined that the corresponding keep-alive signal is not received, terminating the instance at the external computing system to release computing resources associated with the instance. The external computing system device may attempt to transmit a prompt signal to the XR device periodically or irregularly (for example, according to a set schedule) to determine its presence and the activity status of the computer vision-based XR application.

[0055] Optionally, the operation includes performing the following at the external computing system: transmitting a prompt signal to the XR device (e.g., one at a time), determining a total number of corresponding keep-alive signals received from the XR device, and then: if it is determined that the total number of corresponding keep-alive signals is above a set number, maintaining the operation of the instance in the external computing system; if it is determined that the total number of corresponding keep-alive signals is below a set number, terminating the operation of the instance in the external computing system to release computing resources associated with the instance.

[0056] Optionally, the operation includes performing the following at the XR device: determining that the operation of the computer vision-based XR application is about to be terminated or has been terminated, and in response to the determination, transmitting a termination signal to the external computing system to terminate the instance in the external computing system, thereby releasing computing resources associated with the instance. Determining that the operation of the computer vision-based XR application is about to be terminated or has been terminated can be based on user input received at the XR device, the state of the XR device, the state of the computer vision-based XR application, etc.

[0057] Optionally, the operation includes performing the following in the external computing system: receiving a termination signal associated with termination of operation of the computer vision-based XR application from the XR device, and in response to receiving the termination signal, terminating the instance at the external computing system to release computing resources associated with the instance.

[0058] In a first aspect, the computer vision-based XR application may be a computer vision-based augmented reality (AR) application, a computer vision-based virtual reality (VR) application, or a computer vision-based mixed reality (MR) application.

[0059] In the first aspect, the XR device can be an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device. The XR device can be in the form of glasses, a headset, or other wearable and / or handheld computing devices (e.g., phones, tablets, etc.).

[0060] In a first aspect, the external computing system may be a computing system located outside the XR device (and operably connected to the XR device). The external computing system includes means for performing or facilitating the performance of the method of the first aspect. The external computing system may include one or more processors and a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing or facilitating the performance of the method of the first aspect. The external computing system may be provided by one or more computing devices.

[0061] In a second aspect, an XR device is provided, which is arranged to perform or facilitate the performance of the method of the first aspect. The XR device can be arranged to operate as the XR device in the method of the first aspect. The XR device can be an augmented reality (AR) device, a virtual reality (VR) device, or a mixed reality (MR) device. The XR device can be in the form of glasses, a headset, or other wearable and / or handheld computing devices (e.g., phones, tablets, etc.).

[0062] In a third aspect, a computing system is provided, which is arranged to perform or facilitate the performance of the method of the first aspect. The computing system can be arranged to operate as an external computing system in the method of the first aspect. The computing system can be arranged to operate as a computing system for generating computer vision-based XR applications. The computing system includes different means for performing or facilitating the performance of the method of the first aspect. The computing system may include one or more processors and a memory, the memory storing one or more programs configured to be executed by one or more processors, the one or more programs including instructions for performing or facilitating the performance of the method of the first aspect. The computing system may be provided by one or more computing devices. The computing system may be a cloud computing system.

[0063] In a fourth aspect, a system is provided, comprising one or more XR devices of the second aspect and one or more computing systems of the third aspect. The XR devices of the second aspect and the computing systems of the third aspect are arranged to communicate data directly or indirectly via a wired and / or wireless communication link. The system may also include a database (server).

[0064] In a fifth aspect, a non-transitory computer-readable storage medium is provided, storing one or more programs configured to be executed by one or more processors, the one or more programs comprising instructions for performing or facilitating the performance of the method of the first aspect.

[0065] In a sixth aspect, a computer program product is provided, comprising one or more programs configured to be executed by one or more processors, the one or more programs comprising instructions for performing or facilitating the performance of the method of the first aspect. The computer program product may be implemented using hardware and / or software.

[0066] By considering the detailed description and the accompanying drawings, other features and aspects of the present invention will become apparent. Where appropriate and applicable, any one or more features described herein for one aspect or embodiment may be combined with any one or more other features described herein for any one or more other aspects or embodiments.

[0067] Depending on the context, terms of degree such as "generally", "approximately", "substantially", etc. are used to illustrate manufacturing tolerances, degradation, trends, tendencies, imperfect physical conditions, etc.

[0068] Unless stated otherwise, the terms "connected," "coupled," "mounted," etc. are intended to cover both direct and indirect connections, couplings, mountings, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0070] Figure 1 is a schematic diagram of a system for operating or facilitating the operation of a computer vision-based XR application according to some embodiments of the present invention.

[0071] Figure 2 is a block diagram illustrating an application suite for generating computer vision-based XR applications according to one embodiment of the present invention.

[0072] Figure 3 is a block diagram showing a data structure of an application specification of a computer vision-based XR application according to an embodiment of the present invention.

[0073] Figure 4 is a block diagram showing the data structure of a state machine of a computer vision-based XR application according to one embodiment of the present invention.

[0074] Figure 5 1 is a block diagram showing the data structure of a computer vision-based processing algorithm of a computer vision-based XR application according to one embodiment of the present invention.

[0075] Figure 6 and Figure 7 is a flowchart illustrating a method for initializing a computer vision-based XR application according to some embodiments of the present invention.

[0076] Figures 8 to 20is a flowchart illustrating a method for operating a computer vision-based XR application according to some embodiments of the present invention.

[0077] Fig.21 is a schematic diagram of a system (including multiple XR devices, an external computing system, and a database) according to some embodiments of the present invention.

[0078] Fig. 22 is a block diagram of an example information processing system that may be used to implement, at least in part, one or more methods of the present invention in accordance with some embodiments of the present invention.

[0079] Fig.23 is a schematic diagram showing an application suite for generating a computer vision-based AR application according to one embodiment of the present invention.

[0080] Fig.24 and Fig.25 is a schematic diagram illustrating the relationship between an application specification, a state machine, and a computer vision-based processing algorithm of a computer vision-based AR application according to some embodiments of the present invention.

[0081] Fig.26 is a schematic diagram showing the initialization of a computer vision-based AR application according to one embodiment of the present invention.

[0082] Fig. 27 1 is a schematic diagram showing the operation (termination control) of the computer vision-based AR application according to one embodiment of the present invention.

[0083] Fig.28 is a schematic diagram showing a software screen installation operation based on screen mirroring or sharing performed by using a computer vision-based AR application according to an embodiment of the present invention.

[0084] Fig.29 is a diagram showing some pictures of an example software screen installation operation.

[0085] Fig.30 is a schematic diagram of a system for operating a computer vision-based AR application according to an embodiment of the present invention. DETAILED DESCRIPTION

[0086] Figure 1A system 100 for operating or facilitating the operation of a computer vision-based XR application in some embodiments of the present invention is shown. The system 100 includes a plurality of extended reality (XR) devices 20 and a computing system 10 (outside the XR device, therefore referred to as an external computing system) that are operably connected. Each of the XR device 20 and the external computing system 10 can communicate data directly or indirectly via one or more wired or wireless communication links. The external computing system 10 can be formed by one computing device (node) or multiple computing devices (cluster). The external computing system can be a cloud computing system. The computing device can include a server, a computer, etc. The computing system 10 can store or retrieve data that can be accessed and / or manipulated by the XR device 20, and can use its computing resources to host various processing engines used by the XR device 20 for operating computer vision-based XR applications or facilitating the operation of computer vision-based XR applications. The XR (AR, VR and / or MR) device 20 can be in the form of glasses, headphones, etc., which can be worn or carried by a user in a physical environment. The XR device 20 can be an existing XR device known in the art. Each XR device 20 may be installed with an XR (AR, VR and / or MR) application (APP) and may cooperate with the computing system 10 to provide an XR experience in a physical environment for the user of the device 20. The computer vision-based XR application may allow the user of the XR device 20 to access the data of the computing system 10, use the computing resources of the computing system 10 (e.g., the processing engine provided by it) to operate the computer vision-based XR application, etc.

[0087] Figure 2 The generation of an application suite 50 of a computer vision-based XR application according to an embodiment of the present invention is shown. The application suite 50 may include a set of data files that can be used to operate a computer vision-based XR application. In this embodiment, the generation of a computer vision-based XR application is performed by an XR application suite generation tool 200. The tool 200 can be implemented using suitable hardware and software. The tool 200 can be provided or implemented in the form of: a web-based application, a desktop application, a mobile application, etc. The tool 200 can be provided by a computing system separate from the computing system 10. Alternatively, the tool 200 can be provided at least in part by the computing system 10.

[0088] In the present embodiment, the tool 200 uses various data as input to generate the application suite 50. Specifically, in the present example, the tool 200 uses the following data as input: user experience and user interface design data 40A for a computer vision-based XR application, computer vision-based XR instruction data 40B for a computer vision-based XR application, data 40C associated with a storage location storing data associated with the computer vision-based XR application, data transmission control parameters 40D for operating the computer vision-based XR application, and data 40E associated with one or more characteristics of an XR device arranged to operate the computer vision-based XR application. The tool 200 can process the data 40A-40E to generate the application suite 50 for the computer vision-based XR application. The generated application suite 50 can be operably deployed (e.g., installed) at the XR device and / or at an external computing system operably connected to the XR device (which can be but not necessarily be a computer system). Figure 1 An XR device 20 and / or an external computing system 10) is used to operate computer vision-based XR applications.

[0089] For example, the user experience and user interface design data 40A may include audio data and / or visual data that is arranged to be presented by an XR device that is arranged to operate a computer vision-based XR application. The audio data and / or visual data may be associated with one or more user interfaces (UIs) (for visual data, graphical user interfaces GUIs) that are arranged to be displayed or presented to a user of the XR device. The visual data may include images, text, virtual objects, etc. associated with one or more GUIs, and may correspond to menus, instructions, information, commands, guidance, etc. that are arranged to be displayed to a user of the XR device. The audio data may include sounds, noises, music, voices, etc. to be played by the XR device to be presented to the user of the XR device. The visual data and / or audio data may also include data associated with one or more attributes of the visual data (e.g., rendering attributes, UI design-related parameters (e.g., font size, color, transparency), etc.), data associated with one or more attributes of the audio data (e.g., synthesis attributes, sound processing attributes, etc.), etc.

[0090] For example, the computer vision-based XR instruction data 40B may include data associated with computer vision-based object detection (e.g., data of a target object to be detected, a target object detection algorithm / criteria / logic, etc.), data associated with a computer vision-based object detection response (algorithm / logic associated with a response to be provided when a target object is detected by computer vision), etc. The data associated with the computer vision-based object detection response may include data associated with one or more conditions associated with computer vision-based object detection for affecting the appearance of a user interface, such as: data associated with the order of user interfaces; data associated with one or more conditions for activating one or more user interfaces (audio and / or visual), data associated with one or more conditions for switching from one user interface to another user interface (audio and / or visual), data associated with one or more conditions for deactivating one or more user interfaces (audio and / or visual), etc. Each user interface may correspond to a corresponding state.

[0091] For example, data 40C associated with a storage location storing data associated with a computer vision-based XR application may include data associated with a storage location at an external computing system and / or database and / or data associated with a computer vision-based processing algorithm storage location.

[0092] For example, the data transmission control parameter 40D for operating a computer vision-based XR application may be a data transmission control parameter between the XR device and the external computing system (e.g., it is associated with data transmission from the XR device to the external computing system). For example, the data transmission control parameter may include: the number of image frames per transmission from the XR device to the external computing system, the maximum image width for images transmitted from the XR device to the external computing system, the minimum image width for images transmitted from the XR device to the external computing system, the minimum frame rate for images transmitted from the XR device to the external computing system, the maximum frame rate for images transmitted from the XR device to the external computing system, etc.

[0093] For example, data 40E associated with one or more characteristics of an XR device arranged to operate a computer vision-based XR application may be associated with hardware limitations of the XR device arranged to operate a computer vision-based XR application, and may include a brand of the XR device, a model of the XR device, processing capabilities of the XR device, communication capabilities of the XR device, available memory storage of the XR device, power capacity of the XR device, etc.

[0094] like Figure 2As shown, in this embodiment, the application suite 50 of the computer vision-based XR application generated by the tool 200 using the above input includes: application specification data 50A, a state machine 50B, a computer vision-based processing algorithm 50C, and a data transmission strategy 50D. In this embodiment, the application suite 50 is arranged to be partially deployed at the XR device and partially deployed at the external computing system for operating the computer vision-based XR application.

[0095] For example, specification data 50A may include data (e.g., audio and / or visual) associated with different guidance and / or instructions that the computer vision-based XR application is arranged to provide. Figure 3 FIG. 5 shows the data structure of the specification data 50A of this embodiment. Figure 3 As shown, the specification data 50A may include multiple guidance identifiers (each of which corresponds to a respective guidance that the computer vision-based XR application is arranged to provide to a user of the XR device), wherein each guidance identifier is linked to associated visual and / or audio data (visual and / or audio for providing the guidance).

[0096] For example, state machine 50B may include data associated with user interface (audio and / or visual) sequences corresponding to different instructions and / or commands, as well as data for affecting the appearance of the user interface (e.g., generating a new user interface, changing an existing user interface, deactivating / deleting an existing user interface, etc.). The data for affecting the appearance of the user interface may include a target object identifier corresponding to a target object identified by the computer vision-based processing. The target object identifier may be linked to the instruction identifier so that the corresponding instruction may be triggered when the target object is detected. Figure 4 FIG. 5 shows the data structure of the state machine 50B of this embodiment. Figure 4 As shown, the state machine 50B includes a plurality of states, wherein each state includes a target object identifier, a corresponding guidance identifier, and information of a next state (if any).

[0097] For example, the computer vision-based processing algorithm 50C may include data associated with computer vision-based target object detection. The data associated with computer vision-based target object detection may include data of the target object and associated detection criteria. Figure 5 FIG. 5 shows the data structure of the processing algorithm 50C based on computer vision of this embodiment. Figure 5 As shown, the data structure of the computer vision-based processing algorithm 50C may include a plurality of object detections, wherein each object detection includes a target object identifier and a corresponding detection criterion.

[0098] For example, a data transfer strategy 50D for operating a computer vision-based XR application is arranged to regulate data communication between the XR device and an external computing system (particularly data communication from the XR device to the external computing system).

[0099] In this embodiment, the specification data 50A, the state machine 50B and the computer vision-based processing algorithm 50C are interrelated, such as Figures 3 to 5 The data structures shown (e.g., object identifiers in the state machine and in the computer vision-based processing algorithm; guidance identifiers in the application specification and in the state machine).

[0100] In this embodiment, the specification data 50A of the computer vision-based XR application is arranged to be deployed at the XR device, while the state machine 50B and the computer vision-based processing algorithm 50C of the computer vision-based XR application are arranged to be deployed at the external computing system for operating the computer vision-based XR application. The data transmission strategy 50D can be deployed or stored in the XR device, the external computing system, or distributed in both.

[0101] It should be understood that Figure 2 This is merely an example of application suite generation for computer vision-based XR applications. In some other embodiments, tool 200 may use one or some (not all) of data 40A-40E, and include or exclude other data, as input to generate application suite 50. In some other embodiments, application suite 50 generated by tool 200 may include only one or some of outputs 50A-50D, with or without other content. It should also be understood that Figures 3 to 5 This is merely an example data structure of specification data 50A, state machine 50B, and computer vision based processing algorithm 50C.

[0102] Figure 6 and Figure 7 Example methods 600 and 700 for initializing a computer vision-based XR application according to some embodiments of the present invention are shown. In these embodiments, the computer vision-based XR application is arranged to operate through the cooperation of an XR device and an external computing system, where the XR device and the external computing system may be (but not necessarily are) Figure 1 The XR device 20 and the external computing system 10. In some embodiments, the specification data of the computer vision-based XR application may have been stored in or loaded into the XR device. In some embodiments, initialization needs to be performed for each corresponding session of the computer vision-based XR application.

[0103] refer to Figure 6, the method 600 in this embodiment includes transmitting an identifier associated with a computer vision-based XR application from the XR device to the external computing system using the address of the external computing system in step 602. The transmission can be initiated by a user of the XR device logging into a related portal or application at the XR device. Then, in step 604, the identifier transmitted from the XR device is received by the external computing system (as specified by the address). Therefore, in step 606, the external computing system generates an instance (virtual machine) for the computer vision-based XR application. The generation of the instance of the computer vision-based XR application may include allocating computing resources based on the computer vision-based XR application to be initialized, and generating an instance for the computer vision-based XR application using the allocated computing resources. In some embodiments, the instance includes a state machine engine for operating a state machine (part of an application suite) of the computer vision-based XR application and a computer vision-based processing engine for operating a computer vision-based processing algorithm (part of an application suite) of the computer vision-based XR application. In step 608, the external computing system transmits access information for accessing the generated instance of the computer vision-based XR application to the XR device. In step 610, the XR device receives the access information. The XR device may store the access information locally for subsequent use. In some embodiments, the access information is only valid for one session (one generated instance): that is, if the session of the generated instance is terminated, then, to operate the computer vision-based XR application, a new instance needs to be generated and method 600 needs to be executed again to obtain updated access information.

[0104] refer to Figure 7, the method 700 in this embodiment includes receiving or retrieving configuration data at the XR device in step 702 (e.g., from a computing system (which may or may not be the external computing system shown)), the configuration data including an identifier associated with the computer vision-based XR application and an address of the external computing system. The receiving or retrieving may be initiated by a user of the XR device logging into a related portal or application at the XR device. Then, in step 704, the XR device transmits the identifier associated with the computer vision-based XR application to the external computing system using the address of the external computing system. In step 706, the identifier transmitted from the XR device is received by the external computing system (as specified by the address). Therefore, in step 708, the external computing system determines or obtains (e.g., from a computing system (which may not be part of the external computing system and may or may not be the computing system providing the tool 200)) computing resource settings (e.g., predefined) for an instance (virtual machine) of the computer vision-based XR application based on the identifier, and in step 710, generates an instance of the computer vision-based XR application. In some embodiments, the instance includes a state machine engine for operating a state machine (part of an application suite) for operating a computer vision-based XR application and a computer vision-based processing engine (part of an application suite) for operating a computer vision-based XR application. In step 712, the external computing system transmits access information for accessing the generated instance of the computer vision-based XR application to the XR device. In step 714, the XR device receives the access information. The XR device may store the access information locally for subsequent use. In some embodiments, the access information is valid only for one session of the generated instance: that is, if the session of the generated instance is terminated, then, to operate the computer vision-based XR application, a new instance needs to be generated and method 700 needs to be executed again to obtain updated access information.

[0105] Figure 8 An example method 800 for operating a computer vision-based XR application in some embodiments of the present invention is shown. In these embodiments, the computer vision-based XR application is arranged to be operated through the cooperation of an XR device and an external computing system, which can be (but not necessarily) Figure 1 The XR device 20 and the external computing system 10 are preferably used only when the method 800 is initialized (e.g. Figure 6 and Figure 7 Methods 600, 700) in the above method are executed after initializing the computer vision-based XR application.

[0106] refer to Figure 8, the method 800 in this embodiment includes acquiring an image or image data at the XR device in step 802. The acquisition of the image may be capturing the image using an image sensor of the XR device, or may be receiving the image or image data from another device or system operably connected to the XR device. In step 804, the XR device transmits the image or image data to an external computing system that has generated or is generating an instance for a computer vision-based XR application (e.g., it has a state machine engine for a computer vision-based XR application and a computer vision-based processing engine). The external computing system receives the image or image data in step 806, and in step 808, processes the image or image data using computer vision-based processing (provided by the computer vision-based processing engine) to obtain a processing result. The external computing system transmits data associated with the processing result to the XR device in step 810. The data associated with the processing result may include one or more of the following: processing results, responses to be implemented at the XR device, etc. The XR device receives data associated with the processing results in step 812, processes the received data in step 814, and implements a response based on the processing of the data in step 816. In some implementations, application specification data for the computer vision-based XR application may have been stored or loaded into the XR device to assist in implementing the response at the XR device.

[0107] Fig. 9 An example method 900 for operating a computer vision-based XR application according to one embodiment of the present invention is shown. Fig. 9 The method 900 can be considered as Figure 8 A specific implementation of method 800 is provided for performing object detection-triggered graphic changes and object detection-triggered visual / auditory guidance associated with a task or action (for assisting a user of an XR device in completing the task or operation).

[0108] refer to Fig. 9, the method 900 in this embodiment includes, in step 902, capturing an image of an environment in which the XR device is arranged using an image sensor of the XR device. In step 904, the XR device transmits the corresponding image or image data to an external computing system that has generated or is generating an instance of a computer vision-based XR application (e.g., it has a state machine engine and a computer vision-based processing engine for the computer vision-based XR application). The external computing system receives the image or image data in step 906, and processes the image or image data using computer vision-based processing (provided by the computer vision-based processing engine) in step 908 to detect the presence of one or more target objects in the image and / or one or more attributes of one or more target objects in the image. The external computing system transmits data associated with the target object detection result to the XR device in step 910. The data associated with the target object detection result may include one or more of the following: the target object detection result, the response to be implemented by the XR device (e.g., a guidance identifier), etc. The XR device receives data associated with the target object detection results in step 912, determines a visual and / or audio response to be provided to a user of the XR device based on the detected target object in step 914 (e.g., using application specification data), and provides a visual and / or audio response to the user of the XR device using appropriate visual and / or audio components of the XR device in step 916.

[0109] Fig.10 An example method 1000 for operating a computer vision-based XR application according to one embodiment of the present invention is shown. Fig.10 The method 1000 can be considered as Figure 8 A specific implementation of method 800 is provided for performing object detection-triggered graphic changes and object detection-triggered visual / auditory guidance associated with a task or action (for assisting a user of an XR device in completing a task or operation).

[0110] refer to Fig.10, the method 1000 in this embodiment includes, in step 1002, capturing an image of an environment in which the XR device is arranged using an image sensor of the XR device. In step 1004, the XR device transmits the corresponding image or image data to an external computing system that has generated or is generating an instance of a computer vision-based XR application (e.g., it has a state machine engine and a computer vision-based processing engine for the computer vision-based XR application). The external computing system receives the image or image data in step 1006 and processes the image or image data using computer vision-based processing (provided by the computer vision-based processing engine) in step 1008 to detect the presence of one or more target objects in the image and / or one or more attributes of one or more target objects in the image. The external computing system determines a visual and / or audio response to be provided to the user of the XR device based on the detected target object in step 1010, and transmits data associated with the corresponding visual and / or audio response to the XR device in step 1012. The XR device receives data associated with corresponding visual and / or audio responses in step 1014 , processes it in step 1016 , and provides the visual and / or audio responses to a user of the XR device in step 1018 .

[0111] Fig.11 An example method 1100 for operating a computer vision-based XR application according to one embodiment of the present invention is shown. Fig.11 The method 1100 can be viewed as Figure 8 A specific implementation of method 800 is provided, which is used to execute software screen installation guidance associated with the installation task of IT equipment (used to assist the user of the XR device to complete the installation task of IT equipment).

[0112] refer to Fig.11, the method 1100 in this embodiment includes receiving, at the XR device, an image associated with screen content of a screen mirroring or screen sharing device in step 1102. To this end, the XR device has established a data connection with the screen mirroring or screen sharing device (computing device) (e.g., using a screen mirroring or screen sharing application). The screen mirroring or screen sharing device may be a computing device in the field of view of an image sensor of the XR device (and not imaged using the image sensor of the XR device). The screen mirroring or screen sharing device may be a computing device on which a user of the XR device wants to perform an installation (e.g., software installation) task. In step 1104, the XR device transmits the corresponding image or image data to an external computing system that has generated or is generating an instance of a computer vision-based XR application (e.g., it has a state machine engine for a computer vision-based XR application and a computer vision-based processing engine). The external computing system receives the image or image data in step 1106, and processes the image or image data using computer vision-based processing (provided by a computer vision-based processing engine) in step 1108 to detect the presence of one or more target objects in the image and / or one or more attributes of one or more target objects in the image. In step 1110, the external computing system transmits data associated with the target object detection result to the XR device. The data associated with the target object detection result may include one or more of the following: the target object detection result, a response to be implemented by the XR device (e.g., a guidance identifier), etc. The XR device receives the data associated with the target object detection result in step 1112, determines a visual and / or audio response to be provided to a user of the XR device based on the detected target object in step 1114 (e.g., using application specification data), and provides the visual and / or audio response to the user of the XR device using appropriate visual and / or audio components of the XR device in step 1116. The visual / audio response provided to the user of the XR device can assist the user in installing software in a screen mirroring or screen sharing device.

[0113] Fig.12 An example method 1200 for operating a computer vision-based XR application according to one embodiment of the present invention is shown. Fig.12 The method 1200 can be seen as Figure 8 A specific implementation of method 800 is provided, which is used to execute software screen installation guidance associated with the installation task of IT equipment (used to assist the user of the XR device to complete the installation task of IT equipment).

[0114] refer to Fig.12, the method 1200 in this embodiment includes receiving, at step 1202, an image associated with screen content of a screen mirroring or screen sharing device at the XR device. To this end, the XR device has established a data connection with the screen mirroring or screen sharing device (computing device) (e.g., using a screen mirroring or screen sharing application). The screen mirroring or screen sharing device may be a computing device in the field of view of an image sensor of the XR device (and not imaged using the image sensor of the XR device). The screen mirroring or screen sharing device may be a computing device on which a user of the XR device wants to perform an installation (e.g., software installation) task. In step 1204, the XR device transmits the corresponding image or image data to an external computing system that has generated or is generating an instance of a computer vision-based XR application (e.g., it has a state machine engine for a computer vision-based XR application and a computer vision-based processing engine). The external computing system receives the image or image data in step 1206 and processes the image or image data using computer vision-based processing (provided by a computer vision-based processing engine) to detect the presence of one or more target objects in the image and / or one or more attributes of one or more target objects in the image in step 1208. The external computing system determines a visual and / or audio response to be provided to a user of the XR device based on the detected target objects in step 1210, and transmits data associated with the corresponding visual and / or audio response to the XR device in step 1212. The XR device receives the data associated with the corresponding visual and / or audio response in step 1214, processes it in step 1216, and provides the visual and / or audio response to the user of the XR device in step 1218. The visual / audio response provided to the user of the XR device can assist the user in installing software in the screen mirroring or screen sharing device.

[0115] Fig.13 Some exemplary possible operations in providing a visual and / or audio response to a user of an XR device in step 916 of method 900, step 1018 of method 1000, step 1116 of method 1100, and step 1218 of method 1200 are shown. Fig.13As shown, this step may include providing a graphical user interface to the user of the XR device (providing new visual content), changing the graphical user interface being provided to the user of the XR device (changing some visual content), terminating the graphical user interface being provided to the user of the XR device, providing visual instructions or information (in the form of text, images, symbols, etc.) to the user of the XR device, and providing audio instructions or information (in the form of music, commands, voice, etc.) to the user of the XR device. The visual and / or audio content provided will depend on the purpose of the application (e.g., software screen installation, training installation, inspection, maintenance, etc.) and may be task or operation specific.

[0116] Fig.14 An example method 1400 for operating a computer vision-based XR application in some embodiments of the present invention is shown. In these embodiments, the computer vision-based XR application is arranged to be operated through the cooperation of an XR device and an external computing system, which can be (but not necessarily) Figure 1 The XR device 20 and the external computing system 10 are preferably used only when the method 1400 is initialized using an initialization method (e.g. Figure 6 and Figure 7 Methods 600, 700) in the above method are executed after initializing the computer vision-based XR application.

[0117] refer to Fig.14, the method 1400 in this embodiment includes determining, in step 1402, the quality or quality change of the communication link between the XR device and the external computing system and / or the processing performance or the change of the processing performance of the external computing system. In this embodiment, the quality and / or the processing performance (or its / their change) is measured substantially in real time. In some other embodiments, the quality and / or the processing performance (or its / their change) may be a future predicted quality and / or predicted processing performance (or its / their change). For example, the change of the quality or processing performance of the external computing system may involve one or more of the following: the delay of one or more communication links between the XR device and the external computing system (instantaneous, cumulative, average, etc., optionally weighted), the speed of one or more communication links between the XR device and the external computing system (instantaneous, cumulative, average, etc., optionally weighted), the bandwidth of one or more communication links between the XR device and the external computing system (instantaneous, cumulative, average, etc., optionally weighted), the packet loss rate or number of packet losses between the XR device and the external computing system (instantaneous, cumulative, average, etc., optionally weighted). For example, the quality or variation of the processing performance of the external computing system relates to one or more of: the processing speed of the external computing system (instantaneous, cumulative, average, etc., optionally weighted), the processing throughput of the external computing system (instantaneous, cumulative, average, etc., optionally weighted), etc. In some embodiments, the communication quality and / or processing performance of the external computing system is determined based on one or more of: the number of images transmitted from the XR device within a period of time, the number of images received by the external computing system within a period of time, the amount of data associated with the number of processing results generated by the external computing system within a period of time, the amount of data associated with the number of processing results transmitted from the external computing system within a period of time, and the amount of data associated with the number of processing results received at the XR device within a period of time.

[0118] After step 1402, the method proceeds to step 1404, which includes determining, based on the determined quality and / or processing performance, according to the data transmission policy, a resolution and / or frame rate (which relates to the size of the image data) for images transmitted from the XR device to the external computing system. For example, the data transmission policy may define one or more of the following transmission control parameters: the number of image frames per transmission from the XR device to the external computing system, the maximum image width for images transmitted from the XR device to the external computing system, the minimum image width for images transmitted from the XR device to the external computing system, the minimum frame rate for images transmitted from the XR device to the external computing system, the maximum frame rate for images transmitted from the XR device to the external computing system, etc.

[0119] The determinations in steps 1402 and 1404 may be performed at the XR device, at the external computing system, and / or across both.

[0120] In step 1406, the XR device transmits the image (image data) to the external computing system using the determined resolution and frame rate. The transmitting step may correspond to the transmitting steps 804, 904, 1004, 1104, 1204 described above. In step 1408, the image (image data) is received at the external computing system. The receiving step may correspond to the receiving steps 806, 906, 1006, 1106, 1206 described above.

[0121] For example, if the quality of the communication link is at a first quality level, the transmission includes transmitting the image at a first resolution and / or a first frame rate, and if the quality of the communication link is at a second quality level lower than the first quality level, the transmission includes transmitting the image using a second resolution lower than the first resolution and / or a second frame rate lower than the first frame rate.

[0122] For example, if the quality or change in quality of the communication and / or the processing performance or change in the processing performance of the external computing system is determined to be degraded, then: the frame rate associated with the image for transmission to the external computing system is reduced (if the frame rate has not reached the minimum frame rate), the image resolution for transmission to the external computing system is reduced (if the frame rate has reached the minimum frame rate and the resolution has not reached the minimum resolution), and error information is provided at the XR device (if the frame rate has reached the minimum frame rate and the resolution has reached the minimum resolution).

[0123] For example, if the quality or change in quality of the communication and / or the processing performance or change in the processing performance of the external computing system is determined to have increased, then: the resolution of the image for transmission to the external computing system is increased (if the resolution has not reached the maximum resolution), and the frame rate associated with the image for transmission to the external computing system is increased (if the resolution has reached the maximum resolution and the frame rate has not reached the maximum frame rate).

[0124] Figures 15 to 17 Example methods 1500 - 1700 for operating a computer vision-based XR application in one embodiment of the invention are shown. Figure 15 to Figure 17 The method 1500-1700 can be considered as Fig.14 A specific example implementation of method 1400 regarding data transmission control.

[0125] refer to Fig.15In one embodiment, in method 1500, in step 1502, the XR device transmits images (image data) (e.g., one image at a time or multiple images at a time) to the external computing system for processing at a set resolution and frame rate via one or more communication links between the XR device and the external computing system. The transmitting step 1502 may correspond to the transmitting steps 804, 904, 1004, 1104, 1204, 1406 described above.

[0126] In step 1504, the system (XR device and / or external computing system) determines whether the communication quality (provided by one or more communication links) and / or the processing performance of the external computing system can support image transmission at the current resolution and frame rate.

[0127] If it is determined that image transmission at the current resolution and frame rate is supported, then in step 1506, method 1500 determines whether the current resolution and frame rate are both set to maximum values ​​(e.g., as permitted by the data transmission policy). If it is determined that the current resolution and frame rate are both set to maximum values, the system determines that data transmission or communication between the XR device and the external processing system has reached a state of equilibrium and no changes are required in this regard. Method 1500 may then return to step 1502 or 1504. If it is determined that either of the current resolution and frame rate is not set to maximum values, the system determines that data transmission or communication between the XR device and the external processing system is in a stable (balanced) state, and will attempt to restore a state of equilibrium by adjusting the current resolution and / or frame rate.

[0128] On the other hand, if it is determined that image transmission at the current resolution and frame rate is not supported, then in step 1508, method 1500 determines whether the current resolution and frame rate are both set to maximum values ​​(e.g., as allowed by the data transmission policy). If it is determined that the current resolution and frame rate are both set to maximum values, the system determines that the data transmission or communication between the XR device and the external processing system has not reached a stable state and needs to reduce the resolution and / or frame rate. If it is determined that either of the current resolution and frame rate is not set to the maximum value, the system determines that the data transmission or communication between the XR device and the external processing system has not reached a stable state and needs to reduce the resolution and / or frame rate.

[0129] Reference now Fig.16, method 1600 in this embodiment focuses on a method of operating a computer vision-based XR application when an equilibrium state or a stable state has not been reached (e.g., as determined in step 1508). In this embodiment, method 1600 begins with step 1602 of transmitting an image (image data) and step 1604 of determining whether the communication quality (provided by one or more communication links) and / or the processing performance of the external computing system can support the transmitted image at the current resolution and frame rate. Steps 1602 and 1604 are generally related to Fig.15 Steps 1502 and 1504 in are the same, so they are not further described here. After the determination of step 1604, in step 1606, it is determined that a steady state or equilibrium state has not been reached. Then, in step 1608, the system determines whether the current frame rate is at a minimum frame rate (e.g., as allowed by the data transmission policy). If it is determined that the current frame rate is not at the minimum frame rate, the method 1600 proceeds to step 1610 to reduce the frame rate and then returns to step 1602 to transmit the image using the reduced frame rate (and the same resolution). If it is determined in step 1608 that the current frame rate is already at the minimum frame rate, the method 1600 proceeds to step 1612 to determine whether the current resolution is at the minimum resolution (e.g., as allowed by the data transmission policy). If it is determined that the current resolution is not at the minimum resolution, the method 1600 proceeds to step 1614 to reduce the resolution and then returns to step 1602 to transmit the image using the reduced resolution (and the minimum frame rate). If it is determined that the current resolution is already at the minimum resolution (and the frame rate is at the minimum frame rate), method 1600 continues to provide an error message, warning, or related indication to the user of the XR device at the XR device. In some examples, the computer vision-based XR application can terminate in response to the error message.

[0130] Reference now Fig.17 , method 1700 in this embodiment focuses on a method of operating a computer vision-based XR application when a steady state (which may or may not have reached an equilibrium state) is reached (e.g., as determined in step 1506). In this embodiment, method 1700 begins with step 1702 of transmitting an image (image data) and step 1704 of determining whether the quality of the communication (provided by one or more communication links) and / or the processing performance of the external computing system can support the transmitted image at the current resolution and frame rate. Steps 1702 and 1704 are generally related to Fig.15Steps 1502 and 1504 in are the same, and therefore are not further described here. After the determination of step 1704, in step 1706, it is determined that at least a steady state has been reached. Then, in step 1708, the system determines whether the current resolution is at a maximum resolution (e.g., as permitted by the data transmission policy). If it is determined that the current resolution is not the maximum resolution, the method 1700 proceeds to step 1710 to increase the resolution and then returns to step 1702 to transmit the image using the increased resolution (and the same frame rate). If it is determined in step 1708 that the current resolution is already at the maximum resolution, the method 1700 proceeds to step 1712 to determine whether the current frame rate is at the maximum frame rate (e.g., as permitted by the data transmission policy). If it is determined that the current frame rate is not at the maximum frame rate, the method 1700 proceeds to step 1714 to increase the frame rate and then returns to step 1702 to transmit the image using the increased frame rate (and the maximum resolution). If it is determined that the current frame rate is already at the maximum frame rate (and the resolution is at the maximum resolution), the method 1700 determines that an equilibrium state is reached in step 1716. The method 1700 returns to step 1702 to transmit the image using the maximum frame rate and the maximum resolution.

[0131] Fig.18 An example method 1800 for operating a computer vision-based XR application in some embodiments of the present invention is shown. In these embodiments, the computer vision-based XR application is arranged to be operated through the cooperation of an XR device and an external computing system, which can be (but not necessarily) Figure 1 The XR device 20 and the external computing system 10 are preferably used only when the method 1800 is used in an initialization method (e.g. Figure 6 and Figure 7 Methods 600, 700) in the above method are executed after initializing the computer vision-based XR application.

[0132] refer to Fig.18 In this embodiment, the method 1800 includes, in step 1802, an external computing system operating an instance of a computer vision-based XR application for the XR device transmits a prompt signal to the XR device. In step 1804, the XR device receives the prompt signal, and in response, in step 1806, transmits a keep alive signal to the external computing system. In step 1808, the external computing system receives the keep alive signal, and in response, determines that the XR device is still or may still be using the generated instance, and therefore, in step 1810, maintains the operation of the instance (without releasing the associated computing resources).

[0133] Fig.19An example method 1900 for operating a computer vision-based XR application in some embodiments of the present invention is shown. In these embodiments, the computer vision-based XR application is arranged to be operated through the cooperation of an XR device and an external computing system, which can be (but not necessarily) Figure 1 Preferably, method 1900 is performed only when using an initialization method (e.g. Figure 6 and Figure 7 The method 1900 is executed after the computer vision-based XR application is initialized by the method 600 and 700 in the embodiment. In the present embodiment, the method 1900 is executed at an external computing system, which is operating an instance of a computer vision-based XR application for an XR device.

[0134] refer to Fig.19 , the method 1900 in this embodiment includes, in step 1902, an external computing system operating an instance of a computer vision-based XR application for the XR device transmitting a prompt signal to the XR device. Then, in step 1904, the external computing system determines whether any keep-alive signal is received as a response from the XR device (e.g., within a time window). If the external computing system determines that a keep-alive signal is received as a response from the XR device, the external computing system determines that the XR device is still or may still be using the generated instance, and therefore maintains the operation of the instance (without releasing the associated computing resources) in step 1906. If the external computing system determines that no keep-alive signal is received as a response from the XR device, it increases its counter by 1 (or any arbitrary value) in step 1908. In step 1910, the external computing system determines whether the counter exceeds (or in some examples reaches) a predetermined count. If the external computing system determines that the counter has not exceeded (or in some examples has not reached) the predetermined count, the external computing system returns to step 1902 to retry transmitting the prompt signal to the XR device. However, if the external computing system determines that the counter has exceeded (or in some examples has reached) the predetermined count, the external computing system determines that the XR device will no longer use the generated instance, and therefore terminates operation of the instance and releases associated computing resources in step 1912. The external computing system can then use the released computing resources for other purposes (e.g., generating other instances for the same or other computer vision-assisted XR applications in other XR devices).

[0135] Fig. 20An example method 2000 for operating a computer vision-based XR application according to some embodiments of the present invention is shown. The method 2000 involves terminating the operation of the computer vision-based XR application. In these embodiments, the computer vision-based XR application is arranged to operate through the cooperation of an XR device and an external computing system, which may be (but not necessarily be) Figure 1 The XR device 20 and the external computing system 10. Preferably, the method 2000 is performed only when using an initialization method (e.g. Figure 6 and Figure 7 Methods 600, 700) in the above method are executed after initializing the computer vision-based XR application.

[0136] refer to Fig. 20 , the method 2000 in this embodiment includes, in step 2002, the XR device determines that the operation of the computer vision-based XR application is to be terminated or has been terminated. The determination in step 2002 can be based on user input received at the XR device, the operating state of the XR device, the power level of the XR device, etc. In step 2004, the XR device then transmits a termination signal associated with the determination (i.e., terminating the operation of the computer vision-based XR application) to the external computing system. The external computing system operating an instance of the computer vision-based XR application receives the termination signal in step 2006, and then terminates the instance in step 2008 to release computing resources associated with the instance.

[0137] Fig.21 A system of some embodiments of the present invention is shown, which includes a plurality of XR devices 20, an external computing system 10 and a (further) computing system 40. The XR devices may be Figure 1 The external computing system 10 may be Figure 1 The computing system 40 may be arranged to store data associated with resource settings of a computer vision-based XR application (e.g., resources required to generate an associated instance in an external computing system, data transfer control policies, etc.). This data in the system 40 may be accessed by the external computing system 10 for generating an instance. The computing system 40 may further be arranged to provide or implement an AR kit generation tool (such as tool 200).

[0138] Fig.21 The system in can be used to implement any of the above-mentioned embodiments of the present invention. Fig.21In an embodiment, the external computing system 10 may include means for performing or facilitating the execution of the above method examples. The external computing system 10 may be provided by one or more computing devices. The external computing system 10 may be arranged to generate multiple instances for multiple computer vision-based XR applications and / or multiple XR devices. The external computing system 10 may include a control and / or management module for managing instances (particularly for managing their generation, operation, and termination). The external computing system 10 may be extended to support the operation of different numbers of computer vision-based XR applications and / or XR devices. The computing system 40 may be provided by one or more computing devices. The computing system 40 may be arranged to provide a computer vision-based AR application kit generation tool for generating an application kit of a computer vision-based AR application, and a repository for storing data associated with various computer vision-based AR applications (e.g., different application-specific data sets, common data, etc.). In some embodiments, the systems 10, 40 may be at least partially integrated or combined together.

[0139] Fig. 22 22 is an example information processing system 2200 of some embodiments of the present invention, which can be used to at least partially implement one or more methods of the present invention.

[0140] In some embodiments of the present invention, the information processing system 2200 may be used as a server, a database, or other types of information processing systems. For example, the information processing system 2200 may be used to at least partially implement Figure 2 The XR application kit generation tool 200 in . For example, the information processing system 2200 can be used to implement at least in part any of the external computing systems (e.g., system 10) or other computing systems (e.g., system 40) of any of the embodiments shown or described. For example, the information processing system 2200 can be used to at least in part implement an XR device. For example, the information processing system 2200 can be used to perform or facilitate the performance of any of the method embodiments shown or described.

[0141] In these embodiments, the information processing system 2200 generally includes appropriate components required to receive, store and execute appropriate computer instructions, commands and / or codes. The main components of the information processing system 2200 are the processor 2202 and the memory 2204. The processor 2202 may include one or more of the following: one or more CPUs, one or more MCUs, one or more logic circuits, one or more Raspberry Pi chips, one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs) and / or any one or more digital or analog circuits / circuits configured to interpret and / or execute program instructions and / or process signals and / or information and / or data. The memory 2204 may include: one or more volatile memories (e.g., RAM, DRAM, SRAM), one or more non-volatile memories (e.g., ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM), or any combination thereof. Appropriate computer instructions, commands, codes, information and / or data may be stored in the memory 2204. Computer instructions for executing or facilitating the execution of one or more method embodiments of the present invention (including one or more of the above methods) may be stored in the memory 2204. The processor 2202 and the memory 2204 may be integrated or separate (and operably connected). Optionally, the information processing system 2200 also includes one or more input devices 2206. Examples of input devices 2206 include: keyboard, mouse, stylus, image scanner, microphone, tactile / touch input device (e.g., touch-sensitive screen), image / video input device (e.g., camera), etc. Optionally, the information processing system 2200 also includes one or more output devices 2208. Examples of output devices 2208 include: display (e.g., monitor, screen, projector, etc.), speaker, earphone, headset, printer, additive manufacturing machine (e.g., 3D printer), etc. The display may include an LCD display, an LED / OLED display, or other suitable display, which may or may not be touch-sensitive. The information processing system 2200 may also include one or more disk drives 2212, which may include one or more of the following: solid-state drive, hard disk drive, optical drive, flash drive, tape drive, etc. A suitable operating system may be installed in the data processing system 2200, for example, in the disk drive 2212 or the memory 2204. The memory 2204 and the disk drive 2212 may be operated by the processor 2202. Optionally, the information processing system 2200 further includes a communication device 2210 for communicating with a server, a personal computer, a terminal, a tablet computer, a phone, a watch, an Internet of Things (IoT) device, or other wireless computing devices.The communication device 2210 may include one or more of the following: a modem, a network interface card (NIC), an integrated network interface, an NFC transceiver, a ZigBee transceiver, a Wi-Fi transceiver, a Bluetooth transceiver, a radio frequency transceiver, a cellular (2G, 3G, 4G, 5G, 5G above, etc.) transceiver, an optical port, an infrared port, a USB connection or other wired or wireless communication interface. The transceiver can be implemented by one or more devices (integrated transmitter and receiver, separate transmitter and receiver, etc.). The communication link can be wired or wireless, which is used to transmit commands, instructions, information and / or data. In one example, the processor 2202, the memory 2204 (and optional input device 2206, output device 2208, communication device 2210 and disk drive 2212, if present) are directly or indirectly connected to each other through a bus, a peripheral component interconnect (PCI) such as PCI Express, a universal serial bus (USB), an optical bus or other similar bus structures. In one embodiment, at least some of these components may be connected wirelessly, for example, via a network such as the Internet or a cloud computing network. Those skilled in the art will appreciate that. Fig. 22 The information processing system 2200 shown in FIG. 2 is merely an example and in other implementations the information processing system 2200 may have a different configuration (eg, include additional components, have fewer components, etc.).

[0142] Although not required, the embodiments described with reference to the accompanying drawings may be implemented as an application programming interface (API) or a series of libraries for use by developers or may be included in another software application (the other software application operates in, for example, a terminal or computer operating system or a portable computing device operating system). Generally, since program modules include routines, programs, objects, components, and data files that help perform specific functions, those skilled in the art will understand that the functionality of a software application may be distributed among multiple routines, objects, and / or components to achieve the same functionality described herein.

[0143] Fig.23 An application suite for generating a computer vision-based AR application according to an embodiment of the present invention is shown. Fig.23 The generation shown in can be viewed as Figure 2 A more specific implementation of the generation in (see above for detailed description). Fig.23 In the example, the conversion unit generally corresponds to Figure 2 The XR application kit generation tool 200 in FIG. 1 is similar to the XR application kit generation tool 200 in FIG. 1 , but is targeted at AR applications. Fig.23As shown, the conversion unit utilizes designer input and developer input (and optional resource settings) as input to an application suite for generating computer vision-based AR applications. In this example, the designer input may include user experience and user interface design data and computer vision-based XR instruction data, which are arranged for specific purposes (AR instructions, graphical user interface changes triggered by object detection, data associated with software screen installation applications, etc.). In this example, the developer input may include transmission control parameters, computer vision-based processing algorithm storage location data, etc. The designer and developer may be different people or may be the same person. The resource settings used as input (for example, which may be stored in a server / database) may include one or more properties of an AR device for operating a computer vision-based AR application. The conversion unit processes the above inputs and provides as output: application specifications, state machines, DNN plug-ins (computer vision-based processing algorithms), resource settings (data transmission strategies, data resources for generating instances, etc.) (similar to Figure 2 In the present embodiment, the application specification is arranged to be loaded into the AR device, and the state machine and DNN plugin are arranged to be loaded into or operated on an external computing system.

[0144] Fig.24 and Fig.25 It is a schematic diagram of the relationship between an application specification, a state machine, and a computer vision-based processing algorithm of a computer vision-based AR application according to some embodiments of the present invention.

[0145] Fig.26 FIG. 1 is a schematic diagram of initializing an AR application based on computer vision according to an embodiment of the present invention. Fig.26 In the example, “AMS” may correspond to a server, “App” may refer to a computer vision-based AR application installed at the AR device, “cluster” may refer to an external computing system, “ARCVS receiver” may correspond to a control / management module of the external computing system, and “ARDVS” may correspond to a server.

[0146] In this example, some configuration steps need to be performed before the system can be used. First, the URL of the ARCVS receiver (in configuration file 1) needs to be read during the startup of the AMS. In addition, the cluster properties (e.g., nodes per pod, mount path for pods, etc., in configuration file 2) need to be provided to the ARCVS receiver during the startup of the ARCVS receiver. Resource settings such as the DNN plugin image name need to be saved through the ARDVS user interface. If all of these settings are configured correctly, the system can be used, for example, by executing Fig.26 Operations on the sequence of numbers shown in .

[0147] Simply put, in this initialization example, the URL of the ARCVS receiver of the cluster and the application identifier of the computer vision-based AR application to be operated are obtained at the AR device. The AR device loads or stores application specification data of the computer vision-based AR application. The AR device transmits the application identifier to the specified ARCVS receiver. Based on the identifier, the ARCVS receiver obtains the relevant resource settings associated with the computer vision-based AR application from ARDVS. Then, based on the obtained resource settings, the ARCVS receiver generates an instance of the computer vision-based AR application for the AR device. The instance includes a state machine engine and a DNN plug-in (computer vision-based object detection engine) for the computer vision-based AR application of the AR device. The ARCVS receiver returns the path of the generated instance to the AR device. The AR device uses the path of the generated instance and the URL of the ARCVS receiver to subsequently interact with the generated instance in the cluster.

[0148] Fig. 27 It is a schematic diagram of the termination control operation of a computer vision-based AR application according to an embodiment of the present invention. In one example, when the end user of the AR device intends to exit the computer vision-based AR application at the AR device, the AR device will transmit an "ARCVS stop" command to the ARCVS receiver. In response to receiving the stop command, the ARCVS receiver terminates ("kills") the instance and releases the associated computing resources. In one example, in order to handle the "abnormal exit of the application", the ARCVS receiver and the computer vision-based AR application will operate a keep-alive mechanism. In one example of a keep-alive mechanism, the ARCVS receiver transmits a low-level detection / handshake message (low-power signal) to the AR device (computer vision-based AR application) to see the response of the AR device (computer vision-based AR application) (e.g., by returning a keep-alive signal). If the AR device (computer vision-based AR application) does not respond (for example, the ARCVS receiver does not receive any keep-alive signal after a certain number of detection signals are transmitted from the ARCVS receiver for a period of time), the ARCVS receiver will determine that the connection with the AR device (computer vision-based AR application) is dead, and in response, terminate the instance and release all corresponding resources (for example, for use by other AR devices or instances).

[0149] Fig.28A software screen installation operation based on screen mirroring or sharing using a computer vision-based AR application according to one embodiment of the present invention is shown. In this example, a screen sharing front end (a screen sharing or screen mirroring device, such as a computing device) provides a GUI for a user to log in and start screen sharing. When a user logs in through the GUI of the screen sharing front end, the user ID and password are passed to the screen sharing back end, which communicates with the server "AMS" for user authentication. In one example, the screen sharing back end communicates with the screen sharing front end and the computer vision-based AR application (installed in the AR device) through a WebSocket channel, which monitors the online status of the screen sharing front end and the AR device (computer vision-based AR application) and exchanges messages during the initial setup of the real-time peer connection. After the connection is completed, the screen sharing front end transmits an image of its screen content to the AR device (computer vision-based AR application), and the AR device in turn transmits or forwards the image or image data to ARCVs for processing.

[0150] In some embodiments of the present invention, a software screen installation operation based on screen mirroring or sharing (the shared or mirrored screen is the screen in the field of view of the AR device user) performed using a computer vision-based AR application can solve the problem of flickering of the computer screen when it is imaged by the image sensor of the AR device (such as AR glasses).

[0151] Fig.29 is a diagram that includes some pictures of an example software installation screen.

[0152] Fig.30 is a schematic diagram of a system 3000 for operating a computer vision-based AR application according to an embodiment of the present invention. The system 3000 in this embodiment can be considered as Fig.21 A more detailed implementation of the system in . Fig.21 Similar to the system in FIG. 30, the system 3000 includes an external computing system 3002 (corresponding to Fig.21 The external computing system 10 in the AR device 3004 (corresponding to Fig.21 ), and another computing system 3006 (corresponding to Fig.21 computing system 40).

[0153] In this example, the external computing system 3002 includes a control device having a receiver (control / management) module, which generally corresponds to Fig.26 and Fig. 27The ARCVS receiver module in the system 3002 is used to control / manage the operation of an instance of a computer vision-based AR application at the system 3002 (for one or more AR devices). The external computing system 3002 also includes a computing device, which is arranged to generate and operate an instance of a computer vision-based AR application. The computing device includes a state machine module (engine) and a DNN module (computer vision-based processing algorithm (engine)). The control device and computing device of the system 3002 are provided by one or more processors.

[0154] In this example, the AR device 3004 includes one or more processors that provide a login module for accessing the system 3002, a data visualization module, a rendering / compositing module, a communication module for communicating with the system 3002, and a human-machine interface (HMI) module. The AR device 3004 also includes a memory (internal or external), a power source (internal or external), an image sensor, and an AR display system.

[0155] In this example, the computing system 3006 includes one or more processors that provide a user interface module for receiving user input to generate an application suite of computer vision-based AR applications, an application generation module for generating the computer vision-based AR application suite, an application database for storing data associated with the generated computer vision-based AR applications, and a management module for managing the operation of the computer vision-based AR applications at the AR device.

[0156] In this example, during use, an application suite of a computer vision-based AR application is first generated using the application generation module of the system 3006 , and then the data of the application suite is stored in an application database of the system 3006 .

[0157] In order to use the generated computer vision-based AR application at the AR device 3004, the user of the AR device first logs in to the computer vision-based AR application using the login module. Upon or after successful login, the system 3006 will provide a unique identifier (specific to) the computer vision-based AR application and a URL of the receiver (control / management) module in the system 3002. The system 3006 also provides application content (e.g., application specification data) (specific to) the computer vision-based AR application to the data visualization module of the AR device 3004. The AR device 3004 provides the unique identifier to the receiver (control / management) module in the system 3002 using the URL. After receiving the unique identifier, the receiving (control / management) module in the system 3002 obtains or receives the resource settings of the computer vision-based AR application from the application database of the system 3006, and then generates an instance for the (specific to) computer vision-based AR application, which has a state machine engine and a DNN (computer vision-based processing) engine specific to the computer vision-based AR application. After generating the instance, the AR device 3004 and the system 3002 can collaborate to use the computer vision-based AR application to perform various actions or tasks. For example, the AR device 3004 can acquire an image or image data (through its own image sensor or by receiving the image or image data from another device) and transmit the image or image data to the system 3002 for processing. The system 3002 can detect the presence of a target object in the image and / or one or more attributes thereof, and provide a computational result in response. The system 3002 provides data associated with the computational result (e.g., the computational result itself, a response associated with the computational result to be implemented at the AR device, etc.) to the AR device 3004 for processing and / or analysis to execute a response at the AR device (e.g., providing audio / video information).

[0158] System 3000 can perform one or more of the method examples described herein.

[0159] It should also be understood that when the methods and systems of the embodiments of the present invention are fully or partially implemented by a computing system, any appropriate computing system architecture may be utilized, including stand-alone computers, network computers, dedicated or non-dedicated hardware devices, etc. In this document, the terms "computing system" and "computing device" (or similar terms) are intended to include, but are not limited to, computers or information processing hardware (in any appropriate arrangement) capable of implementing the described functions.

[0160] The present invention may also provide a system having appropriate means for performing one or more method embodiments of the present invention, an extended reality (XR) device having appropriate means for performing one or more method embodiments of the present invention, a non-transitory computer-readable storage medium (which stores one or more programs configured to be executed by one or more processors, the one or more programs including instructions for performing or facilitating the performance of one or more method embodiments of the present invention), and a computer program product (which includes one or more programs configured to be executed by one or more processors, the one or more programs including instructions for performing or facilitating the performance of one or more method embodiments of the present invention).

[0161] The above-mentioned embodiments of the present invention provide various advantages. For example, some embodiments of the present invention provide systems and methods that can facilitate the generation of extended reality (XR, such as AR) applications based on computer vision. For example, some embodiments of the present invention provide systems and methods that can convert designer input and developer input into application specification data, state machines, resource settings, and corresponding computer vision algorithms (e.g., DNN plug-ins). For example, some embodiments of the present invention provide operational control processes for supporting multiple instances of AR applications based on computer vision. For example, some embodiments of the present invention provide specific methods for AR guidance for software screen installation of IT equipment (sharing screens, generating event tables, etc.). For example, some embodiments of the present invention provide transmission control strategies between AR devices and corresponding external computing systems (e.g., ARCVS), which are used to optimize the performance of AR applications based on computer vision (e.g., to ensure that quality images or image data are used to obtain as reliable results as possible).

[0162] Those skilled in the art will appreciate that various variations and / or modifications may be made to the invention as shown in a particular embodiment to provide other embodiments of the invention. Therefore, the embodiments of the invention described should be considered in all respects to be illustrative and not restrictive. Example optional features of some aspects of the invention are set forth in the Summary of the Invention section above. Some embodiments of the invention may include one or more of these optional features (some of which are not specifically shown in the drawings). Some embodiments of the invention may lack one or more of these optional features (some of which are not specifically shown in the drawings). One or more features in one embodiment and one or more features in another embodiment may be combined to provide further embodiments of the invention. For example, where appropriate and applicable, the order of method steps of a method embodiment may be different from that illustrated. For example, some method steps may be performed at least partially simultaneously. The same computer vision-based XR application may operate (substantially independently) at multiple XR devices. Although some embodiments shown and / or described in the present invention (e.g., reference to Figures 23 to 30 The embodiments described herein relate to computer vision-based AR applications and AR devices, but it should be noted that these embodiments or the more general present invention may be broadly applicable to computer vision-based XR (AR, VR, and / or MR) applications and XR (AR, VR, and / or MR) devices.

Claims

1. A method for facilitating the generation and / or operation of a computer vision-based extended reality application; the method comprising generating an application kit for a computer vision-based extended reality application, the application kit comprising a set of data files for operating the computer vision-based extended reality application; The generating comprises: Receive data, the data comprising at least: User experience and user interface design data for the computer vision-based extended reality application, and Computer vision-based extended reality instruction data for the computer vision-based extended reality application; and The data is processed to generate the application suite of the computer vision-based extended reality application, wherein the application suite is arranged to be deployed on an extended reality device and / or an external computing system for operating the computer vision-based extended reality application.

2. The method according to claim 1, characterized in that The data received and / or processed also includes: Data associated with a storage location storing data associated with the computer vision-based extended reality application.

3. The method according to claim 1, characterized in that: The application suite includes at least: Specification data for said computer vision-based extended reality application, a state machine for the computer vision-based extended reality application, and a computer vision-based processing algorithm for the computer vision-based extended reality application; wherein the specification data of the computer vision-based extended reality application is arranged to be deployed on the extended reality device for operating the computer vision-based extended reality application; and Wherein, the state machine of the computer vision-based extended reality application and the computer vision-based processing algorithm of the computer vision-based extended reality application are arranged to be deployed on an external computing system for operating the computer vision-based extended reality application.

4. The method according to any one of claims 1 to 3, characterized in that: The data received and / or processed may also include one or more of the following: data transmission control parameters for operating the computer vision-based extended reality application, and data associated with one or more characteristics of an extended reality device for operating the computer vision-based extended reality application; as well as The application suite further includes: A data transmission strategy for operating the computer vision-based extended reality application.

5. The method according to any one of claims 1 to 3, characterized in that The method comprises: Initialize the computer vision-based extended reality application.

6. The method according to claim 5, characterized in that: The initialization includes performing the following steps at the extended reality device: transmitting an identifier associated with the computer vision-based extended reality application to the external computing system using an address of the external computing system; as well as In response, receiving access information for accessing the instance of the computer vision-based extended reality application from the external computing system, wherein the instance is arranged to be operated by the external computing system and includes a state machine engine of the computer vision-based extended reality application and a computer vision-based processing engine of the computer vision-based extended reality application; and The instance of the computer vision-based extended reality application is generated at the external computing system in response to the external computing system receiving an identifier associated with the computer vision-based extended reality application.

7. The method according to claim 5, characterized in that: The initialization includes performing the following steps at the external computing system: receiving an identifier associated with the computer vision-based extended reality application; generating an instance for the computer vision-based extended reality application based on the identifier, wherein the instance is arranged to be operated by the external computing system and includes a state machine engine of the computer vision-based extended reality application and a computer vision-based processing engine of the computer vision-based extended reality application; as well as transmitting access information for accessing the instance of the computer vision-based extended reality application to an extended reality device; Wherein, the generating includes: determining or obtaining a computing resource setting for the instance of the computer vision-based extended reality application based on the identifier, and The instance is generated based on the determined or obtained computing resource settings.

8. The method according to claim 6, characterized in that The method comprises: Operating the computer vision-based extended reality application.

9. The method according to claim 8, characterized in that: The operation includes performing the following steps at the extended reality device: Get the image; transmitting the image to an external computing system, and performing computer vision-based processing on the image by the instance of the computer vision-based extended reality application; receiving data associated with processing results of the computer vision-based processing of the image; as well as The received data associated with the processing result is processed to implement a response on the extended reality device.

10. The method according to claim 9, characterized in that Acquiring the image at the extended reality device includes: Capturing an image of an environment using an image sensor of the extended reality device, wherein the extended reality device is arranged in the environment; or An image associated with screen content of a screen mirroring or screen sharing device is received.

11. The method according to claim 9, characterized in that: The computer vision based processing is arranged to detect one or more target objects in the image; and The data associated with the processing results includes data associated with target object detection results in response to detecting one or more target objects in the image.

12. The method according to claim 11, characterized in that The data associated with the target object detection result includes: data for implementing a response at the extended reality device in response to detecting one or more target objects in the image.

13. The method according to claim 9, characterized in that: The response comprises a visual and / or an audio response; The visual and / or audio response is associated with extended reality-based instructions or guidance information to be provided by the extended reality device to a user of the extended reality device; The extended reality-based instructions or guidance information is arranged to assist or facilitate the user in performing an action or task; as well as (a) the visual response includes the generation, change or termination of a graphical user interface arranged to be provided by the extended reality device and / or (b) the audio response includes the generation, change or termination of a sound arranged to be provided by the extended reality device.

14. The method according to claim 9, characterized in that: The transmission of the image is performed according to a data transmission strategy, and the data transmission strategy is adapted to (i) the quality or quality changes of the communication link between the extended reality device and the external computing system and / or (ii) the processing performance or processing performance changes of the external computing system; as well as The data transmission strategy is arranged to affect the resolution of the image and / or the frame rate associated with the image based on the communication quality or quality changes between the extended reality device and the external computing system and / or the processing performance or processing performance changes of the external computing system.

15. The method according to claim 14, characterized in that The quality or change in quality of the communication is related to one or more of the following: latency of one or more communication links between the extended reality device and the external computing system, a speed of one or more communication links between the extended reality device and the external computing system, The bandwidth of one or more communication links between the extended reality device and the external computing system, and the packet loss rate or number of packet losses in one or more communication links between the extended reality device and the external computing system.

16. The method according to claim 9, characterized in that: The operation also includes performing the following steps before the transmitting: Determining, in substantially real time, a quality or a change in the quality of a communication link between the extended reality device and the external computing system and / or a processing performance or a change in the processing performance of the external computing system; as well as determining a resolution of the image and / or a frame rate associated with the image for transmission to the external computing system based on the determined quality or change in quality and / or processing performance or change in processing performance; as well as The transmission of the image is based on the determined resolution and / or frame rate.

17. The method according to claim 16, characterized in that: If it is determined that the communication quality or quality change and / or the processing performance or processing performance change of the external computing system has decreased, then: if the frame rate does not reach a minimum frame rate, reducing a frame rate associated with the image for transmission to the external computing system; If the frame rate has reached a minimum frame rate and the resolution has not reached a minimum resolution, reducing the resolution of the image for transmission to the external computing system; as well as If the frame rate has reached a minimum frame rate and the resolution has reached a minimum resolution, providing an error message at the extended reality device; and / or If it is determined that the communication quality or quality change and / or the processing performance or processing performance change of the external computing system has increased, then: If the resolution does not reach the maximum resolution, increasing the resolution of the image for transmission to the external computing system; as well as If the resolution has reached a maximum resolution and the frame rate has not reached a maximum frame rate, increasing a frame rate associated with the image for transmission to the external computing system.

18. The method according to claim 8, characterized in that: The operation includes performing the following steps at the external computing system: receiving an image; processing the image using computer vision-based processing in an instance for use in the computer vision-based extended reality application; as well as Data associated with a result of the computer vision-based processing of the image is transmitted to an extended reality device to facilitate implementation of a response at the extended reality device.

19. The method according to claim 18, characterized in that The data associated with the processing result includes data associated with a response to be implemented in the extended reality device.

20. The method according to claim 18, characterized in that: The computer vision based processing is arranged to detect one or more target objects in the image, and the data associated with the processing results comprises data associated with target object detection results responsive to detecting the one or more target objects in the image.

21. The method according to claim 20, characterized in that The data associated with the target object detection results includes data for implementing a response at the extended reality device in response to detecting one or more target objects in the image.

22. The method according to claim 18, characterized in that: The response comprises a visual and / or an audio response; The visual and / or audio response is associated with extended reality-based instructions or guidance information to be provided by the extended reality device to a user of the extended reality device; The extended reality-based instructions or guidance information is arranged to assist or facilitate the user in performing an action or task; as well as (a) the visual response includes the generation, change or termination of a graphical user interface arranged to be provided by the extended reality device and / or (b) the audio response includes the generation, change or termination of a sound arranged to be provided by the extended reality device.

23. The method according to claim 8, characterized in that: The operation includes performing the following steps at the extended reality device: receiving a prompt signal from an external computing system; and In response to receiving the prompt signal, a keep-alive signal is transmitted to the external computing system to maintain operation of the instance of the computer vision-based extended reality application on the external computing system.

24. The method according to claim 8, characterized in that: The operation includes performing the following steps at the external computing system: transmitting a prompt signal to an extended reality device; determining whether a corresponding keep-alive signal is received from the extended reality device; as well as If it is determined that a corresponding keep-alive signal is received, maintaining the operation of the instance of the computer vision-based extended reality application in the external computing system; as well as If it is determined that the corresponding keep-alive signal is not received, the instance is terminated in the external computing system to release computing resources associated with the instance.

25. The method according to claim 8, characterized in that: The operation includes performing the following steps at the external computing system: transmitting a prompt signal to an extended reality device; determining a total number of corresponding keep-alive signals received from the extended reality device; as well as If it is determined that the total number of corresponding keep-alive signals is greater than a set number, maintaining the operation of the instance of the computer vision-based extended reality application on the external computing system; as well as If it is determined that the total number of corresponding keep-alive signals is lower than a set number, the instance is terminated in the external computing system to release computing resources associated with the instance.

26. The method according to claim 8, characterized in that: The operation includes performing the following steps at the extended reality device: Determining that the operation of the computer vision-based extended reality application is about to be terminated or has been terminated, and in response to the determination, transmitting a termination signal to an external computing system to terminate the instance of the computer vision-based extended reality application of the external computing system, thereby releasing computing resources associated with the instance.

27. The method according to claim 8, characterized in that: The operation includes performing the following steps at the external computing system: receiving, from the extended reality device, a termination signal associated with termination of operation of the computer vision-based extended reality application; as well as In response to receiving the termination signal, the instance of the computer vision-based extended reality application is terminated in the external computing system, thereby releasing computing resources associated with the instance.

28. A system for facilitating generation and / or operation of computer vision-based extended reality applications, the system comprising: An extended reality device arranged to perform or facilitate the performance of a method according to any one of claims 1 to 27; and / or An external computing system arranged to perform or facilitate the performance of a method according to any one of claims 1 to 27.

Images