Augmented Reality Testing Method, Device, Terminal Device, Server, and Storage Medium

Abstract

The present application discloses an augmented reality testing method, apparatus, terminal device, server and storage medium. The method includes: the server receives test configuration information sent by the first terminal device, the test configuration information includes first destination information, and the location of the first terminal device does not belong to the first destination; the server generates an AR test task and a virtual target object according to the test configuration information, and the virtual target object is used to perform the AR test task in the virtual environment corresponding to the first destination; in the process of the virtual target object performing the AR test task, the server displays part or all of the field of view of the virtual target object itself on the screen of the first terminal device; when the specified AR image appears on the screen of the first terminal device, the server determines that the current AR application meets the test requirements. AR remote testing can be achieved using the embodiments disclosed in this application.

Description

Technical Field

[0001] The present application relates to the field of software testing technology, and specifically to an augmented reality (AR) testing method, apparatus, terminal device, server, computer-readable storage medium, and computer program product. Background Art

[0002] With the continuous iteration of augmented reality (AR) technology, many applications have been found in various fields of the economy and society. For example, various AR software products focusing on digital cities, education, transportation, medical care, shopping, entertainment and other fields can superimpose real environments and virtual objects in the same screen or space in real time, providing people with a digital experience different from the past. At present, for the testing and acceptance of AR software products, in most cases, testers still need to go to the site to conduct real-life testing, run the AR software at the destination, and combine real-time positioning information to verify whether the AR product can generate the expected AR effect at the specified location. However, because this testing process requires personnel to be present, it will cause many pain points. For example, on-site testing requires a high investment of time and manpower costs, resulting in delays in the completion of testing tasks and hindering the product release process. Summary of the Invention

[0003] In view of this, embodiments of the present application provide an augmented reality (AR) testing method, apparatus, terminal device, server, computer-readable storage medium, and computer program product for solving at least one technical problem.

[0004] In a first aspect, an embodiment of the present application provides an AR testing method, which is applied to a server, including: the server receives test configuration information sent by a first terminal device, the test configuration information includes first destination information, and the location of the first terminal device does not belong to the first destination; the server generates an AR test task and a virtual target object according to the test configuration information, and the virtual target object is used to perform the AR test task in a virtual environment corresponding to the first destination; during the process of the virtual target object performing the AR test task, the server displays part or all of the field of view of the virtual target object itself on the screen of the first terminal device; when a specified AR image appears on the screen of the first terminal device, the server determines that the current AR application meets the test requirements.

[0005] In a second aspect, an embodiment of the present application provides an AR testing method, which is applied to a terminal device, comprising: in the operating environment of an AR application to be tested, a first terminal device receives test configuration information input externally and sends it to a server; wherein, the test configuration information includes first destination information, the location of the first terminal device does not belong to the first destination, and the test configuration information is used to enable the server to generate an AR test task and a virtual target object, and the virtual target object is used to perform the AR test task in a virtual environment corresponding to the first destination; during the process of the virtual target object performing the AR test task, the first terminal device displays part or all of the field of view of the virtual target object itself on the screen; the first terminal device sends relevant information of the AR image appearing on the screen to the server.

[0006] In a third aspect, an embodiment of the present application provides an AR testing device, comprising:

[0007] a transceiver module, configured to receive test configuration information sent by a first terminal device, the test configuration information including first destination information, and the location of the first terminal device does not belong to the first destination;

[0008] a generating module, configured to generate an AR test task and a virtual target object according to the test configuration information, wherein the virtual target object is used to execute the AR test task in a virtual environment corresponding to the first destination;

[0009] A display processing module is configured to display part or all of the field of view of the virtual target object itself on the screen of the first terminal device during the process of the virtual target object performing the AR test task;

[0010] The determination module is used to determine whether the current AR application meets the test requirements when a specified AR image appears on the screen of the first terminal device.

[0011] In a fourth aspect, an embodiment of the present application provides an AR testing device, comprising:

[0012] a transceiver processing module, configured to receive externally input test configuration information and transmit it to a server in an operating environment of an AR application to be tested; wherein the test configuration information includes first destination information, the location of the first terminal device does not belong to the first destination, and the test configuration information is used to enable the server to generate an AR test task and a virtual target object, wherein the virtual target object is used to execute the AR test task in a virtual environment corresponding to the first destination;

[0013] A display processing module, configured to display part or all of the field of view of the virtual target object itself on a screen during the process of the virtual target object performing the AR test task;

[0014] The transceiver processing module is further configured to send information related to the AR image appearing on the screen to the server.

[0015] In a fifth aspect, an embodiment of the present application provides a server device comprising: a processor and a memory storing computer program instructions; the processor implements the steps of the method described above when executing the computer program instructions.

[0016] In a sixth aspect, an embodiment of the present application provides a terminal device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the steps of the method described above are implemented.

[0017] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the steps of the method described above are implemented.

[0018] In an eighth aspect, an embodiment of the present application provides a computer program product, which includes computer program instructions, and when the computer program instructions are executed by a processor, the steps of the method described above are implemented.

[0019] By using the solution provided in the embodiment of the present application to perform AR testing, there is no need for personnel to be present on site. Instead, the test is carried out in a constructed virtual environment to obtain reliable test results and achieve the test purpose. Therefore, the embodiment of the present application can address the pain points in current AR product testing, making on-site testing no longer an indispensable link. It can be completed with a lower investment in time and manpower costs, which is conducive to the release and promotion of AR technology and products. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings in the embodiments of the present application.

[0021] Figure 1 This is a schematic diagram of the AR system architecture based on servers and terminal devices in an embodiment of the present application.

[0022] Figure 2 It is a schematic diagram of virtual-reality fusion images using a mobile phone APP for AR navigation.

[0023] Figure 3 It is a flowchart of the AR testing method on the server side of an embodiment of the present application.

[0024] Figure 4 It is a flowchart of the AR testing method on the terminal device side of an embodiment of the present application.

[0025] Figure 5 This is a test flow diagram of the AR test method based on an embodiment of the present application.

[0026] Figure 6 is based on Figure 5 A schematic diagram of a scenario showing the AR effect that should appear when a virtual person in an embodiment moves in a virtual environment.

[0027] Figure 7 is with Figure 6 A schematic diagram of a mobile phone screen image bound to a virtual person in an embodiment.

[0028] Figure 8 This is a structural block diagram of the AR testing device on the server side of an embodiment of the present application.

[0029] Figure 9 This is a structural block diagram of the AR testing device on the terminal device side of an embodiment of the present application.

[0030] Figure 10 Schematic diagram of an electronic device used to implement the AR testing method of an embodiment of the present application.

[0031] Figure 11 It is a software structure diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that the purpose of providing these embodiments is to make the principles and spirit of the present application clearer and more thorough, so that those skilled in the art can better understand and implement the principles and spirit of the present application. The exemplary embodiments provided herein are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments herein, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of this application.

[0033] Those skilled in the art will appreciate that the embodiments of the present application may be implemented as a system, apparatus, device, method, computer-readable storage medium, or computer program product. Therefore, the present application may be implemented in at least one of the following forms: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0034] According to the implementation methods of the present application, the present application requests protection for an augmented reality (AR) testing method, apparatus, terminal device, server, and computer-readable storage medium.

[0035] In this document, terms such as first, second, and third are only used to distinguish one entity (or operation) from another entity (or operation), and are not intended to require or imply any order or relationship between these entities (or operations).

[0036] The embodiments of the present application can be applied to servers and terminal devices. Figure 1 , schematically shows a schematic diagram of an AR system architecture based on a server and a terminal device. The AR system architecture includes a server 10 and several terminal devices 20. In some examples, the terminal device 20 is an AR device, which can be a dedicated AR device, such as a head-mounted AR device (Head-mounted displays, HMD), smart gloves, clothing and other wearable electronic devices. In some examples, the terminal device 20 can be a general AR device, such as a mobile phone, portable computer, laptop computer, tablet computer, virtual reality (VR) device or vehicle-mounted device installed with AR function software, etc.

[0037] Taking AR helmets or AR glasses as an example, a head-mounted display, machine vision system, mobile computer, and other components can be integrated into a wearable device. This device has a display that resembles glasses and is worn on the user's head during operation. The device can transmit augmented reality information to the display or project it into the user's eyeballs, thereby enhancing the user's visual immersion. In some examples, AR devices also have a camera, which can be a wide-angle camera, a telephoto camera, or a structured light camera (also known as a point cloud depth camera, a 3D structured light camera, or a depth camera). Structured light cameras are based on 3D vision technology and can obtain the plane and depth information of objects. Structured light cameras use near-infrared lasers to project light with certain structural features onto the object being photographed. The reflected light is then collected by an infrared camera and processed by a processor chip. The calculation principle is to calculate the object's position and depth information based on the changes in the light signal caused by the object, thus presenting a 3D image. Conventional devices, such as mobile phones, display two-dimensional images but cannot show the depth of different locations within the image. Structured light cameras can capture 3D image information data, providing not only information such as color at different locations within the image, but also depth information at different locations, which can be used for AR ranging. Of course, ordinary devices can also use optical cameras to capture 2D images and combine them with deep learning algorithms to obtain depth information from 2D images, ultimately rendering 3D images.

[0038] In some examples, the terminal device 20 is installed with software or an application program (APP) with AR functionality. The server 10 can be a management server or application server for the software or APP. The server 10 can be a single server, a server cluster consisting of multiple servers, a cloud server, or the like. The terminal device 20 is integrated with a module with networking functionality, such as a wireless fidelity (Wi-Fi) module, a Bluetooth module, a 2G / 3G / 4G / 5G communication module, etc., to connect to the server 10 via a network.

[0039] Taking an app with AR navigation functionality as an example, the app may have, for example, high-precision map navigation capabilities, environmental understanding capabilities, and virtual-reality fusion rendering capabilities. The app may report current geographic location information to the server 10 via the terminal device 20, and the server 10 provides AR navigation services to the user based on the real-time geographic location information. For example, in the case where the terminal device 20 is a mobile phone, in response to the user launching the app, the mobile phone may activate the camera to capture an image of the real environment. The system then performs AR enhancement on the real-world image captured by the camera, integrating or overlaying rendered AR effects (such as navigation route signs, road names, business information, advertising displays, etc.) on the real-world image, and displaying the virtual-reality fusion image on the mobile phone screen.

[0040] For example, the first user can log in to the first user account through the APP installed in the mobile phone, and the second user can log in to the second user account through the software installed in the AR glasses.

[0041] Figure 2 This schematic illustrates a fusion of virtual and real worlds using a mobile app for AR navigation. AR navigation arrows are superimposed on the real road surface and space, and digital products promoted by merchants are displayed within buildings in the form of small gift boxes carried by parachutes. While walking with their phone, users can view the AR navigation route on the phone screen and easily access digital information about their surroundings.

[0042] The embodiments of the present application relate to terminal devices and / or servers. The principles and spirit of the present application will be explained in detail below through several exemplary embodiments or representative implementations.

[0043] First, a brief description of the concepts and technical terms that may be involved in the embodiments of this application is given.

[0044] There are many types of AR software products, and currently they are mainly software programs installed on hardware terminal devices, clients, application programs (APPs), or mini-programs within APPs, etc. The embodiments of this application have no special requirements for the type or operating mode of AR software products. In other words, all types of AR software products can be tested using the embodiments of this application.

[0045] Generally, software testing refers to the process of verifying the correctness, integrity, security, and reliability of software. It is also a process of comparing and auditing expected outputs with actual outputs. During software testing, software programs are operated under specified conditions to identify program errors, measure software quality, and evaluate whether the software meets design requirements.

[0046] When testing AR software products, since a considerable portion of their functionality involves geolocation, testing requires obtaining the actual geographic location of the terminal device hosting the AR software. Therefore, as mentioned above, most AR software product testing is conducted by testers in real-world scenarios. For example, adding AR effects to a building's interior requires field testing, where testers carry AR equipment to designated locations within the building to verify that it can display the designed AR effects. In this field, many pain points associated with field testing remain unresolved.

[0047] In view of this, the embodiment of the present application proposes an augmented reality AR testing method, which is applied to the server, referring to Figure 3 , including the following steps:

[0048] S101, a server receives test configuration information sent by a first terminal device, where the test configuration information includes first destination information, and the location of the first terminal device does not belong to the first destination;

[0049] S102, the server generates an AR test task and a virtual target object according to the test configuration information, wherein the virtual target object is used to execute the AR test task in the virtual environment corresponding to the first destination;

[0050] S103, during the process of the virtual target object performing the AR test task, the server displays part or all of the field of view of the virtual target object itself on the screen of the first terminal device;

[0051] S104: When a specified AR image appears on the screen of the first terminal device, the server determines that the current AR application meets the test requirements.

[0052] According to an AR testing method according to an embodiment of the present application, after receiving test configuration information sent by a terminal device, the server generates an AR test task for the test destination and also generates a virtual target object (e.g., a virtual person or a virtual camera) for testing. Here, the terminal device itself is not required to be located at the test destination. Instead, AR testing is accomplished by having the virtual target object perform the AR test task within the virtual environment of the test destination. During the test, to confirm the AR visual effect generated by the AR software within the test destination, the field of view of the virtual target object is displayed on the terminal device screen for the tester to observe and confirm the test results. The principle of this process is that since the virtual environment and the virtual target object are both constructed and generated by the server, the position and posture of the virtual target object, such as the virtual person, in the virtual environment at each moment is known on the server side, and the content within the virtual person's field of view is also accessible. Therefore, when the virtual person enters the designated area, if the AR effect display is triggered, the AR effect will be displayed on the tester's terminal device, indicating that the test of the designated area point is successful. Conversely, if the AR effect does not appear on the terminal device, it indicates that there is a problem with the designated area point and adjustment and troubleshooting is required. In this way, testers only need to use terminal devices in a remote location to know the AR visual effects produced by the AR software at the test destination, and then confirm whether the AR software has passed the test, thereby achieving the purpose of remote testing of the AR software. This can save testers the trouble of traveling to the test destination, save time and labor costs, and facilitate the promotion of AR technology.

[0053] Corresponding to the processing on the server side, the embodiment of the present application also provides an AR testing method, which is applied to the terminal device, referring to Figure 4 , including the following steps:

[0054] S201: In an operating environment of an AR application to be tested, a first terminal device receives externally input test configuration information and sends it to a server; wherein the test configuration information includes first destination information, the location of the first terminal device does not belong to the first destination, and the test configuration information is used to cause the server to generate an AR test task and a virtual target object, wherein the virtual target object is used to execute the AR test task in a virtual environment corresponding to the first destination;

[0055] S202, during the process of the virtual target object performing the AR test task, the first terminal device displays part or all of the field of view of the virtual target object itself on the screen;

[0056] S203: The first terminal device sends information related to the AR image appearing on the screen to the server.

[0057] According to the AR testing method of the embodiment of the present application, on the terminal side, the tester inputs configuration information through the terminal device used for testing. The configuration information needs to include information about the test destination. For example, the tester and the terminal device used for testing are located in city K, and the test destination is assumed to be a landmark building in city L. The configuration information should include information about the landmark building in city L, so that the server can generate an AR test task and a virtual target object for the landmark building. In addition, during the execution of the AR test task by the virtual target object, the terminal device can present the field of view of the virtual target object on the screen of the terminal device. When an AR image appears, the terminal device sends relevant information of the AR image (such as name, presentation duration, etc.) to the server, so that the server can determine whether it is consistent with the specified AR image. If it is consistent, it indicates that the software testing requirements are met.

[0058] To more clearly illustrate the advantages of the embodiments of the present application, the following describes the essential differences between the embodiments of the present application and conventional tests using a test task of examining the AR effect of walking from location or point S1 to point S2 in shopping mall A. Assuming that, according to the design of the AR software, a user wearing AR glasses or carrying a mobile phone with the app installed can see an AR navigation route from S1 to S2 with a virtual arrow indicating the direction of travel superimposed on the ground. When following the arrow to point P along the way, the user can see a group of cartoon characters floating in the air.

[0059] For the above test tasks, a common solution involves the tester acting as the user and conducting a field test in Mall A. While walking from point S1 to point S2, the tester must verify whether a virtual moving arrow appears on the ground and whether a group of floating cartoon characters appears in mid-air upon reaching point P. If all of the above AR effects display normally on the test device, the test is considered passed.

[0060] Unlike conventional field tests, in the embodiments of the present application, testers do not need to be present in Mall A. Instead, they can be in a location that can be connected to the server. They can input configuration information through a test terminal, such as a mobile phone. After uploading it to the server, the server can retrieve the modeling data of Mall A from the database based on the configuration information, generate a virtual environment, or digital twin, of Mall A, and also generate test tasks and virtual humans. The test task can be to have the virtual human walk from point S1 to point S2, passing through point P. As the virtual human walks, the server transmits the image "seen" by the virtual human back to the mobile phone and plays it back to confirm whether the AR effect is displayed normally and the test purpose is achieved. For example, if a virtual moving arrow is superimposed on the road surface in the image, and a group of cartoon characters appear suspended in the air when the virtual human reaches point P, the test can be considered to have passed.

[0061] It can be seen that the implementation of AR testing using the embodiments of the present application does not require on-site personnel. Instead, the test is carried out in a constructed virtual environment, achieving the test objectives and obtaining reliable test results. Therefore, the embodiments of the present application can address the pain points in current AR product testing, making on-site testing no longer an essential step and can be completed with lower time and labor costs, which is conducive to the release and promotion of AR technology and products.

[0062] According to an embodiment of the present application, the following processing may also be included: binding the first terminal device to the virtual target object; during the process of the virtual target object performing the test task, adjusting the position and / or posture of the virtual target object in the virtual environment by adjusting the position and / or posture of the first terminal device.

[0063] The reason for this is that in order to confirm the test effect, it is necessary to observe different directions in the virtual environment. Therefore, during the movement of the virtual target object, the posture of the virtual target object also needs to be changed, thereby changing its field of view. In order to quickly adjust the posture of the virtual target object, the test terminal device can be bound to the virtual target object. In this way, the tester can directly adjust the posture of the virtual target object by changing the posture of the terminal device. The adjustment process is quick and intuitive. In application, as an example, the terminal device can be installed on a hardware device with motion capture and acquisition functions, such as a pan-tilt head or bracket with multiple degrees of freedom adjustment, which can quickly adjust the posture of the terminal device to achieve posture adjustment of the virtual target object.

[0064] According to an embodiment of the present application, the following processing may also be included: during the execution of the AR test task by the virtual target object, if a specified AR image appears on the screen of the first terminal device, the first terminal device sends a first indication message to the server to indicate that the specified AR image has appeared on the screen of the first terminal device.

[0065] Correspondingly, after receiving the first indication information, the server can determine that the specified AR image has appeared on the screen of the first terminal device.

[0066] The reason for this is that during the test, it is necessary to confirm whether the AR effects designed in the software are displayed normally. For example, for the virtual cartoon character that should appear at point P mentioned in the previous example, if the virtual cartoon character appears when the virtual target object reaches point P during the test, the server can be notified to record the content through an indication message. Optionally, the first indication message includes at least one of the following: a string in a predetermined format, a data frame with predetermined bits, or a message in a predetermined format. After the test, if all specified AR effects appear, it means that the test meets the requirements.

[0067] As an example, the first indication information may include a screen image when a specified AR image appears on the screen of the first terminal device. The screen image is used to replay the specified AR image when testing again. If the same AR image appears on the terminal device screen when testing again, it is determined that the current test meets the requirements.

[0068] For example, the terminal device can retain the AR effect image that appears during the test by means of screenshots, screen recording, etc., and can upload the AR effect image and related time information, location information, etc. to the server. It can also be saved in the terminal memory for subsequent testing, so that the effects of multiple tests can be compared intuitively.

[0069] The above process can also be replaced by recording and replaying during the test. That is, the data information generated and interacted between the server and the terminal during the current test is recorded to form a script program. When the test needs to be repeated later, the script can be replayed to facilitate analysis of the test results.

[0070] During the testing and application process of the embodiments of the present application, the AR product should be tested again when the test destination changes, the software version is iterated, or the software operating environment changes. By utilizing the solution provided by the above embodiments of the present application, reporting instruction information, reporting AR effect images, or performing recording and playback, problems can be quickly located and solved in multiple tests, which is conducive to solving problems in AR software regression testing and compatibility testing.

[0071] According to an embodiment of the present application, generating an AR test task and a virtual target object according to the test configuration information can be achieved by the following processing:

[0072] (1) generating a virtual environment corresponding to the first destination based on the modeling data of the first destination and the related panoramic image data;

[0073] (2) generating a virtual target object, wherein the virtual target object is located in the virtual environment;

[0074] (3) Generating an AR test task, where the AR test task includes the virtual target object moving from a first position to a second position along a specified route in the virtual environment.

[0075] Additionally, the following processing can be performed:

[0076] (1) setting a plurality of movable virtual auxiliary objects in the virtual environment;

[0077] (2) During the process of the virtual target object performing the AR test task, the positioning information of the virtual target object and the virtual auxiliary object is recorded.

[0078] In detail, the modeling data and related panoramic image data of the test destination, such as Shopping Mall A, can be acquired in advance, for example, by scanning with a depth camera or AR camera, or directly acquired from the outside (for example, from the construction or design party of Shopping Mall A) and stored in the system database. When it is necessary to build a virtual digital twin of the test destination, the server can call this data from the database for modeling to generate a virtual environment corresponding to Shopping Mall A.

[0079] Furthermore, the server can generate multiple virtual people and AR test tasks, among which one virtual person can be used as the test subject and bound to the terminal. The virtual person travels along the route of the AR test task, starting from the first position and arriving at the second position. The remaining large number of virtual people can be set to move randomly in the virtual environment of Mall A, which can restore the real scene as much as possible, which is conducive to improving the robustness of the test.

[0080] According to an embodiment of the present application, the test configuration information may also include the model information of the first terminal device. That is, during the test, in addition to configuring the test destination, the model of the terminal device used for the test may also be configured. This is because different terminal devices provide different computing environments for AR software, and the performance of AR software is also different, such as display position, jump mode, etc. It is necessary to confirm whether the AR effect is displayed normally under various models, so as to facilitate timely adjustment of relevant program instructions and ensure that the performance of AR software on different terminal devices meets the design requirements. The following describes two situations of the embodiment of the present application.

[0081] Case 1: The test destination is the same, but the test terminal models are different

[0082] In the operating environment of the AR application to be tested, new test configuration information is received through the second terminal device, wherein the new test configuration information includes second destination information and model information of the second terminal device, the location of the second terminal device does not belong to the second destination, and the second destination is the same as the first destination, and the model of the second terminal device is different from the model of the first terminal device (for example, the first terminal device is an H-brand mobile phone with an Android system, and the second terminal device is an H-brand tablet with an Android system); a new AR test task and a new virtual target object are generated according to the new test configuration information, and after the new virtual target object executes the new test task, difference information between the AR image appearing on the second terminal device and the AR image appearing on the first terminal device is determined, and the difference information is used to modify the AR application.

[0083] Case 2: Different test destinations, same test terminal model

[0084] In the operating environment of the AR application to be tested, new test configuration information is received through a third terminal device, where the new test configuration information includes third destination information and model information of the third terminal device. The location of the third terminal device does not belong to the third destination, and the third destination is different from the first destination. The model of the third terminal device is the same as that of the first terminal device (for example, the first destination is a shopping mall in City T, and the third destination is a landmark building in City W). A new AR test task and a new virtual target object are generated according to the new test configuration information. After the new virtual target object executes the new test task, the difference information between the AR image appearing on the third terminal device and the AR image appearing on the first terminal device is determined, and the difference information is used to modify the AR application.

[0085] The above describes the implementation of the AR testing method of the embodiment of the present application and the advantages brought by it through multiple embodiments. The following describes the specific processing process of the embodiment of the present application in detail with reference to specific examples.

[0086] refer to Figure 5 、 Figure 6 and Figure 7 As an example, the following takes the test terminal as mobile phone 1 and the test task as path navigation as an example to describe the specific operation process that can be taken in the embodiment of the present application.

[0087] refer to Figure 5 First, the mobile phone 1 can be connected to a visual operation interface. The tester can input the test configuration information through the visual operation interface, for example, input the following configuration information:

[0088] Configuration 1: Positioning scene configuration information; for example, the test destination can be positioned, for example, the virtual environment and internal scene of the test destination can be pre-generated. In this embodiment, the test destination is a landmark tower in a 5A-level scenic spot as an example;

[0089] Configuration 2: Virtual human configuration information and route configuration information; that is, configure the virtual human and its route for testing. For example, the test task of the virtual human is to start at the entrance of the iconic city tower of the 5A scenic spot and move along the tour route until reaching the top floor.

[0090] Configuration 3: configuration information such as the mobile phone model; for example, you can enter the model of mobile phone 1 and the version information of the operating system it carries.

[0091] Then, the information of configurations 1-3 is sent to the control system, which processes the configuration information accordingly and stores it in the database to prepare for subsequent steps. The computing power of the control system can be provided by the server.

[0092] The database stores scanned data of the test destination, including model data and panoramic images required for modeling. This data can be stored in advance by staff using AR equipment to scan the real scene and store it in the database. Based on the scanned data of the test destination and combined with the information in Configurations 1-3 above, the positioning mock service configuration can be completed. The virtual human and route configuration can also be completed (so that the virtual human moves according to the route of the test task in "Configuration 2"). The entire test process can also be adapted to the model of Mobile Phone 1. For example, the layout of the AR image on the display interface, the legend and position of related buttons, etc. can be adapted according to the display ratio of Mobile Phone 1 or whether it is a foldable screen.

[0093] In this embodiment, a mobile phone 1 is bound to a virtual human, and the amplitude, frequency, and posture of the virtual human's walking can be changed by adjusting the phone. As an example, the mobile phone 1 can be placed on a motion capture module, such as a full-degree-of-freedom gimbal, and its posture adjustment mode can be set to change the posture of the mobile phone 1 in a set manner. Alternatively, the mobile phone 1 can be set to move randomly, or the tester can change the posture of the mobile phone 1 by holding it.

[0094] refer to Figure 6 The figure schematically illustrates the scenario that should occur when a virtual person 601 travels in the virtual environment of the iconic city tower of the 5A scenic area during testing, according to the AR software design. An AR navigation arrow 702 is displayed on the ground of road 602, and the virtual person travels in the direction of arrow 702. When the virtual person 601 is about to reach an intersection, an AR image, specifically a floating virtual flag 701, is displayed in space. As an example, when the virtual person sees flag 701, it means that they are about to reach the AR navigation destination (e.g., the top floor of the city tower).

[0095] In order to test whether the AR software is operating normally, the AR effects that need to be confirmed this time are mainly: whether the virtual person can see the AR navigation arrow 702 on the ground of the road 602, and whether the flag 701 floating in the air can be displayed normally when the virtual person arrives at the intersection.

[0096] Therefore, according to the AR testing solution of the embodiment of the present application, the tester can confirm the AR effect by observing the screen of the mobile phone bound to the virtual person 601. Figure 6 Correspondingly, Figure 7 The screen of the mobile phone bound to the virtual person 601 is schematically shown. As can be seen, Figure 7The image shows a partial or full field of view from the virtual person's perspective. A virtual direction arrow 702' is superimposed on the road 602', and a flag 701' can be seen floating above the intersection. This demonstrates that the AR imagery designed by the AR software is functioning properly and that the test met expectations.

[0097] Among them, it is noted Figure 6 The virtual indicator arrow 702 in Figure 7 The patterns of the virtual indicator arrows 702' in the image are not exactly the same. For such design and performance inconsistency issues, engineers can correct the program instructions of the AR software after testing to ensure that the AR effect can be fully and accurately displayed to users.

[0098] The above process corresponds to Figure 5 The control system obtains the position information and posture information of the virtual person bound to the mobile phone 1 according to the motion capture acquisition module, and then renders the AR navigation logo and other AR content based on the current data and displays them on the mobile phone 1. In this embodiment, the virtual indicator arrow 702' and the floating flag 701' are displayed. Moreover, after these AR images are displayed normally, the mobile phone 1 can send an instruction message to the control system to confirm the AR effect. In addition, the image displayed on the screen of the mobile phone 1 can be synchronously displayed on the visual interface to provide an AR effect display, which is convenient for the test personnel to verify and confirm the test effect.

[0099] From the process described above, it can be seen that when using the embodiment of the present application to perform AR testing, there is no need for testers to arrive at the test destination. Instead, the test can be completed by testers remotely operating at a different location, which can effectively solve the problem of high time and labor costs caused by real-scene testing.

[0100] In addition, during the test, all images of AR effects appearing on the mobile phone 1 can be recorded, so that the difference in AR effects can be directly compared when testing again.

[0101] As an example, a recording and playback module can also be set up to record the communication information between the server and the mobile phone when the virtual person 601 performs the test task, forming a script. When the test needs to be performed again, the script can be played back to analyze the test results.

[0102] In this way, during another test, after screening the case samples, it is determined that mobile phone 2 is used as the test terminal. The model of mobile phone 2 is different from that of mobile phone 1. Therefore, the specific display plan of the configured test task may be different from the test configuration of the previous test using mobile phone 1. By directly comparing the AR image effects or running the script of the recording and playback module, the testers can quickly and easily discover the differences in AR effect display of different models of terminal devices, which is convenient for engineers to correct program instructions in a targeted manner.

[0103] In addition, similar differences may exist between different product forms. For example, the display mode on the mobile app is likely to be different from that in the mini program, and there may also be differences in display effects between different versions of the product. Multiple tests can be performed based on any of the above methods. On the basis of realizing AR remote testing, it is helpful to solve problems in regression testing and compatibility testing.

[0104] Corresponding to the method provided in this application, this application also provides an AR testing device. Figure 8 FIG. 1 shows a schematic diagram of the structure of an embodiment of an object data processing device provided by the present application. Figure 8 As shown, the AR testing device 100 includes:

[0105] The transceiver module 110 is configured to receive test configuration information sent by a first terminal device, where the test configuration information includes first destination information, and the location of the first terminal device does not belong to the first destination;

[0106] A generating module 120, configured to generate an AR test task and a virtual target object according to the test configuration information, wherein the virtual target object is used to execute the AR test task in a virtual environment corresponding to the first destination;

[0107] The display processing module 130 is configured to display a portion or all of the field of view of the virtual target object itself on the screen of the first terminal device during the process of the virtual target object performing the AR test task;

[0108] The determination module 140 is configured to determine whether the current AR application meets the test requirements when a specified AR image appears on the screen of the first terminal device.

[0109] Corresponding to the method provided in this application, this application also provides an AR testing device. Figure 9 FIG. 1 shows a schematic diagram of the structure of an embodiment of an object data processing device provided by the present application. Figure 9 As shown, the AR testing device 200 includes:

[0110] The transceiver processing module 210 is configured to receive externally input test configuration information and transmit it to a server in the operating environment of the AR application to be tested; wherein the test configuration information includes first destination information, the location of the first terminal device does not belong to the first destination, and the test configuration information is used to enable the server to generate an AR test task and a virtual target object, wherein the virtual target object is used to execute the AR test task in the virtual environment corresponding to the first destination;

[0111] The display processing module 220 is configured to display part or all of the field of view of the virtual target object itself on the screen during the process of the virtual target object performing the AR test task;

[0112] The transceiver processing module 210 is further configured to send information related to the AR image appearing on the screen to the server.

[0113] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0114] Those skilled in the art should understand that the embodiments described herein are preferred embodiments, and the actions, steps, modules, or units involved are not necessarily required for the embodiments of the present application. In the above embodiments, the embodiments of the present application have different focuses on the description of each embodiment. For parts not described in detail in a particular embodiment, please refer to the relevant description of other embodiments.

[0115] Figure 10 This is a structural diagram of an electronic device 10 provided in an embodiment of the present application. The electronic device 10 includes a processor 11, a memory 12, and a communication bus for connecting the processor 11 and the memory 12, wherein the memory 12 stores a computer program that can be run on the processor 11, and when the processor 11 runs the computer program, it can execute or implement the steps in the method of each embodiment of the present application. The electronic device 10 can be a server in the embodiment of the present application, and the electronic device 10 can also be a cloud server. The electronic device 10 can also be a terminal device or an AR device in the embodiment of the present application. Under appropriate circumstances, the electronic device can also be referred to as a computing device. The electronic device 10 can also be a cloud server. The electronic device 10 also includes a communication interface for receiving and sending data.

[0116] In some embodiments, the processor 11 may be a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), a modem processor, an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural network processing unit (NPU), or the like. The processor 11 may also be another general-purpose processor, an application-specific integrated circuit (ASIC), an off-the-shelf field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. A general-purpose processor may be a microprocessor or any conventional processor. The neural network processor (NPU), drawing on the structure of biological neural networks, can rapidly process input information and continuously self-learn. The NPU electronic device 10 can implement intelligent cognitive applications such as image recognition, facial recognition, semantic recognition, speech recognition, and text comprehension.

[0117] In some embodiments, the memory 12 may be an internal storage unit of the electronic device 10, such as the hard drive or memory of the electronic device 10. The memory 12 may also be an external storage device of the electronic device 10, such as a plug-in hard drive, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. equipped on the electronic device 10. The memory 12 may also include both the internal storage unit of the electronic device 10 and an external storage device. The memory 12 may be used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of a computer program. The memory 12 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). The memory 12 is used to store program code executed by the electronic device 10 and transmitted data. The memory 12 may also be used to temporarily store data that has been output or is about to be output.

[0118] Those skilled in the art will understand that Figure 10 This is only an example of the electronic device 10 and does not constitute a limitation on the electronic device 10. The electronic device 10 may include more or fewer components than shown in the figure, or combine certain components, or include different components. For example, it may also include input and output devices, network access devices, etc.

[0119] Figure 11 Schematic diagram of the software structure of the terminal device of the embodiment of the present application. Taking the Android system as an example, in some embodiments, the Android system is divided into four layers: application layer, application framework layer (framework, FWK), system layer and hardware abstraction layer, and the layers communicate with each other through software interfaces.

[0120] First, the application layer can include multiple application packages. The application packages can be various application apps such as calls, cameras, videos, navigation, weather, instant messaging, education, etc., or application apps based on AR technology.

[0121] Second, the application framework layer FWK provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer may include some predefined functions, such as a function for receiving events sent by the application framework layer.

[0122] The application framework layer may include a window manager, a resource manager, and a notification manager, etc.

[0123] The window manager manages windowed applications. It can obtain the display size, determine whether a status bar is present, lock the screen, and take screenshots. The content provider stores and retrieves data and makes it accessible to applications. This data can include video, images, audio, incoming and outgoing calls, browsing history and bookmarks, and the phone book.

[0124] Among them, the resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc.

[0125] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads, message reminders, and more. The Notification Manager can also display notifications in the form of icons or scrolling text in the system's top status bar, such as notifications from background applications, or in the form of dialog windows on the screen. Examples include displaying text messages in the status bar, sounding notifications, vibrating electronic devices, and flashing indicator lights.

[0126] The application framework layer also includes a view system, which includes visual controls, such as controls for displaying text and images. The view system can be used to build applications. A display interface can consist of one or more views. For example, the display interface of a text notification icon can include a view for displaying text and a view for displaying images.

[0127] Third, the system layer can include multiple functional modules, such as sensor service module, physical state recognition module, 3D graphics processing library (e.g. OpenGLES), etc.

[0128] Among them, the sensor service module is used to monitor the sensor data uploaded by various sensors at the hardware layer and determine the physical state of the mobile phone; the physical state recognition module is used to analyze and recognize user gestures, faces, etc.; the 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc.

[0129] In addition, the system layer can also include a surface manager and a media library. The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files.

[0130] Finally, the hardware abstraction layer is the layer between hardware and software. The hardware abstraction layer can include display drivers, camera drivers, sensor drivers, etc., which are used to drive the related hardware of the hardware layer, such as displays, cameras, sensors, etc.

[0131] An embodiment of the present application also provides a computer-readable storage medium storing a computer program or instructions, which, when executed, implements the steps in the method designed in the above embodiment.

[0132] The present application also provides a computer program product, including a computer program or instructions, which, when executed, implement the steps of the method designed in the above embodiment. Exemplarily, the computer program product can be a software installation package.

[0133] Those skilled in the art will appreciate that the methods, steps, or functions of the related modules / units described in the embodiments of the present application may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it may be implemented in whole or in part in the form of a computer program product, or it may be implemented by a processor executing a computer program instruction. Wherein, the computer program product includes at least one computer program instruction, and the computer program instruction may be composed of corresponding software modules, which may be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, mobile hard disks, read-only compact discs (CD-ROMs), or any other form of storage medium well known in the art. The computer program instruction may be stored in a computer-readable storage medium, or may be transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program instruction may be transmitted from a website, computer, server, or data center to another website, computer, server, or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access, or may be a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (eg, an SSD).

[0134] Regarding the various devices / products described in the above embodiments, the modules / units contained therein may be software modules / units, hardware modules / units, or may be partially software modules / units and partially hardware modules / units. For example, for a device / product applied to or integrated into a chip, the various modules / units contained therein may all be implemented in the form of hardware such as circuits, or at least some of the modules / units may be implemented in the form of software programs, running on a processor integrated within the chip, while the remaining modules / units may be implemented in the form of hardware such as circuits. For another example, for a device / product applied to or integrated into a terminal, the various modules / units contained therein may all be implemented in the form of hardware such as circuits, or at least some of the modules / units may be implemented in the form of software programs, running on a processor integrated within the terminal, while the remaining modules / units may be implemented in the form of hardware such as circuits.

[0135] It should be understood that the scope of protection of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the scope of protection of the present application.

Claims

1. An augmented reality (AR) testing method, applied to a server, characterized in that: The AR testing method includes: The server receives test configuration information sent by the first terminal device, where the test configuration information includes first destination information, and the location of the first terminal device does not belong to the first destination; The server generates an AR test task and a virtual target object according to the test configuration information, wherein the virtual target object is used to execute the AR test task in the virtual environment corresponding to the first destination; During the process of the virtual target object performing the AR test task, the server displays part or all of the field of view of the virtual target object itself on the screen of the first terminal device; When a specified AR image appears on the screen of the first terminal device, the server determines that the current AR application meets the test requirements.

2. The method according to claim 1, characterized in that Also includes: Binding the first terminal device to the virtual target object; During the process of the virtual target object performing the test task, the position and / or posture of the virtual target object in the virtual environment is adjusted by adjusting the position and / or posture of the first terminal device.

3. The method according to claim 1, characterized in that Also includes: During the process of the virtual target object performing the AR test task, first indication information sent by the first terminal device is received, which is used to indicate that a specified AR image has appeared on the screen of the first terminal device.

4. The method according to claim 3, characterized in that The first indication information includes at least one of the following: a character string in a predetermined format, a data frame carrying predetermined bits, and a message in a predetermined format.

5. The method according to claim 3, characterized in that The first indication information includes a screen image when a specified AR image appears on the screen of the first terminal device. The screen image is used to play back the specified AR image when testing again. If the same AR image appears on the screen of the terminal device when testing again, it is determined that the current AR application meets the test requirements.

6. The method according to claim 1, characterized in that Generating an AR test task and a virtual target object according to the test configuration information includes: generating a virtual environment corresponding to the first destination based on the modeling data of the first destination and the related panoramic image data; generating a virtual target object, wherein the virtual target object is located in the virtual environment; An AR test task is generated, where the AR test task includes moving the virtual target object from a first position to a second position along a specified route in the virtual environment.

7. The method according to claim 6, characterized in that Also includes: Setting a plurality of movable virtual auxiliary objects in the virtual environment; During the process of the virtual target object performing the AR test task, the positioning information of the virtual target object and the virtual auxiliary object is recorded.

8. The method according to claim 1, characterized in that The test configuration information also includes model information of the first terminal device.

9. The method according to claim 8, characterized in that Also includes: In an operating environment of the AR application to be tested, new test configuration information is received through a second terminal device, where the new test configuration information includes second destination information and model information of the second terminal device, the second terminal device is located at a location that does not belong to the second destination, the second destination is the same as the first destination, and the model of the second terminal device is different from that of the first terminal device; A new AR test task and a new virtual target object are generated according to the new test configuration information. After the new virtual target object executes the new test task, the difference information between the AR image appearing on the second terminal device and the AR image appearing on the first terminal device is determined. The difference information is used to modify the AR application.

10. The method according to claim 8, characterized in that Also includes: In an operating environment of the AR application to be tested, new test configuration information is received through a third terminal device, where the new test configuration information includes third destination information and model information of the third terminal device, the third terminal device is located at a location that does not belong to the third destination, the third destination is different from the first destination, and the model of the third terminal device is the same as that of the first terminal device; A new AR test task and a new virtual target object are generated according to the new test configuration information. After the new virtual target object executes the new test task, the difference information between the AR image appearing on the third terminal device and the AR image appearing on the first terminal device is determined, and the difference information is used to modify the AR application.

11. An augmented reality (AR) testing method, applied to a terminal device, characterized in that: The AR testing method includes: In an operating environment of an AR application to be tested, a first terminal device receives externally input test configuration information and sends it to a server; wherein the test configuration information includes first destination information, the location of the first terminal device does not belong to the first destination, and the test configuration information is used to enable the server to generate an AR test task and a virtual target object, and the virtual target object is used to execute the AR test task in a virtual environment corresponding to the first destination; During the process of the virtual target object performing the AR test task, the first terminal device displays part or all of the field of view of the virtual target object itself on the screen; The first terminal device sends relevant information of the AR image appearing on the screen to the server.

12. The method according to claim 11, characterized in that Also includes: The first terminal device is bound to the virtual target object; During the process of the virtual target object performing the AR test task, the position and / or posture of the virtual target object in the virtual environment is adjusted by adjusting the position and / or posture of the first terminal device.

13. The method according to claim 11, characterized in that Also includes: During the process of the virtual target object performing the AR test task, if a specified AR image appears on the screen of the first terminal device, the first terminal device sends a first indication message to the server to indicate that the specified AR image has appeared on the screen of the first terminal device.

14. The method according to claim 13, characterized in that The first indication information includes at least one of the following: a character string in a predetermined format, a data frame carrying predetermined bits, and a message in a predetermined format.

15. The method according to claim 13, characterized in that The first indication information includes a screen image when a specified AR image appears on the screen of the first terminal device. The screen image is used to play back the specified AR image when testing again. If the same AR image appears on the screen of the terminal device when testing again, it is determined that the current AR application meets the test requirements.

16. The method according to claim 11, characterized in that The test configuration information also includes model information of the first terminal device.

17. The method according to claim 16, characterized in that Also includes: In the operating environment of the AR application to be tested, the second terminal device receives new test configuration information and sends it to the server, where the new test configuration information includes second destination information and model information of the second terminal device, the second terminal device is located at a location that does not belong to the second destination, the second destination is the same as the first destination, and the model of the second terminal device is different from that of the first terminal device; After the server generates a new AR test task and a new virtual target object according to the new test configuration information, when the new virtual target object performs the new AR test task, the second terminal device displays part or all of the field of view of the new virtual target object itself on the screen; The second terminal device determines difference information between the AR image appearing on its screen and the AR image appearing on the screen of the first terminal device, and the difference information is used to modify the AR application.

18. The method according to claim 16, characterized in that Also includes: In the operating environment of the AR application to be tested, a third terminal device receives new test configuration information and sends it to the server, where the new test configuration information includes third destination information and model information of the third terminal device, the location of the third terminal device does not belong to the third destination, the third destination is different from the first destination, and the model of the third terminal device is the same as that of the first terminal device; After the server generates a new AR test task and a new virtual target object according to the new test configuration information, when the new virtual target object performs the new AR test task, the third terminal device displays part or all of the field of view of the new virtual target object itself on the screen; The third terminal device determines difference information between the AR image appearing on its own screen and the AR image appearing on the screen of the first terminal device, where the difference information is used to modify the AR application.

19. An augmented reality (AR) testing device, characterized in that: include: a transceiver processing module, configured to receive, by a first terminal device, test configuration information input externally and send the information to a server in an operating environment of an AR application to be tested; wherein the test configuration information includes first destination information, and the location of the first terminal device does not belong to the first destination; A generation module, configured for a server to generate an AR test task and a virtual target object according to the test configuration information, wherein the virtual target object is used to execute the AR test task in a virtual environment corresponding to the first destination; A display processing module is configured to, during the process of the virtual target object performing the AR test task, display a portion or all of the field of view of the virtual target object itself on the screen of the first terminal device by the server; The determination module is used to determine, by the server, whether the current AR application meets the test requirements when a specified AR image appears on the screen of the first terminal device.

20. A server device, characterized in that: include: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method according to any one of claims 1 to 10 is implemented.

21. A terminal device, characterized in that: include: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method according to any one of claims 11 to 18 is implemented.

22. A computer-readable storage medium, characterized in that Computer program instructions are stored on the computer storage medium, and when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 18 is implemented.

23. A computer program product, characterized in that The method comprises computer program instructions, which implement the method according to any one of claims 1 to 18 when executed by a processor.

Images