Manual space annotation method, system, electronic device and storage medium

Abstract

Embodiments of the present invention relate to the field of remote assistance technology and disclose a method, system, electronic device, and storage medium for manual space annotation. The method includes: capturing a video image and recording the location information of the video image; sending the video image and the corresponding location information to an assistant expert terminal, receiving the annotation information of the video image by the assistant expert; and displaying the annotation information on a display screen in real time through calculation based on the annotation information. By implementing the embodiments of the present invention, only the annotation information can be sent to the AR device for display on the display screen, which improves transmission efficiency and ensures real-time performance.

Description

Technical Field

[0001] The present invention relates to the field of remote assistance technology, and in particular to an artificial space annotation method, system, electronic device and storage medium based on augmented reality. Background Art

[0002] Existing remote assistance generally relies on AR devices and expert terminals. The images transmitted in real time by the AR device are received by the expert terminal, and the expert annotates the image accordingly, such as adding a box to mark the location of the problem and / or other auxiliary text, and then sends the image and annotation content to the AR device for on-site personnel to refer to and solve the corresponding faults.

[0003] This method has gradually matured, but it has a serious problem. After each annotation, the image and the annotation content must be sent to the AR device. On the one hand, it is affected by the transmission efficiency and the real-time performance is relatively poor. On the other hand, it cannot adapt to the position adjustment of the AR device. That is, when the on-site personnel rotate a certain angle, the annotation box cannot adapt to its direction after rotation. That is, the on-site personnel have to find the object corresponding to the annotation box based on the transmitted image and annotation content. Especially in emergency situations, the defects of this method are further magnified. Summary of the Invention

[0004] To address the above-mentioned drawbacks, embodiments of the present invention disclose a manual space annotation method, system, electronic device, and storage medium, which only require the transmission of certain annotation information, thereby improving transmission efficiency.

[0005] A first aspect of an embodiment of the present invention discloses a manual space annotation method, the method comprising:

[0006] Capturing video images and recording location information of the video images;

[0007] Sending the video image and the corresponding location information to the auxiliary expert terminal, and receiving the annotation information of the video image by the auxiliary expert;

[0008] The annotation information is displayed in real time on a display screen through calculation based on the annotation information.

[0009] As a preferred embodiment, in the first aspect of the embodiment of the present invention, capturing a video image and recording the position information of the video image includes:

[0010] Capture video images of the space where the AR device is located through a camera;

[0011] Obtaining the real-time three-dimensional world coordinates of the space where the AR device is located, the coordinate conversion matrix between the AR device shooting coordinate system and the AR device world coordinate system, the real-time spatial model data of the space where the AR device is located, and the shooting coordinate system coordinates of the four corners of the video image on the rear clipping surface;

[0012] The real-time three-dimensional world coordinates of the space where the AR device is located, the coordinate conversion matrix between the AR device shooting coordinate system and the AR device world coordinate system, and the shooting coordinate system coordinates of the four corners of the video image on the rear clipping surface are defined as the position information of the video image.

[0013] As a preferred embodiment, in the first aspect of the embodiment of the present invention, capturing a video image and recording the position information of the video image includes:

[0014] A video image is captured at a preset time interval, each video image and its position information are combined to form a data packet, and a capture time corresponding to the video image is set for the data packet.

[0015] As a preferred embodiment, in the first aspect of the embodiment of the present invention, sending the video image and the corresponding position information to the auxiliary expert terminal, and receiving the auxiliary expert's annotation information on the video image, includes:

[0016] The auxiliary expert terminal automatically generates annotation information according to the annotation content of the video image by the auxiliary expert. The annotation information is one or more of an annotation frame, text added at the annotation position, and corresponding content in the read video image.

[0017] As a preferred embodiment, in the first aspect of the embodiment of the present invention, when the annotation information is a annotation box;

[0018] The auxiliary expert terminal automatically generates annotation information based on the annotation content of the video image by the auxiliary expert, including:

[0019] The auxiliary expert terminal calculates the world coordinate system coordinates of the four corners of the video image based on the shooting coordinate system coordinates of the four corners on the rear clipping surface:

[0020] PW i =PC i ×Trans

[0021] Among them, PW i is the world coordinate system coordinate of the i-th corner, PC i is the coordinate of the i-th corner in the shooting coordinate system on the back shearing plane, and Trans is the coordinate transformation matrix between the AR device shooting coordinate system and the AR device world coordinate system;

[0022] The auxiliary expert terminal calculates the world coordinates of the center of the annotation box based on the shooting coordinates of the AR device in the plane where the four corners of the video image are located:

[0023] CW=CC×Trans

[0024] Where CW is the world coordinate system coordinate of the center position of the annotation box in the plane where the four corners of the video image are located, and CC is the shooting coordinate system coordinate of the AR device;

[0025] The auxiliary expert terminal automatically generates annotation information based on the world coordinate system coordinates of the four corners, the world coordinate system coordinates of the center position of the annotation box on the plane where the four corners of the video image are located, and the shape and size of the annotation box.

[0026] As a preferred embodiment, in the first aspect of the embodiment of the present invention, displaying the annotation information on a display screen in real time through calculation based on the annotation information includes:

[0027] Calculate the spatial straight line equations of the four corners of the video image based on the world coordinates of the center of the marked box and the world coordinates of the AR device.

[0028] Calculate the world coordinate system coordinates of the object surface corresponding to the center position of the annotation box according to the spatial straight line equation and the real-time spatial model data of the space where the AR device is located;

[0029] The world coordinate system coordinates of the object surface corresponding to the center position of the annotation frame and the shape and size of the annotation frame are input into the world coordinate system display module and displayed on the display screen of the AR device.

[0030] A second aspect of an embodiment of the present invention provides an AR device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the manual space annotation method described in the first aspect of the embodiment of the present invention.

[0031] A third aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute the manual space annotation method described in the first aspect of the embodiment of the present invention.

[0032] A fourth aspect of an embodiment of the present invention provides a manual space annotation method, including:

[0033] Receive a video image captured by an AR device and location information of the video image;

[0034] Based on the annotation content of the video image by the auxiliary expert, obtaining annotation information corresponding to the video image in real time;

[0035] The annotation information is sent to the AR device, so that the AR device displays the annotation information in real time on a display screen through calculation based on the annotation information.

[0036] As a preferred embodiment, in the fourth aspect of the embodiments of the present invention, receiving a video image captured by an AR device and location information of the video image includes:

[0037] Receive a video image of the space where the AR device is located, captured by the AR device through a camera on the AR device;

[0038] And receive the position information corresponding to the video image, the position information including the real-time three-dimensional world coordinates of the space where the AR device is located, the coordinate conversion matrix between the AR device shooting coordinate system and the AR device world coordinate system, and the shooting coordinate system coordinates of the four corners of the video image on the rear clipping surface.

[0039] As a preferred embodiment, in the fourth aspect of the embodiments of the present invention, the annotation information corresponding to the video image is calculated in real time based on the annotation content of the video image by the auxiliary expert, including:

[0040] The annotation content is a annotation box;

[0041] Calculate the world coordinate system coordinates of the four corners of the video image based on the shooting coordinate system coordinates of the four corners on the rear clipping surface:

[0042] PW i =PC i ×Trans

[0043] Among them, PW i is the world coordinate system coordinate of the i-th corner, PC i is the coordinate of the i-th corner in the shooting coordinate system on the back shearing plane, and Trans is the coordinate transformation matrix between the AR device shooting coordinate system and the AR device world coordinate system;

[0044] The center position of the annotation box is calculated based on the shooting coordinate system coordinates of the AR device, and the world coordinate system coordinates of the plane where the four corners of the video image are located are:

[0045] CW=CC×Trans

[0046] Where CW is the world coordinate system coordinate of the center position of the annotation box in the plane where the four corners of the video image are located, and CC is the shooting coordinate system coordinate of the AR device;

[0047] The annotation information is automatically generated based on the world coordinate coordinates of the four corners, the center position of the annotation box, the world coordinate coordinates of the plane where the four corners of the video image are located, and the shape and size of the annotation box.

[0048] As a preferred embodiment, in a fourth aspect of the embodiments of the present invention, sending the annotation information to an AR device so that the AR device displays the annotation information in real time on a display screen through calculation based on the annotation information includes:

[0049] Sending the annotation information to the AR device;

[0050] The AR device calculates the spatial straight line equations of the four corners of the video image based on the world coordinate system coordinates of the plane where the center position of the marked box is located and the world coordinate system coordinates of the AR device.

[0051] The AR device calculates the world coordinate system coordinates of the object surface corresponding to the center position of the annotation box based on the spatial straight line equation and the real-time spatial model data of the space where the AR device is located;

[0052] The AR device inputs the world coordinate coordinates of the object surface corresponding to the center position of the annotation box and the shape and size of the annotation box into the world coordinate display module, and displays them on the display screen of the AR device.

[0053] The fifth aspect of the present invention provides an auxiliary expert terminal, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute a manual space annotation method described in the fourth aspect of an embodiment of the present invention.

[0054] A sixth aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the manual space annotation method described in the fourth aspect of an embodiment of the present invention.

[0055] A seventh aspect of an embodiment of the present invention provides a manual space annotation system, the system comprising an AR device and an auxiliary expert terminal;

[0056] The AR device is used to capture video images and record location information of the video images; and send the video images and corresponding location information to the auxiliary expert terminal;

[0057] The auxiliary expert terminal is used to receive the video image and the corresponding position information, calculate the annotation information according to the annotation content of the video image by the auxiliary expert and the position information of the video image, and send the annotation content to the AR device;

[0058] The AR device is further configured to display the annotation information in real time on a display screen of the AR device through calculation based on the annotation information.

[0059] An eighth aspect of an embodiment of the present invention discloses a computer program product. When the computer program product runs on a computer, the computer executes a manual space annotation method disclosed in the first aspect or the fourth aspect of the embodiment of the present invention.

[0060] A ninth aspect of an embodiment of the present invention discloses an application publishing platform, which is used to publish a computer program product. When the computer program product runs on a computer, the computer executes an artificial space annotation method disclosed in the first aspect or the fourth aspect of the embodiment of the present invention.

[0061] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0062] The embodiment of the present invention only sends the annotation information to the AR device for display on the display screen, thereby improving transmission efficiency and ensuring real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0064] Figure 1 This is a flow chart of a manual space annotation method disclosed in the first embodiment of the present invention;

[0065] Figure 2 is a schematic diagram of the structure of video image related coordinates disclosed in an embodiment of the present invention;

[0066] Figure 3 This is a flow chart of a manual space annotation method disclosed in the second embodiment of the present invention;

[0067] Figure 4 This is a schematic diagram of the structure of a manual space annotation system disclosed in the third embodiment of the present invention;

[0068] Figure 5 This is a structural diagram of an electronic device disclosed in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0069] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0070] It should be noted that the terms "first," "second," "third," "fourth," etc. in the description and claims of the present invention are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having," as well as any variations thereof, in the embodiments of the present invention, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0071] Embodiments of the present invention disclose a manual space annotation method, system, electronic device, and storage medium. On the one hand, these methods only send annotation information to an AR device for display on a display screen, improving transmission efficiency and ensuring real-time performance. On the other hand, they can adjust the position of the annotation box based on the position of the AR device. A detailed description is provided below in conjunction with the accompanying drawings.

[0072] Example 1

[0073] See also Figure 1 , Figure 1 This is a flow chart of a manual space annotation method disclosed in an embodiment of the present invention. The manual space annotation method acts on an AR device, that is, the execution subject is the AR device and related components or associated devices or software in the AR device. Figure 1 As shown, the manual space annotation method includes the following steps:

[0074] S110: Capture video images and record location information of the video images.

[0075] Because the video images captured by the AR device need to be sent to the auxiliary expert terminal for manual annotation by the auxiliary expert, in a preferred embodiment of the present invention, the AR device explores whether there is a connectable auxiliary expert terminal. If so, the user of the AR device operates the AR device to select an auxiliary expert terminal for connection application. After the connection is successful, information can be transmitted between each other, including images, text, etc., and of course, voice, etc.

[0076] The AR device can capture a video image through its video acquisition unit, such as a camera, and obtain the corresponding position information of the video image. Of course, in some other embodiments, the video acquisition unit of the AR device, such as a camera, can also perform uninterrupted data acquisition. For example, it can capture a video image at a preset interval, such as 100ms, and package the corresponding position information of the captured video image with it to form a data packet. The video image acquisition time is set for the data packet, and the data packet is sent to the auxiliary expert terminal in the form of a data packet.

[0077] The processing of uninterrupted image acquisition is basically similar to that of acquiring only one image and sending it to the auxiliary expert terminal. The only difference is that during uninterrupted image acquisition, the auxiliary expert terminal can play the video stream formed by the uninterrupted image acquisition and capture a video image from the video stream. Since each video image corresponds to a data packet, its location information is also determined accordingly.

[0078] Here, taking the acquisition of a video image as an example, the real-time three-dimensional world coordinates of the space where the AR device is located, the coordinate conversion matrix between the AR device shooting coordinate system and the AR device world coordinate system, the real-time spatial model data of the space where the AR device is located, and the shooting coordinate system coordinates of the four corners of the video image on the rear clipping surface are obtained.

[0079] The spatial model data in the real-time spatial model data of the space where the AR device is located refers to the surface model data of all objects in the space where the AR device is located that are not transparent to visible light. This data is collected by the spatial scanning unit of the AR device at the AR device shooting point.

[0080] The coordinate conversion matrix between the AR device's shooting coordinate system and the AR device's world coordinate system can be determined based on the camera's parameters. The real-time 3D world coordinates of the AR device's space can be acquired using the device's integrated sensors. The shooting coordinate system coordinates of the four corners of the video image on the back clipping plane are manually set.

[0081] In the above data, the real-time three-dimensional world coordinates of the space where the AR device is located, the coordinate conversion matrix between the AR device shooting coordinate system and the AR device world coordinate system, and the shooting coordinate system coordinates of the four corners of the video image on the rear clipping surface are defined as the position information of the video image and sent to the auxiliary expert terminal together with the video image.

[0082] S120: Send the video image and the corresponding position information to the auxiliary expert terminal, and receive the annotation information of the video image by the auxiliary expert.

[0083] After receiving the video image, the user of the auxiliary expert terminal, i.e. the auxiliary expert, makes relevant annotations on the video image. The annotation content includes but is not limited to: marking two-dimensional boundaries with a certain shape, adding text at the marked position, reading numbers from the picture within the shape area, etc.

[0084] When the auxiliary expert completes the relevant annotation, the auxiliary expert terminal will calculate according to the annotation content and position information to obtain the annotation information of the video image.

[0085] Specifically, first, the calculation processing unit of the auxiliary expert terminal calculates the world coordinate system coordinates of the four corners of the video image (i.e., Figure 2 The world coordinate system coordinates of P1-P4 in the image are calculated using the following formula:

[0086] PW i =PC i ×Trans

[0087] Among them, PW i is the world coordinate system coordinate of the i-th corner, PC i is the shooting coordinate system coordinate of the i-th corner on the back clipping plane, and Trans is the coordinate transformation matrix between the AR device shooting coordinate system and the AR device world coordinate system.

[0088] Then, the calculation processing unit of the auxiliary expert terminal marks the world coordinate system coordinates of the center position of the frame on the plane where the four corners of the video image are located.

[0089] CW=CC×Trans

[0090] Among them, CW is the world coordinate system coordinate of the center position of the annotation box in the plane where the four corners of the video image are located, and CC is the shooting coordinate system coordinate of the AR device.

[0091] Finally, the auxiliary expert terminal sends the calculated world coordinate system coordinates of the four corners of the video image, the world coordinate system coordinates of the center position of the annotation box on the plane where the four corners of the video image are located, and the shape and size of the annotation box to the AR device. If there is other annotation content, it will also be sent together, such as added text.

[0092] S130: Display the annotation information in real time on a display screen through calculation based on the annotation information.

[0093] First, the AR device uses the center position of the annotation box to mark the world coordinates of the plane where the four corners of the video image are located (i.e. Figure 2 Q2 in the AR device) and the real-time 3D world coordinates of the shooting point in the space where the AR device is located (i.e. Figure 2Calculate the equation of the line connecting the two points using Q1 in the previous example. Based on this equation and the real-time spatial model data of the space where the AR device resides, calculate the world coordinate system coordinates of the object surface corresponding to the center of the annotation box.

[0094] A typical calculation method is to input the coordinate data of each point scanned by the spatial model into the straight line equation so that the corresponding coordinates that make the equation valid are the world coordinate system coordinates of the object surface corresponding to the center of the required annotation box.

[0095] Then, based on the world coordinate coordinates of the object surface corresponding to the center of the annotation box, it, along with the added text (if any) and the shape and size of the proportionally scaled annotation box, are input into the world coordinate display module and displayed on the AR device display.

[0096] The world coordinate display module can also correct the display position of the annotation content (annotation box and / or added text) on the AR device display in real time based on the received world coordinate coordinates of the object surface corresponding to the center of the annotation box and the real-time 3D world coordinates of the AR device shooting point in space. A typical world coordinate display module is the world coordinate display module included with Microsoft HoloLen 2.

[0097] As can be seen from the above, when the real-time 3D world coordinates of the shooting point of the AR device in the space where it is located change, the display position of the annotation content on the display screen also changes accordingly.

[0098] If the auxiliary expert terminal no longer sends new annotation information, the annotation information sent previously will continue to be presented on the display screen of the AR device (an example is that if the real-time three-dimensional world coordinates of the AR device shooting point in the space deviate from the world coordinate system coordinates of the object surface corresponding to the center of the annotation box, the annotation box will disappear, and then adjust the real-time three-dimensional world coordinates of the AR device shooting point in the space to match the world coordinate system coordinates of the object surface corresponding to the center of the annotation box, and the annotation box will reappear). If the auxiliary expert terminal sends new annotation information again, the previous annotation information of the AR device will be overwritten by the new annotation information, and then the new annotation content will be presented in the AR device by recalculation.

[0099] Example 2

[0100] See also Figure 3 , Figure 3 This is a flow chart of a manual space annotation method disclosed in an embodiment of the present invention. The manual space annotation method acts on an auxiliary expert terminal, that is, the execution subject is the auxiliary expert terminal, which can be a server with certain processing capabilities, or a tablet computer, mobile phone, ordinary desktop computer or laptop computer, etc. In addition to hardware, it is also equipped with relevant software. Figure 3 As shown, the manual space annotation method includes the following steps:

[0101] S210: Receive a video image captured by an AR device and location information of the video image.

[0102] The auxiliary expert terminal receives a video image of the space where the AR device is located, which is captured by the camera on the AR device; and receives position information corresponding to the video image, wherein the position information includes the real-time three-dimensional world coordinates of the space where the AR device is located, the coordinate conversion matrix between the AR device shooting coordinate system and the AR device world coordinate system, and the shooting coordinate system coordinates of the four corners of the video image on the rear clipping surface. The process corresponds to step S110 in Example 1.

[0103] S220 , obtaining annotation information corresponding to the video image by real-time calculation based on the annotation content of the video image by the auxiliary expert.

[0104] When the annotation content includes an annotation frame, the world coordinate system coordinates of the four corners of the video image are calculated based on the shooting coordinate system coordinates of the four corners on the rear clipping surface:

[0105] PW i =PC i ×Trans

[0106] Among them, PW i is the world coordinate system coordinate of the i-th corner, PC i is the coordinate of the i-th corner in the shooting coordinate system on the back shearing plane, and Trans is the coordinate transformation matrix between the AR device shooting coordinate system and the AR device world coordinate system;

[0107] The center position of the annotation box is calculated based on the shooting coordinate system coordinates of the AR device, and the world coordinate system coordinates of the plane where the four corners of the video image are located are:

[0108] CW=CC×Trans

[0109] Where CW is the world coordinate system coordinate of the center position of the annotation box in the plane where the four corners of the video image are located, and CC is the shooting coordinate system coordinate of the AR device;

[0110] The annotation information is automatically generated based on the world coordinate coordinates of the four corners, the center position of the annotation box, the world coordinate coordinates of the plane where the four corners of the video image are located, and the shape and size of the annotation box.

[0111] The process corresponds to step S120 in the first embodiment.

[0112] S230: Send the annotation information to the AR device, so that the AR device displays the annotation information in real time on a display screen through calculation based on the annotation information.

[0113] The AR device calculates the spatial straight line equations of the two based on the world coordinate system coordinates of the plane where the four corners of the video image are located and the world coordinate system coordinates of the AR device based on the center position of the annotation box; the AR device calculates the world coordinate system coordinates of the object surface corresponding to the center position of the annotation box based on the spatial straight line equation and the real-time spatial model data of the space where the AR device is located; the AR device inputs the world coordinate system coordinates of the object surface corresponding to the center position of the annotation box and the shape and size of the annotation box into the world coordinate system display module, and displays it on the display screen of the AR device.

[0114] This process corresponds to step S130 in the first embodiment.

[0115] Example 3

[0116] Example 3 discloses a manual space annotation system, please refer to Figure 4 As shown, the manual space annotation system includes: an AR device 310 and an auxiliary expert terminal 320; the AR device 310 and the auxiliary expert terminal 320 establish communication, and the information that can be communicated includes but is not limited to images, text, sound and other types of data such as location information data.

[0117] The AR device 310 is used to capture video images and record location information of the video images; and send the video images and corresponding location information to the auxiliary expert terminal;

[0118] The auxiliary expert terminal 320 is used to receive the video image and the corresponding position information, calculate the annotation information according to the annotation content of the video image by the auxiliary expert and the position information of the video image, and send the annotation content to the AR device;

[0119] The AR device 310 is further configured to display the annotation information in real time on a display screen of the AR device through calculation based on the annotation information.

[0120] Example 4

[0121] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. Figure 5 As shown, the electronic device may be an AR device, an auxiliary expert terminal, or a combination of the two. The electronic device may include:

[0122] A memory 410 storing executable program code;

[0123] a processor 420 coupled to the memory 410;

[0124] The processor 420 calls the executable program code stored in the memory 410 to execute part or all of the steps in a manual space annotation method in the first or second embodiment.

[0125] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute part or all of the steps in a manual space annotation method in embodiment one or two.

[0126] An embodiment of the present invention further discloses a computer program product, wherein when the computer program product is run on a computer, the computer is caused to execute part or all of the steps in a manual space annotation method in embodiment one or two.

[0127] An embodiment of the present invention further discloses an application publishing platform, wherein the application publishing platform is used to publish a computer program product, wherein when the computer program product is run on a computer, the computer executes some or all steps of a manual space annotation method in embodiment one or two.

[0128] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the processes does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0129] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the objectives of this embodiment as needed.

[0130] In addition, the functional units in the embodiments of the present invention may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The integrated unit may be implemented in the form of hardware or software functional units.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a memory and includes several requests for causing a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the method described in each embodiment of the present invention.

[0132] In the embodiments provided herein, it should be understood that "B corresponding to A" means that B is associated with A and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.

[0133] Those skilled in the art will appreciate that some or all of the steps in the various methods of the embodiments may be performed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0134] The above is a detailed introduction to an artificial space annotation method, system, electronic device and storage medium disclosed in an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A manual space annotation method, characterized in that: include: Capturing video images and recording location information of the video images; Sending the video image and the corresponding location information to the auxiliary expert terminal, and receiving the annotation information of the video image by the auxiliary expert; Displaying the annotation information in real time on a display screen through calculation based on the annotation information; Capturing video images and recording location information of the video images, including: Capture video images of the space where the AR device is located through a camera; Obtaining the real-time three-dimensional world coordinates of the space where the AR device is located, the coordinate conversion matrix between the AR device shooting coordinate system and the AR device world coordinate system, the real-time spatial model data of the space where the AR device is located, and the shooting coordinate system coordinates of the four corners of the video image on the rear clipping surface; The real-time three-dimensional world coordinates of the space where the AR device is located, the coordinate conversion matrix between the AR device shooting coordinate system and the AR device world coordinate system, and the shooting coordinate system coordinates of the four corners of the video image on the back clipping plane are defined as the position information of the video image; The method includes: sending the video image and the corresponding position information to the auxiliary expert terminal, and receiving the annotation information of the video image by the auxiliary expert, including: The auxiliary expert terminal automatically generates annotation information based on the annotation content of the video image by the auxiliary expert, wherein the annotation information is one or more of an annotation frame, text added at the annotation position, and corresponding content in the read video image; When the annotation information is a annotation box; The auxiliary expert terminal automatically generates annotation information based on the annotation content of the video image by the auxiliary expert, including: The auxiliary expert terminal calculates the world coordinate system coordinates of the four corners of the video image based on the shooting coordinate system coordinates of the four corners on the rear clipping surface: PW i =PC i ×Trans Among them, PW i is the world coordinate system coordinate of the i-th corner, PC i is the coordinate of the i-th corner in the shooting coordinate system on the back shearing plane, and Trans is the coordinate transformation matrix between the AR device shooting coordinate system and the AR device world coordinate system; The auxiliary expert terminal calculates the world coordinates of the center of the annotation box based on the shooting coordinates of the AR device in the plane where the four corners of the video image are located: CW=CC×Trans Where CW is the world coordinate system coordinate of the center position of the annotation box in the plane where the four corners of the video image are located, and CC is the shooting coordinate system coordinate of the AR device; The auxiliary expert terminal automatically generates annotation information based on the world coordinate system coordinates of the four corners, the world coordinate system coordinates of the center position of the annotation box on the plane where the four corners of the video image are located, and the shape and size of the annotation box.

2. The manual space annotation method according to claim 1, characterized in that: Capturing video images and recording location information of the video images, including: A video image is captured at a preset time interval, each video image and its position information are combined to form a data packet, and a capture time corresponding to the video image is set for the data packet.

3. The manual space annotation method according to claim 1, characterized in that: Displaying the annotation information in real time on a display screen through calculation based on the annotation information includes: Calculate the spatial straight line equations of the four corners of the video image based on the world coordinates of the center of the marked box and the world coordinates of the AR device. Calculate the world coordinate system coordinates of the object surface corresponding to the center position of the annotation box according to the spatial straight line equation and the real-time spatial model data of the space where the AR device is located; The world coordinate system coordinates of the object surface corresponding to the center position of the annotation frame and the shape and size of the annotation frame are input into the world coordinate system display module and displayed on the display screen of the AR device.

4. An AR device, characterized in that: include: a memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the manual space annotation method according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute the manual space annotation method according to any one of claims 1 to 3.

6. A manual space annotation method, characterized in that: include: Receive a video image captured by an AR device and location information of the video image; Based on the annotation content of the video image by the auxiliary expert, obtaining annotation information corresponding to the video image in real time; Sending the annotation information to the AR device, so that the AR device displays the annotation information in real time on a display screen through calculation based on the annotation information; Receiving a video image captured by an AR device and location information of the video image, including: Receive a video image of the space where the AR device is located, captured by the AR device through a camera on the AR device; and receiving position information corresponding to the video image, the position information including the real-time three-dimensional world coordinates of the space where the AR device is located, a coordinate conversion matrix between the AR device shooting coordinate system and the AR device world coordinate system, and the shooting coordinate system coordinates of the four corners of the video image on the back clipping plane; Based on the annotation content of the video image by the auxiliary expert, the annotation information corresponding to the video image is calculated in real time, including: The annotation content is a annotation box; Calculate the world coordinate system coordinates of the four corners of the video image based on the shooting coordinate system coordinates of the four corners on the rear clipping surface: PW i =PC i ×Trans Among them, PW i is the world coordinate system coordinate of the i-th corner, PC i is the coordinate of the i-th corner in the shooting coordinate system on the back shearing plane, and Trans is the coordinate transformation matrix between the AR device shooting coordinate system and the AR device world coordinate system; The center position of the annotation box is calculated based on the shooting coordinate system coordinates of the AR device, and the world coordinate system coordinates of the plane where the four corners of the video image are located are: CW=CC×Trans Where CW is the world coordinate system coordinate of the center position of the annotation box in the plane where the four corners of the video image are located, and CC is the shooting coordinate system coordinate of the AR device; The annotation information is automatically generated based on the world coordinate coordinates of the four corners, the center position of the annotation box, the world coordinate coordinates of the plane where the four corners of the video image are located, and the shape and size of the annotation box.

7. The manual space annotation method according to claim 6, characterized in that: The method includes: sending the annotation information to the AR device so that the AR device displays the annotation information in real time on a display screen through calculation based on the annotation information, including: Sending the annotation information to the AR device; The AR device calculates the spatial straight line equations of the four corners of the video image based on the world coordinate system coordinates of the plane where the center position of the marked box is located and the world coordinate system coordinates of the AR device. The AR device calculates the world coordinate system coordinates of the object surface corresponding to the center position of the annotation box based on the spatial straight line equation and the real-time spatial model data of the space where the AR device is located; The AR device inputs the world coordinate coordinates of the object surface corresponding to the center position of the annotation box and the shape and size of the annotation box into the world coordinate display module, and displays them on the display screen of the AR device.

8. An auxiliary expert terminal, characterized in that: include: a memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the manual space annotation method according to any one of claims 6 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute the manual space annotation method according to any one of claims 6 to 7.

10. A manual space annotation system, characterized in that: The system includes AR equipment and auxiliary expert terminals. The AR device is used to capture video images and record location information of the video images; and send the video images and corresponding location information to the auxiliary expert terminal; The auxiliary expert terminal is used to receive the video image and the corresponding position information, calculate the annotation information according to the annotation content of the video image by the auxiliary expert and the position information of the video image, and send the annotation content to the AR device; The AR device is further configured to display the annotation information in real time on a display screen of the AR device through calculation based on the annotation information; Receiving a video image captured by an AR device and location information of the video image, including: Receive a video image of the space where the AR device is located, captured by the AR device through a camera on the AR device; and receiving position information corresponding to the video image, the position information including the real-time three-dimensional world coordinates of the space where the AR device is located, a coordinate conversion matrix between the AR device shooting coordinate system and the AR device world coordinate system, and the shooting coordinate system coordinates of the four corners of the video image on the back clipping plane; Based on the annotation content of the video image by the auxiliary expert, the annotation information corresponding to the video image is calculated in real time, including: The annotation content is a annotation box; Calculate the world coordinate system coordinates of the four corners of the video image based on the shooting coordinate system coordinates of the four corners on the rear clipping surface: PW i =PC i ×Trans Among them, PW i is the world coordinate system coordinate of the i-th corner, PC i is the coordinate of the i-th corner in the shooting coordinate system on the back shearing plane, and Trans is the coordinate transformation matrix between the AR device shooting coordinate system and the AR device world coordinate system; The center position of the annotation box is calculated based on the shooting coordinate system coordinates of the AR device, and the world coordinate system coordinates of the plane where the four corners of the video image are located are: CW=CC×Trans Where CW is the world coordinate system coordinate of the center position of the annotation box in the plane where the four corners of the video image are located, and CC is the shooting coordinate system coordinate of the AR device; The annotation information is automatically generated based on the world coordinate coordinates of the four corners, the center position of the annotation box, the world coordinate coordinates of the plane where the four corners of the video image are located, and the shape and size of the annotation box.

Images