A processing method, a processing device and an electronic device

Abstract

The application discloses a processing method, a processing device and an electronic device. The application sets an image acquisition device in a patrol environment in advance. When it is necessary to patrol an object in the patrol environment, the electronic device is used to obtain selection operation information of a selection operation performed on the object in the patrol environment. The observation object currently in the patrol environment is determined according to the obtained selection operation information. At least one target image acquisition device matched with the observation object is determined according to position information of the observation object, and the at least one target image acquisition device is set in the patrol environment. An image of the observation object collected by the at least one target image acquisition device is obtained and displayed.

Description

Technical Field

[0001] This application belongs to the field of human-computer interaction technology, and in particular relates to a processing method, processing device and electronic device. Background Technology

[0002] In information inspection scenarios involving equipment such as meters, some inspection points (i.e., the specific objects to be inspected, such as meters) are far away, requiring inspectors to climb stairs or crawl into the equipment to see the specific content to be inspected, which greatly increases inspection time. Some inspection points even require inspectors to zoom in with their mobile phone cameras to view them, but the angle at which the inspectors are positioned and the magnification supported by the mobile phone camera can affect the accuracy of the inspection results and reduce inspection efficiency. Summary of the Invention

[0003] Therefore, this application discloses the following technical solution:

[0004] A processing method applied to an electronic device, the method comprising:

[0005] Obtain selection operation information for the selection operations performed on objects in the inspection environment;

[0006] The current observation object in the patrol environment is determined based on the selected operation information;

[0007] Based on the location information of the observed object, at least one target image acquisition device that matches the observed object is identified from the image acquisition device set up in the patrol environment.

[0008] Obtain an image of the observed object acquired by at least one target image acquisition device, and display the obtained image.

[0009] Optionally, determining the currently observed object in the patrol environment based on the selection operation information includes:

[0010] Based on the selection operation information, the selection operation is mapped to a selection operation on the target object model in the inspection environment model, and the object represented by the target object model is taken as the current observation object;

[0011] The inspection environment model is a virtual model obtained by modeling the inspection environment.

[0012] Optionally, determining the currently observed object in the patrol environment based on the selection operation information includes:

[0013] Determine the current pose of the electronic device;

[0014] Determine the depth information of the target point indicated by the selection operation information;

[0015] The target point coordinates are determined based on the current pose, the depth information, and the selection operation information, and the object corresponding to the target point coordinates in the inspection environment is taken as the current observation object.

[0016] Optionally, determining at least one target image acquisition device matching the observed object from the image acquisition device set up within the patrol environment based on the location information of the observed object includes:

[0017] Based on the location information of the observed object, determine the spatial positional relationship between each image acquisition device in the image acquisition device set up in the patrol environment and the observed object;

[0018] Based on the spatial relationship between each image acquisition device and the observed object, at least one target image acquisition device in the image acquisition device set that matches the observed object is determined.

[0019] Optionally, the displayed image includes:

[0020] Based on the spatial relationship between the at least one target image acquisition device and the observed object, the images acquired by the at least one target image acquisition device are displayed.

[0021] Optionally, displaying the images acquired by the at least one target image acquisition device according to the spatial positional relationship between the at least one target image acquisition device and the observed object includes:

[0022] Based on the spatial relationship between the at least one target image acquisition device and the observed object, the images acquired by the at least one target image acquisition device are displayed simultaneously.

[0023] Optionally, displaying the images acquired by the at least one target image acquisition device according to the spatial positional relationship between the at least one target image acquisition device and the observed object includes:

[0024] Display the main image, which is an image that matches the current observation perspective of the observed object from the images acquired by the at least one target image acquisition device;

[0025] In response to a change in the observation angle that exceeds the view angle range corresponding to the main image, the main image is updated to an image that matches the changed observation angle from the images acquired by the at least one target image acquisition device and then displayed.

[0026] Optionally, displaying the images acquired by the at least one target image acquisition device according to the spatial positional relationship between the at least one target image acquisition device and the observed object includes:

[0027] Identify the target location of the observed object in the current observation view, and display the images acquired by the at least one target image acquisition device within a preset range around the target location.

[0028] A processing apparatus for use in an electronic device, the apparatus comprising:

[0029] The acquisition module is used to obtain selection operation information of the selection operations performed on objects in the inspection environment;

[0030] The first determining module is used to determine the observation object currently in the patrol environment based on the selection operation information;

[0031] The second determining module is used to determine, based on the location information of the observed object, at least one target image acquisition device in the image acquisition device set up in the patrol environment that matches the observed object;

[0032] The display module is used to acquire and display images of the observed object acquired by at least one target image acquisition device.

[0033] An electronic device, comprising:

[0034] Memory, used to store at least one set of computer instructions;

[0035] A processor for implementing the processing method described above by invoking and executing the instruction set stored in the memory.

[0036] As can be seen from the above solutions, the processing method, processing device, and electronic device disclosed in this application, in an object inspection scenario, utilize electronic devices to obtain selection operation information of the selection operation performed on the object in the inspection environment, determine the observation object currently in the inspection environment based on the selection operation information, determine at least one target image acquisition device matching the observation object from the image acquisition device set in the inspection environment based on the location information of the observation object, and obtain and display the image of the observation object acquired by at least one target image acquisition device. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating one of the processing methods provided in this application;

[0039] Figure 2 This is an exemplary inspection environment provided in this application;

[0040] Figure 3 This is an exemplary processing flow for object inspection based on an environment model provided in this application;

[0041] Figure 4 This is another flowchart illustrating the processing method provided in this application;

[0042] Figure 5 This is an example of adjusting the viewing angle range of the main image provided in this application;

[0043] Figure 6 This application provides an example of performing a magnification adjustment operation on the main image to achieve image scaling.

[0044] Figure 7 This is a structural diagram of the processing apparatus provided in this application;

[0045] Figure 8 This is a structural diagram of the electronic device provided in this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] This application discloses a processing method, processing device, and electronic device for information inspection scenarios involving objects such as meters. It supports users in observing detailed information about objects in the inspection scenario more intuitively, efficiently, and accurately through human-computer interaction. The processing method of this application can be applied to electronic devices, which may be, but are not limited to, AR (Augmented Reality) or VR (Virtual Reality) XR (Extended Reality) devices (such as AR / VR glasses), or terminal devices such as mobile phones, tablets, laptops, and all-in-one computers.

[0048] See Figure 1 The flowchart shown illustrates the processing method. The processing method disclosed in this application includes at least the following processing steps:

[0049] Step 101: Obtain the selection operation information of the selection operation performed on the object in the inspection environment.

[0050] Objects in the inspection environment can be, but are not limited to, meters or other objects whose information (such as measurement information) needs to be observed.

[0051] This application pre-sets multiple image acquisition devices within the inspection environment and, based on human-machine interaction, uses user-end electronic devices in conjunction with the image acquisition devices set up within the inspection environment to achieve information observation of objects within the inspection environment.

[0052] The above method supports two inspection modes: on-site inspection and remote inspection. Users, such as inspection personnel, can choose either mode according to their actual needs to inspect objects in the inspection environment.

[0053] In one optional implementation, for either on-site inspection mode or remote inspection mode, the inspection environment can be modeled, and the inspection of objects in the inspection environment can be carried out based on the modeled inspection environment model. Preferably, the modeled inspection environment model is a 3D virtual model that matches the real inspection environment.

[0054] Furthermore, for the two inspection modes mentioned above, the inspection environment can be modeled using either pre-modeling or real-time modeling methods. The two modeling methods are as follows:

[0055] (I) Pre-modeling method

[0056] In this approach, image / video information of the overall inspection environment and its various objects from multiple different perspectives is collected in advance, and the collected multi-view image / video information is used as the basic data to construct a 3D virtual model of the inspection environment.

[0057] (II) Real-time modeling method

[0058] In this approach, for the on-site inspection mode, inspectors carrying electronic devices (such as wearing AR glasses or holding a mobile phone) can use the image acquisition device of the electronic device to collect images / video information of the overall inspection environment and the various objects contained within it from multiple different perspectives during the initial period of the on-site inspection. The on-site collected image / video information is then submitted to the electronic device or uploaded to the backend server as the basic data for modeling. The electronic device or the backend server then constructs a 3D virtual model of the inspection environment based on the received collected data.

[0059] For example, typically, users can wear AR glasses to conduct on-site inspections. During the initial inspection period, the AR glasses use their cameras to identify the on-site environment and, based on their configurations for environmental modeling (such as hardware and software configurations), use technologies such as SLAM (Simultaneous Localization and Mapping) to build a matching 3D virtual model for the identified inspection environment.

[0060] For the remote inspection mode, during the initial period of the remote inspection, the inspection robot in the real inspection environment can collect images / video information of the overall inspection environment and the various objects contained therein from multiple different perspectives. The robot then transmits the collected images / video information to the electronic device or the back-end server through remote communication with the electronic device or the back-end server. The electronic device or the back-end server then constructs a 3D virtual model of the inspection environment based on the received data.

[0061] By pre-modeling or real-time modeling, the model should be matched as closely as possible to the real inspection environment and the spatial relationships, object structure / morphology, etc. of the series of objects it contains.

[0062] In this modeling-based implementation, for the on-site inspection mode, users can interact with the real-world scene of the inspection environment to select observation objects, such as meters, as the inspection target (i.e., the target inspection object). Based on this, human-computer interaction is used to observe / inspect the object. This can preferably be achieved using AR devices, such as AR glasses worn by the user. Based on the virtual-real integration function of AR glasses, it supports user interaction with the real-world scene of the inspection environment and human-computer interaction based on this interaction. Users can select / specify observation objects as inspection targets in the real-world inspection environment by performing aerial gestures / hand gestures (e.g., pointing to an object in the real-world scene and clicking it), eye-tracking operations, and / or voice input. AR glasses and other electronic devices or backend servers detect the user's selection operations and parse their corresponding operation information based on technologies such as aerial gesture tracking and recognition, eye tracking, and voice recognition. This includes, but is not limited to, the operation position, gesture type (e.g., pointing to an object in the real-world scene and clicking it), gesture direction, and the focal position when the gesture points in a certain direction for aerial gestures; the operation position, gaze direction, and gaze focus position for eye-tracking operations; or the voice content for voice input.

[0063] In remote inspection mode, optionally, users can make selections on target objects in images transmitted back by the on-site inspection robot. For example, the inspection images captured and transmitted from the robot's current inspection perspective can be highlighted in the virtual model of the inspection environment as foreground, while the environment model serves as the background. This simulates the user's real inspection situation from the robot's perspective and supports the user in making selections on objects in the transmitted images. Operation methods include, but are not limited to, touch operation, air gestures, eye tracking, keyboard and mouse operation, or voice input. Electronic devices or backend servers can use technologies such as touch gesture recognition, air gesture tracking and recognition, eye tracking, keyboard / mouse operation recognition, and voice recognition to analyze the operation information corresponding to the user's selections. This includes, but is not limited to, the gesture type (e.g., click) for touch gestures, the focal position indicated by the touch gesture on the image, the gesture type (e.g., pointing at an object in the air and clicking), the gesture direction, and the focal position when the gesture points in a certain direction; the gaze direction and gaze focus position for eye tracking; the operation type (e.g., single click, double click, etc.) and the focal position indicated on the image for keyboard and mouse operation; or the voice content for voice input.

[0064] However, this is not the only option. In the remote inspection mode, users can also select target objects in the 3D virtual model of the inspection scene in any of the above methods. The electronic device or the back-end server can then identify and analyze the user's selected operation to determine the observation object specified by the user in the inspection environment.

[0065] In other implementations, for on-site inspection mode or remote inspection mode, users can also inspect objects in the inspection environment based on a non-modeling method.

[0066] In the non-modeling scenario, the on-site inspection mode also allows users to interact with the real-world environment to select observation objects, such as meters, as inspection targets. Based on this, human-computer interaction enables observation / inspection of these objects. The user's selection process, the detection and parsing of user actions by the electronic device / backend server, and the resulting selection information are similar to those in the modeling-based on-site inspection mode, and will not be detailed further. The main difference between the non-modeling and modeling-based on-site inspection modes lies in the process by which the electronic device / backend server determines the specified observation object in the inspection environment based on the parsed selection information.

[0067] In the non-modeling scenario, for the remote inspection mode, the user can also select the target object model on the image returned by the on-site inspection robot to specify the observation object to be inspected. In this implementation, the user's selection operation, the detection and parsing of the user's operation by the electronic device / backend server, and the parsed selection operation information are similar to the user's selection operation on the image returned by the on-site inspection robot in the modeling-based remote inspection mode, and will not be described in detail here. The main difference between the non-modeling remote inspection mode and the modeling-based remote inspection mode lies in the different processing procedures of the electronic device / backend server in determining the observation object specified by the selection operation in the inspection environment based on the parsed selection operation information.

[0068] Step 102: Determine the observation objects currently in the inspection environment based on the obtained selection operation information.

[0069] In the model-based on-site inspection mode, the electronic device or backend server can first map the user's selection operation to a selection operation on the target object model in the inspection environment model based on the selection operation information corresponding to the selection operation. For example, based on the user's air gesture operation position, gesture type, gesture direction, and the focal position corresponding to the gesture pointing in a certain direction, through position mapping, object mapping, and other processing, the air selection gesture performed by the user on a real object in the inspection environment can be mapped to a selection gesture on the corresponding object model in the inspection environment model. Alternatively, based on the user's eye movement operation operation position, gaze direction, and gaze focus position, through position mapping, gaze mapping, object mapping, and other processing, the eye movement selection operation performed by the user on a real object in the inspection environment can be mapped to an eye movement selection operation on the corresponding object model in the inspection environment model. Based on this, the object represented by the mapped target object model is taken as the observation object to be observed by the user in the current inspection environment.

[0070] For model-based remote inspection modes, if a user makes a selection of a target object in an image transmitted back by the on-site inspection robot, the electronic device or backend server can map the user's selection of the target object in the image to a selection of the corresponding object model in the inspection environment model. For example, based on the selection operation information, the object indicated by the user's operation in the image can be located. Combined with the perspective of the inspection robot when acquiring the image, the matching of the object in the image with the object model in the inspection environment model achieves the mapping of the user's selection of the target object in the image to a selection of the corresponding object model in the inspection environment model. Based on this, the object represented by the mapped target object model becomes the observation object to be observed by the user in the current inspection environment. If the user performs a selection operation on an object model in the inspection environment model, the target object model specified by the selection operation can be directly determined based on the selection operation information, and the object represented by the target object model becomes the observation object to be observed by the user in the current inspection environment.

[0071] For on-site or remote inspection modes without modeling, the current pose of electronic devices such as AR glasses (on-site inspection mode) or inspection robots (remote inspection mode) can be determined, and the depth information of the target point indicated by the selection operation information can be determined. Then, the coordinates of the target point are determined based on the determined current pose, the depth information and the selection operation information, and the object corresponding to the target point coordinates in the inspection environment is taken as the user's current observation object in the inspection environment.

[0072] The pose of the electronic device / inspection robot describes its position and attitude within the spatial coordinate system of the inspection environment. Specifically, it can be determined by position detection and attitude perception of the electronic device / inspection robot. The determined pose information may include the position coordinates and attitude information (such as orientation, pitch / hook angles, etc.) of the electronic device / inspection robot within the spatial coordinate system of the inspection environment. The position of the electronic device / inspection robot within the spatial coordinate system of the inspection environment can be detected, but is not limited to, using 3D vision positioning technology, while the attitude of the electronic device / inspection robot can be perceived through attitude sensors (such as gyroscopes).

[0073] Optionally, the depth information of the target point indicated by the selection operation information can be obtained by extracting the depth information of the target point image pixels in the image captured by the depth camera, which is mounted on an electronic device or a patrol robot. The depth camera can be, but is not limited to, a camera based on principles such as structured light, TOF (Time of Flight), or binocular vision.

[0074] The determined target point coordinates are specifically the three-dimensional coordinates of the target point indicated by the selection operation in the spatial coordinate system of the inspection environment. Specifically, this can be determined based on the position coordinates and attitude information of the electronic device / inspection robot in the spatial coordinate system of the inspection environment, the depth information of the target point indicated by the selection operation, and the selection operation information corresponding to the selection operation, such as the gesture direction and the focal position corresponding to the gesture pointing in a certain direction (the focal position here usually only refers to the position in a planar coordinate system, such as a plane perpendicular to the orientation of the electronic device / inspection robot). The object corresponding to the target point coordinates in the inspection environment is then used as the observation object to be observed by the user in the current inspection environment.

[0075] Step 103: Based on the location information of the observed object, determine at least one target image acquisition device in the image acquisition device set up in the patrol environment that matches the observed object.

[0076] Specifically, based on the location information of the observed object and the location information of each image acquisition device in the image acquisition device set up in the patrol environment, the spatial positional relationship between each image acquisition device and the observed object can be determined, and based on the spatial positional relationship between each image acquisition device and the observed object, at least one target image acquisition device in the image acquisition device set that matches the observed object can be determined.

[0077] A target image acquisition device that matches the observed object refers to an image acquisition device that can acquire identifiable image information of the observed object from the corresponding viewpoint (such as left viewpoint, right viewpoint, etc.) when the observed object falls within its acquisition viewpoint range.

[0078] Furthermore, optionally, when determining at least one target image acquisition device that matches the observed object, the maximum coverage area (viewing angle range, coverage area / distance, etc.) of the image acquisition device can be determined based on parameters such as the maximum rotation radian and lens magnification of the image acquisition device. Combined with the spatial positional relationship between the image acquisition device and the observed object, as well as the maximum coverage area of ​​the image acquisition device, at least one target image acquisition device that matches the observed object can be determined so that the determined target image acquisition device can acquire clear and distinguishable image information of the observed object from the corresponding angle.

[0079] Specifically, for the model-based on-site / remote inspection mode, the three-dimensional position coordinates of the observed object and each image acquisition device in the inspection environment can be extracted based on the three-dimensional modeling data of the inspection environment. Based on the camera parameters configured for each camera model in the modeling data, such as the maximum rotation radius and lens magnification, the maximum coverage area of ​​the camera is determined. Then, by combining the three-dimensional position coordinates of the observed object and each image acquisition device in the inspection environment, as well as the maximum coverage area of ​​each camera, it is determined which camera's monitoring screen the observed object (the same observed object from various perspectives) can clearly fall on. In this way, at least one target image acquisition device corresponding to the observed object can be matched.

[0080] For on-site / remote inspection modes without modeling, at least one target image acquisition device matching the observed object can be determined based on the real-time detected object location information and the pre-configured location information and camera parameters of each image acquisition device. Specifically, the object location information and the location information of each image acquisition device can be three-dimensional coordinates within the inspection environment's spatial coordinate system.

[0081] Step 104: Obtain an image of the observed object acquired by at least one target image acquisition device, and display the acquired image.

[0082] After identifying at least one target image acquisition device that matches the observed object, the electronic device activates the interfaces of each target image acquisition device. Through communication with these devices within the survey environment, it acquires images of the observed object captured by each target image acquisition device and displays them on the electronic device. Different target image acquisition devices can acquire detailed information about the observed object from different perspectives (such as looking up, looking down, looking left, looking right, looking straight ahead, etc.) and transmit clear images of the detailed information to the electronic device for display. This allows users to conveniently, intuitively, and efficiently observe the detailed information of the observed object from multiple perspectives.

[0083] Preferably, the electronic device and the image acquisition device transmit image data or exchange commands via wireless communication methods such as Wi-Fi or Bluetooth.

[0084] After obtaining images of the observed object acquired by at least one target image acquisition device, the electronic device can display the images acquired by the at least one target image acquisition device according to the spatial positional relationship between the at least one target image acquisition device and the observed object. This process can be further implemented as any of the following embodiments 11)-13):

[0085] 11) Based on the spatial positional relationship between the at least one target image acquisition device and the observed object, simultaneously display the images acquired by the at least one target image acquisition device.

[0086] In this embodiment 11), optionally, the images captured by the at least one target image acquisition device can be arranged in relative positions on the display interface (such as a mobile phone interface or AR interface) according to the spatial positional relationship between the at least one target image acquisition device and the observed object. For example, the left view captured by the camera located to the left of the observed object can be deployed in a preset area on the left side of the interface, the right view captured by the camera located to the right of the observed object can be deployed in a preset area on the right side of the interface, the top view captured by the camera located at the top of the observed object can be deployed in a preset area at the top of the interface, and the bottom view captured by the camera located at the bottom of the observed object can be deployed in a preset area at the bottom of the interface, etc., so that multi-view images of the observed object can be displayed on the interface at the same time, and each image can be arranged according to the spatial positional relationship between its corresponding camera and the observed object, so that users can intuitively observe the observed object from different perspectives.

[0087] In practical applications, images captured by electronic devices (such as AR glasses) or inspection robots from their inspection perspective can be displayed together with images captured by various target image acquisition devices. The images captured from the inspection perspective serve as the front view of the observed object and can be displayed, but are not limited to, in the central area of ​​the interface.

[0088] In other embodiments, optionally, the images captured by each target image acquisition device can be arranged and displayed simultaneously on the display interface, and each displayed image can be associated with corresponding viewpoint information, such as viewpoint labels like "left view," "right view," "top view," or "bottom view" according to the actual viewpoint of each image. By associating corresponding viewpoint information with each listed object, the images captured by at least one target image acquisition device can be displayed simultaneously according to the spatial positional relationship between at least one target image acquisition device and the observed object. Optionally, in this embodiment, images captured by electronic devices or inspection robots from their inspection perspective can also be used as the frontal view of the observed object.Figure 1 It displays and associates the corresponding front view label information with it.

[0089] It is easy to understand that in practical applications, the above two methods can also be combined. That is, according to the spatial positional relationship between the at least one target image acquisition device and the observed object, the images acquired by the at least one target image acquisition device (or, combined with the front view image under the actual inspection perspective) are arranged in relative positions on the display interface, and the corresponding perspective information is displayed for each position.

[0090] 12) Display the main image. In response to a change in the observation angle and the changed observation angle exceeding the view angle range corresponding to the main image, update the main image to an image that matches the changed observation angle from the images acquired by the at least one target image acquisition device and display it.

[0091] The main image displayed is an image that matches the current observation perspective of the observed object from the images acquired by the at least one target image acquisition device.

[0092] In other words, under this embodiment 12), although the electronic device acquires images acquired by each target image acquisition device, it only displays the main image that matches the current observation viewpoint on the display interface. Alternatively, other non-main images can be displayed in conjunction with the main image, but at least the main image should be highlighted or emphasized by configuring display parameters such as layout position, display area occupied by the image, and / or whether it is highlighted. For example, the main image can be displayed in the center of the interface and its display area can be larger than the area occupied by other non-main images.

[0093] Simultaneously, the system tracks the user's viewing angle of the observed object in real time. When the user's viewing angle changes and the changed viewing angle exceeds the viewing angle range corresponding to the currently displayed main image, the electronic device responds to the event by updating the displayed main image in real time and matching the updated main image with the changed viewing angle. This allows the system to follow the user's changing viewing angle and synchronously display an image that matches the user's current viewing angle on the display interface. This makes it easier for the user to observe the object information under their current viewing angle in more detail and intuitively through the interface display.

[0094] 13) Identify the target position of the observed object in the current observation view and display the images acquired by the at least one target image acquisition device within a preset range around the target position.

[0095] In this embodiment 13), the electronic device is specifically an AR device, such as AR glasses.

[0096] For images acquired by at least one target image acquisition device, this embodiment identifies the target position of the observed object in the current observation view. The target position can be the three-dimensional position coordinates of the observed object in the spatial coordinate system of the inspection environment in the current observation view. The AR device, with its virtual-real combination function, displays the images acquired by at least one target image acquisition device within a preset range around the target position. For example, a left view is displayed within a preset range to the left of the target position, and a right view is displayed within a preset range to the right. From the user's perspective, the images of the observed object from various perspectives can be observed within a preset range around the observed object through virtual-real combination. This allows for the observation of detailed information about the observed object during the inspection process by displaying images from different perspectives around the observed object.

[0097] The following is an application example of the method of this application.

[0098] The inspection environment in this example is as follows: Figure 2 As shown, the actual scene of the inspection environment has three available cameras: camera1, camera2, and camera3, deployed at different locations within the inspection environment. Users wear AR glasses to inspect objects in the environment. During the initial inspection period, the AR glasses collect image / video information of the overall inspection environment and its constituent objects from multiple different perspectives to create a 3D model of the real environment. Based on this, an exemplary implementation process for object inspection in the inspection environment using AR glasses is as follows:

[0099] 21) The user points to and clicks on the target point object that needs to be inspected in the real scene by gesture, assuming it is the bottle in the scene diagram.

[0100] 22) The AR glasses, based on environmental modeling data, match camera1 and camera2 that can (clearly) transmit images of the bottle.

[0101] Camera1 and Camera2 are the target image acquisition devices that are matched with the bottle.

[0102] 23) Link camera1 and camera2 to obtain the bottle images captured by camera1 and camera2.

[0103] 24) Display the real-time images of camera1 and camera2 on the AR interface so that users can observe the detailed information of the selected target point object, i.e., the bottle, from different perspectives.

[0104] The AR glasses process involves matching camera1 and camera2, which can clearly transmit images of the bottle based on environmental modeling data, and then displaying the bottle images captured by camera1 and camera2. For details, please refer to [link to relevant documentation]. Figure 3 The process includes: mapping the user's air-tap gestures to a selection operation on a target object model in the environment model to determine the target point object (i.e., the bottle) to be inspected by the user's operation; extracting the 3D position information of the bottle in the 3D environment modeling data (including the 3D positions corresponding to multiple different faces); extracting the 3D position information of each environment camera, i.e., camera1, camera2, and camera3, based on the 3D environment modeling data, and combining parameters such as the maximum rotation arc and lens magnification of each camera to obtain the maximum coverage area (viewing angle, area, distance, etc.) of each camera in the real environment; matching the camera capable of (clearly) transmitting the bottle image, i.e., camera1 and camera2, based on the 3D position information of the bottle, the 3D position information of each environment camera, and the maximum coverage area of ​​each camera; acquiring the images of camera1 and camera2, and displaying the images of camera1 and camera2 according to their relative positional relationship with the bottle.

[0105] Among them, such as Figure 3 As shown, the image from camera 1 is displayed as the left view of the bottle, and the image from camera 2 is displayed as the top view of the bottle. Optionally, the front view image of the bottle captured by the AR glasses' camera is also displayed at the same time.

[0106] Therefore, based on the method of this application, users can select the target object to be inspected in the inspection environment on an electronic device (such as an AR device), and retrieve the camera images near the target object by combining the spatial position relationship between the target object and the environmental camera in the inspection environment, so as to efficiently and intuitively observe the detailed situation of the target object.

[0107] The processing method of this embodiment pre-sets a series of image acquisition devices within the inspection environment and uses the user's electronic device in conjunction with these devices. When the user needs to inspect objects in the environment, they can perform a selection operation on their electronic device to specify the object to be inspected. Combined with the location information of the object, at least one image acquisition device matching the object is retrieved to obtain an image of the object. This provides the user with detailed visual information about the object, eliminating the need for the user to climb stairs or choose a viewing angle during the inspection. This provides support for convenient, intuitive, and efficient observation of detailed information about the object. Based on the human-machine interaction method provided in this application, the accuracy of the inspection results for objects in the inspection environment can be improved, and inspection time can be reduced, thus increasing inspection efficiency.

[0108] In one embodiment, see Figure 4 The provided processing method flowchart indicates that, after displaying the obtained image, the processing method disclosed in this application may further include the following processing:

[0109] Step 105: In response to the display control operation on the displayed image, perform corresponding display control on the image of the observed object.

[0110] The display control operation for the displayed image can specifically be a display control operation for the image displayed based on embodiment 11), or a display control operation for the image displayed based on embodiment 12), or a display control operation for the image displayed based on embodiment 13), without limitation.

[0111] The display control operation for the image shown in any of the above embodiments can have a variety of operation types, including but not limited to the selection operation of the corresponding image, the exit operation, the viewing angle adjustment operation, the zoom operation (i.e., the magnification control operation), etc.

[0112] Accordingly, step 105 can be further implemented as follows:

[0113] 31) In response to the selection operation of the corresponding image among the displayed images, the selected image is displayed as the current main image of the observation object; in response to the exit operation of the current main image, the display of the current main image ends.

[0114] For example, for the multi-view image of the observed object displayed based on any of the embodiments 11)-13), the user can click on it as needed (such as an air gesture for the AR interface, or a touch gesture for the mobile phone interface) to select a target view image that needs to be observed in detail, so as to display the target view image as the current main image, thereby making it easier for the user to observe its detailed information. In other words, the determination of the main image can be based on the user's actual observation perspective of the observed object in the real scene, or it can be based on the user's selection operation on the image display interface.

[0115] Subsequently, when users need to view images from other perspectives in detail, they can exit the current main image by clicking the exit button associated with the main image on the interface, or by performing a swipe gesture towards the edge of the interface on the main image, which can also be an air gesture or a touch gesture. Then, they can select other corresponding perspective images that need to be observed in detail, so that the newly selected image is displayed with the effect of the main image. Alternatively, without exiting the current main image, users can directly click on other corresponding perspective images that need to be observed in detail on the multi-view image thumbnails on the interface, and the main image will be updated directly based on this selection operation.

[0116] 32) In response to the viewpoint adjustment operation of the main image, a viewpoint adjustment command is sent to the target image acquisition device that outputs the main image to instruct the target image acquisition device that outputs the main image to adjust the acquisition viewpoint within the rotatable range, thereby adjusting the viewpoint of the main image accordingly, or updating the main image to the image output by the target image acquisition device that matches the target viewpoint indicated by the viewpoint adjustment operation.

[0117] For the main image displayed on the screen, users can adjust its viewing angle. The device's current display interface for the main image (such as an AR interface) can then be used as the control interface for the corresponding camera in the real environment, enabling the camera to respond to the user's viewing angle adjustment. Optionally, this viewing angle adjustment can be a movement operation where the user moves along the x / y direction (left / right / up / down) of the screen, a voice input operation, or a configuration operation based on information configuration, etc.

[0118] For example, a user can perform a movement operation along the x and y directions of the screen on the current main image interface (i.e., a movement operation to the left / right or up / down directions, which can be a movement operation based on air gestures or touch gestures), causing the electronic device to send a rotation command in the corresponding direction to the camera corresponding to the main image in the real environment. By linking the rotation of the camera, the image content of the main image can be controlled to change in real time along the up / down / left / right viewing direction within the maximum field of view of the corresponding camera.

[0119] In practical applications, optionally, the interactive functions of the main image can be activated in advance by performing a preset activation operation on the main image. Based on this, the viewpoint of the main image can be adjusted in the desired direction by performing the above-mentioned movement operation with the activation point as the origin.

[0120] Once the number of steps the user moves on the PTZ interface exceeds the number of steps corresponding to the maximum field of view of the camera corresponding to the current main view, the main image will be automatically switched to the image provided by the camera with the field of view corresponding to the current number of steps. In other words, the main image is updated at this time, and is updated to the image output by the target image acquisition device that matches the target field of view indicated by the current number of steps (representing the current field of view adjustment operation).

[0121] An example of adjusting the viewpoint range of the main image is as follows: Figure 5 As shown.

[0122] Optionally, if there is no target image acquisition device matching the target's viewpoint in the inspection environment, the main view is updated to the image output by the image acquisition device of an electronic device (such as AR glasses) (i.e., the front view).

[0123] 33) In response to the magnification adjustment operation of the main image, a magnification adjustment command is sent to the target image acquisition device that outputs the main image to instruct the target image acquisition device that outputs the main image to adjust its lens magnification and focus on the observed object.

[0124] This implementation allows users to perform magnification adjustments on the main image, achieving the effect of scaling the main image. Optionally, the magnification adjustment operation can be a movement operation where the user moves along the depth direction (z-axis direction) of the display screen, a voice input operation, or a configuration operation based on information configuration, etc.

[0125] When a user performs a magnification adjustment operation on the main image, such as moving along the z-axis, the AR device synchronously sends corresponding control commands to the camera corresponding to the main image in the real environment. These control commands include at least a lens zoom command, instructing the target image acquisition device outputting the current main image to adjust its lens magnification, thereby achieving the purpose of scaling the output main image. Optionally, in response to a movement operation moving from light to dark along the z-axis, the main image is magnified; in response to a movement operation moving from dark to light along the z-axis, the main image is scaled down.

[0126] Preferably, when performing a magnification adjustment operation on the main image, the control command sent by the electronic device to the corresponding camera also includes a lens focusing command, so that the target image acquisition device outputting the main image can automatically focus on the current observation object by zooming while adjusting its lens magnification, thereby ensuring the image acquisition effect of the observation object.

[0127] An example of performing a magnification adjustment on the main image to achieve image scaling is as follows: Figure 6 As shown.

[0128] Optionally, for cases where the magnification adjustment operation involves moving along the depth direction of the screen, a tolerance distance for the movement operation in the depth direction of the display screen can be preset to avoid misoperation caused by the user's hand unconsciously moving in the depth direction when performing gimbal / up / down / left / right movements.

[0129] This embodiment provides users with diverse operation functions for images from different perspectives on the display interface, which can further meet users' diverse viewing needs for information on observed objects in real inspection scenarios, making it more convenient, intuitive and efficient for users to observe information on observed objects in real inspection scenarios.

[0130] Corresponding to the above processing method, this application also discloses a processing apparatus, the structure of which is as follows: Figure 7 As shown, it includes:

[0131] The acquisition module 701 is used to obtain selection operation information of the selection operation performed on objects in the inspection environment;

[0132] The first determining module 702 is used to determine the observation object currently in the patrol environment based on the selection operation information;

[0133] The second determining module 703 is used to determine, based on the location information of the observed object, at least one target image acquisition device in the image acquisition device set up in the patrol environment that matches the observed object;

[0134] The display module 704 is used to obtain an image of the observed object acquired by at least one target image acquisition device and to display the obtained image.

[0135] In one embodiment, the first determining module 702 is specifically used to: map the selection operation to a selection operation on a target object model in the inspection environment model according to the selection operation information, and take the object represented by the target object model as the current observation object;

[0136] The inspection environment model is a virtual model obtained by modeling the inspection environment.

[0137] In one embodiment, the first determining module 702 is specifically used for:

[0138] Determine the current pose of the electronic device;

[0139] Determine the depth information of the target point indicated by the selection operation information;

[0140] The target point coordinates are determined based on the current pose, the depth information, and the selection operation information, and the object corresponding to the target point coordinates in the inspection environment is taken as the current observation object.

[0141] In one embodiment, the second determining module 703 is specifically used for:

[0142] Based on the location information of the observed object, determine the spatial positional relationship between each image acquisition device in the image acquisition device set up in the patrol environment and the observed object;

[0143] Based on the spatial relationship between each image acquisition device and the observed object, at least one target image acquisition device in the image acquisition device set that matches the observed object is determined.

[0144] In one embodiment, the display module 704 is specifically used for:

[0145] Based on the spatial relationship between the at least one target image acquisition device and the observed object, the images acquired by the at least one target image acquisition device are displayed.

[0146] In one embodiment, when the display module 704 displays the images acquired by the at least one target image acquisition device according to the spatial positional relationship between the at least one target image acquisition device and the observed object, it is specifically used for:

[0147] Based on the spatial relationship between the at least one target image acquisition device and the observed object, the images acquired by the at least one target image acquisition device are displayed simultaneously.

[0148] In one embodiment, when the display module 704 displays the images acquired by the at least one target image acquisition device according to the spatial positional relationship between the at least one target image acquisition device and the observed object, it is specifically used for:

[0149] Display the main image, which is an image that matches the current observation perspective of the observed object from the images acquired by the at least one target image acquisition device;

[0150] In response to a change in the observation angle that exceeds the view angle range corresponding to the main image, the main image is updated to an image that matches the changed observation angle from the images acquired by the at least one target image acquisition device and then displayed.

[0151] In one embodiment, when the display module 704 displays the images acquired by the at least one target image acquisition device according to the spatial positional relationship between the at least one target image acquisition device and the observed object, it is specifically used for:

[0152] Identify the target location of the observed object in the current observation view, and display the images acquired by the at least one target image acquisition device within a preset range around the target location.

[0153] The processing apparatus disclosed in this application corresponds to the processing methods disclosed in the above method embodiments, so the description is relatively simple. For related similarities, please refer to the descriptions of the above method embodiments, which will not be detailed here.

[0154] This application also discloses an electronic device, which may be, but is not limited to, a device in a variety of general or special computing device environments or configurations, such as: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor devices, etc.

[0155] The composition and structure of electronic devices, such as Figure 8 As shown, it includes at least:

[0156] Memory 10 is used to store the computer instruction set;

[0157] Computer instruction sets can be implemented in the form of computer programs.

[0158] The processor 20 is configured to implement the data processing method disclosed in any of the above method embodiments by executing a computer instruction set.

[0159] The processor 20 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices.

[0160] Electronic devices have a display device and / or have a display interface and can connect to an external display device.

[0161] Optionally, the electronic device may also include a camera assembly, and / or be connected to an external camera assembly.

[0162] In addition to these components, electronic devices may also include communication interfaces, communication buses, and other parts. Memory, processor, and communication interface communicate with each other through the communication bus.

[0163] Communication interfaces are used for communication between electronic devices and other devices. Communication buses can be Peripheral Component Interconnect (PCI) buses or Extended Industry Standard Architecture (EISA) buses, and can be categorized into address buses, data buses, control buses, etc.

[0164] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0165] For ease of description, the above systems or devices are described separately as various modules or units based on their functions. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware components.

[0166] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0167] Finally, it should be noted that in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0168] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A processing method applied to an electronic device, the method comprising: Obtain selection operation information for the selection operations performed on objects in the inspection environment; The current observation object in the patrol environment is determined based on the selected operation information; Based on the location information of the observed object, at least one target image acquisition device that matches the observed object is identified from the image acquisition device set up in the patrol environment. Obtain an image of the observed object acquired by at least one target image acquisition device, and display the obtained image; The image obtained by display includes at least one of the following: Display a main image, which is an image that matches the current observation angle of the observed object among the images acquired by the at least one target image acquisition device; in response to a change in the observation angle and the changed observation angle exceeding the view angle range corresponding to the main image, update the main image to an image that matches the changed observation angle among the images acquired by the at least one target image acquisition device and display it. Identify the target location of the observed object in the current observation view, and display the images acquired by the at least one target image acquisition device within a preset range around the target location.

2. The method according to claim 1, wherein determining the currently observed object in the patrol environment based on the selection operation information includes: Based on the selection operation information, the selection operation is mapped to a selection operation on the target object model in the inspection environment model, and the object represented by the target object model is taken as the current observation object; The inspection environment model is a virtual model obtained by modeling the inspection environment.

3. The method according to claim 1, wherein determining the currently observed object in the patrol environment based on the selection operation information includes: Determine the current pose of the electronic device; Determine the depth information of the target point indicated by the selection operation information; The target point coordinates are determined based on the current pose, the depth information, and the selection operation information, and the object corresponding to the target point coordinates in the inspection environment is taken as the current observation object.

4. The method according to claim 1, wherein determining at least one target image acquisition device matching the observed object from the image acquisition device set up within the patrol environment based on the location information of the observed object comprises: Based on the location information of the observed object, determine the spatial positional relationship between each image acquisition device in the image acquisition device set up in the patrol environment and the observed object; Based on the spatial relationship between each image acquisition device and the observed object, at least one target image acquisition device in the image acquisition device set that matches the observed object is determined.

5. The method according to claim 1, wherein displaying the obtained image further comprises: Based on the spatial relationship between the at least one target image acquisition device and the observed object, the images acquired by the at least one target image acquisition device are displayed simultaneously.

6. A processing apparatus applied to an electronic device, the apparatus comprising: The acquisition module is used to obtain selection operation information of the selection operations performed on objects in the inspection environment; The first determining module is used to determine the observation object currently in the patrol environment based on the selection operation information; The second determining module is used to determine, based on the location information of the observed object, at least one target image acquisition device in the image acquisition device set up in the patrol environment that matches the observed object; A display module is configured to acquire and display images of the observed object acquired by at least one target image acquisition device; the display of the acquired images includes at least one of the following: displaying a main image, wherein the main image is an image among the images acquired by the at least one target image acquisition device that matches the current observation perspective of the observed object; in response to a change in the observation perspective and the changed observation perspective exceeding the perspective range corresponding to the main image, updating the main image to an image among the images acquired by the at least one target image acquisition device that matches the changed observation perspective and displaying it; Identify the target location of the observed object in the current observation view, and display the images acquired by the at least one target image acquisition device within a preset range around the target location.

7. An electronic device, comprising: Memory, used to store at least one set of computer instructions; A processor for implementing the processing method as described in any one of claims 1-5 by invoking and executing the instruction set stored in the memory.

Images