Data Processing Method, Control Terminal, AR Terminal, AR System and Storage Medium

Abstract

An embodiment of the present application discloses a data processing method, a control terminal, an AR terminal, an AR system, and a storage medium. The generation method includes: after establishing a connection with the AR terminal in the AR system, sending a shooting instruction to the AR terminal; receiving the original image data sent by the AR terminal; based on the target format and the shooting parameters corresponding to the AR terminal, calling the first shooting device to collect image data; replacing the image data with the original image data, and calling the first ISP module to perform data processing to obtain processed data. After the AR terminal establishes a connection with the control terminal in the AR system, it receives the shooting instruction sent by the control terminal; determines the shooting parameters, and based on the target format and the shooting parameters, calls the second shooting device to collect the original image data; sends the original image data to the control terminal to complete the data processing through the first ISP module of the control terminal.

Description

[0001] This application is a divisional application of Chinese patent application No. 202010833074.9, filed on August 18, 2020, and entitled “Data processing method, control terminal, AR terminal, AR system and storage medium”. Technical Field

[0002] The present invention relates to the field of virtual reality technology, and in particular to a data processing method, a control terminal, an AR terminal, an AR system, and a storage medium. Background Art

[0003] Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and video, and applies it to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0004] A split AR glasses system can be composed of a terminal device and AR glasses. Among them, the data processing capability and performance of the image signal processor (ISP) in AR glasses are generally poor. When performing some complex or time-consuming image processing, there is a situation where power consumption increases instantaneously, which in turn causes the power consumption of the terminal device used for power supply to increase instantaneously, leading to unpredictable situations such as terminal device crashes.

[0005] It can be seen that the existing split-type AR glasses system is still unable to perform complex image data processing tasks well. Therefore, there is an urgent need for a data processing method to improve the data processing capabilities of the split-type AR glasses system and enhance the intelligence of the split-type AR glasses system. Summary of the Invention

[0006] The embodiments of the present application provide a data processing method, a control terminal, an AR terminal, an AR system, and a storage medium, which effectively improve the data processing capability of the AR system and enhance the intelligence of the AR system.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a control terminal in an AR system. The method includes:

[0009] After establishing a connection with an AR terminal in the AR system, sending a shooting instruction to the AR terminal; wherein the shooting instruction carries a target format;

[0010] Receive the original image data sent by the AR terminal; wherein, the original image conforms to the target format;

[0011] Based on the target format and the shooting parameters corresponding to the AR terminal, call the first shooting device to collect image data;

[0012] Replace the image data with the original image data, and call the first ISP module to perform data processing to obtain processed data.

[0013] In a second aspect, an embodiment of the present application provides a data processing method, which is applied to an AR terminal in an AR system, and the method includes:

[0014] After establishing a connection with the control terminal in the AR system, receive the shooting instruction sent by the control terminal; wherein, the shooting instruction carries the target format;

[0015] Determine the shooting parameters, and based on the target format and the shooting parameters, call the second shooting device to collect the original image data; wherein, the original image conforms to the target format;

[0016] Send the original image data to the control terminal to complete data processing through the first ISP module of the control terminal.

[0017] In a third aspect, an embodiment of the present application provides a control terminal, which includes: a first sending unit, a first receiving unit, a first collecting unit, a replacing unit, and a processing unit.

[0018] The first sending unit is configured to send a shooting instruction to the AR terminal after establishing a connection with the AR terminal in the AR system; wherein, the shooting instruction carries the target format;

[0019] The first receiving unit is configured to receive the original image data sent by the AR terminal; wherein, the original image conforms to the target format;

[0020] The first collecting unit is configured to call the first shooting device to collect image data based on the target format and the shooting parameters corresponding to the AR terminal;

[0021] The replacing unit is configured to replace the image data with the original image data;

[0022] The processing unit is configured to call the first ISP module to perform data processing to obtain processed data.

[0023] Fourth aspect, an embodiment of the present application provides a control terminal, which includes a first processor and a first memory storing instructions executable by the first processor. When the instructions are executed by the first processor, the data processing method described above is implemented.

[0024] Fifth aspect, an embodiment of the present application provides an AR terminal, which includes: a second receiving unit, a determining unit, a second collecting unit, and a second sending unit.

[0025] The second receiving unit is configured to receive a shooting instruction sent by the control terminal after establishing a connection with the control terminal in the AR system; wherein, the shooting instruction carries a target format.

[0026] The determining unit is configured to determine shooting parameters.

[0027] The second collecting unit is configured to call a second shooting device to collect original image data based on the target format and the shooting parameters; wherein, the original image conforms to the target format.

[0028] The second sending unit is configured to send the original image data to the control terminal to complete data processing through the first ISP module of the control terminal.

[0029] Sixth aspect, an embodiment of the present application provides an AR terminal, which includes a second processor and a second memory storing instructions executable by the second processor. When the instructions are executed by the second processor, the data processing method described above is implemented.

[0030] Seventh aspect, an embodiment of the present application provides an AR system, which includes: a control terminal and an AR terminal. Among them, the control terminal is configured with a first shooting device and a first ISP module, and the AR terminal is configured with a second shooting device and a second ISP module. The control terminal calls the first shooting device and the first ISP module to implement the method described in the first aspect, and the AR terminal calls the second shooting device and the second ISP module to implement the method described in the second aspect.

[0031] Eighth aspect, an embodiment of the present application provides an AR system, which includes a third processor and a third memory storing instructions executable by the third processor. When the instructions are executed by the third processor, the data processing method described above is implemented.

[0032] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program is stored and applied to an AR system. When the program is executed by a first processor, the method described in the first aspect is implemented. When the program is executed by a second processor, the method described in the second aspect is implemented.

[0033] An embodiment of the present application provides a data processing method, a control terminal, an AR terminal, an AR system, and a storage medium. After the control terminal establishes a connection with the AR terminal in the AR system, it sends a shooting instruction to the AR terminal. The shooting instruction carries a target format. It receives the original image data sent by the AR terminal, where the original image conforms to the target format. Based on the target format and the shooting parameters corresponding to the AR terminal, it calls the first shooting device to collect image data. It replaces the image data with the original image data and calls the first ISP module to perform data processing to obtain the processed data. After the AR terminal in the AR system establishes a connection with the control terminal, it receives the shooting instruction sent by the control terminal. The shooting instruction carries a target format. It determines the shooting parameters and, based on the target format and the shooting parameters, calls the second shooting device to collect the original image data, where the original image conforms to the target format. It sends the original image data to the control terminal to complete data processing through the first ISP module of the control terminal. That is to say, in the embodiment of the present application, the AR terminal in the AR system can transmit the collected original image data to the control terminal, then use the original image data to replace the image data collected by the control terminal, and then input the original image data into the first ISP module in the control terminal for data processing. Since the performance and capabilities of the first ISP module are superior to those of the second ISP module, by using the first ISP module to process the data of the AR terminal, the data processing ability of the AR system can be effectively improved, and the intelligence of the AR system can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 Schematic diagram of the implementation process of the data processing method Figure 1 ;

[0035] Figure 2 Schematic diagram of the AR system Figure 1 ;

[0036] Figure 3 Schematic diagram of the control terminal;

[0037] Figure 4 Schematic diagram of the implementation process of the data processing method Figure 2 ;

[0038] Figure 5 Schematic diagram of the implementation process of the data processing method Figure 3 ;

[0039] Figure 6 It is a flow chart for ISP processing;

[0040] Figure 7 It is a schematic diagram of the implementation process of the data processing method Figure 4 ;

[0041] Figure 8 It is a schematic diagram of the AR system Figure 2 ;

[0042] Figure 9 It is a schematic diagram of the structure of the AR glasses;

[0043] Figure 10 It is a schematic diagram of the AR terminal;

[0044] Figure 11 It is a schematic diagram of the implementation process of the data processing method Figure 5 ;

[0045] Figure 12 It is a schematic diagram of the implementation process of the data processing method Figure 6 ;

[0046] Figure 13 It is a schematic diagram of the implementation process of the data processing method Figure 7 ;

[0047] Figure 14 It is a schematic diagram of the implementation process of the data processing method Figure 8 ;

[0048] Figure 15 It is a schematic diagram of the implementation process of the data processing method Figure 9 ;

[0049] Figure 16 It is a schematic diagram of the composition structure of the control terminal Figure 1 ;

[0050] Figure 17 It is a schematic diagram of the composition structure of the control terminal Figure 2 ;

[0051] Figure 18 It is a schematic diagram of the composition structure of the AR terminal Figure 1 ;

[0052] Figure 19 It is a schematic diagram of the composition structure of the AR terminal Figure 2 ;

[0053] Figure 20 It is a schematic diagram of the composition structure of the AR system Figure 1 ;

[0054] Figure 21 It is a schematic diagram of the composition structure of the AR system Figure 2 . Detailed implementation manners

[0055] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. It can be understood that the specific embodiments described herein are only used to explain the related application, rather than limiting the application. In addition, it should be noted that, for the sake of convenience of description, only the parts related to the relevant application are shown in the drawings.

[0056] The three major technical points of AR: three-dimensional registration (tracking and registration technology), virtual reality fusion display, and human-computer interaction. The process is as follows: First, the real scene is collected through a camera and sensors, and the data is transmitted to a processor for analysis and reconstruction. Then, through accessories such as cameras, gyroscopes, and sensors on the AR headset or intelligent mobile AR system, the spatial position change data of the user in the real environment is updated in real time, so as to obtain the relative position between the virtual scene and the real scene, realize the alignment of the coordinate system, and perform the fusion calculation of the virtual scene and the real scene. Finally, the synthesized image is presented to the user. The user can collect control signals through interaction accessories on the AR headset or intelligent mobile AR system, such as microphones, eye trackers, infrared sensors, cameras, sensors, etc., of the AR system, and perform corresponding human-computer interaction and information update to realize the interactive operation of augmented reality. Among them, three-dimensional registration is the core of AR technology, that is, taking two-dimensional or three-dimensional objects in the real scene as markers, and matching the virtual information with the real scene information, that is, the position, size, movement path, etc. of the virtual object must be perfectly matched with the real environment, reaching the state of combining the virtual and the real.

[0057] When users usually receive information, the images they see are all 2D or 2.5D. AR devices, such as AR smart glasses, can present holographic stereoscopic images, allowing users to receive more vivid information, so that users can be completely immersed in the AR audio-visual scene.

[0058] For the split-type AR glasses system, generally, the Universal Serial Bus (USB) 3.0 scheme is adopted to supply power to the AR glasses by the terminal device. The AR glasses can use the configured shooting device to collect images, and use the configured image processor to perform ISP processing on the original image data obtained by the collection. However, in some image processing processes with complex functions and long time consumption, the image processor configured in the AR glasses performs poorly in terms of power consumption, function, and performance, and often there will be a situation where the power consumption increases instantaneously, resulting in an instantaneous increase in the power consumption of the terminal device that supplies power, and then causing unpredictable situations such as the terminal device crashing.

[0059] Compared with the image processor configured in the AR glasses, the image processor configured in the terminal device in the AR system has much better performance and functions. However, the image processor of the terminal device is connected to the shooting device through the Mobile Industry Processor Interface (MIPI), that is, the image processor of the terminal device must be used in combination with the shooting device of the terminal device, that is, the image processor of the terminal device cannot be used as an independent data processing device for the AR glasses.

[0060] That is to say, although the image processor of the terminal device in the AR system has excellent performance, it cannot be used for image data processing of the AR glasses, resulting in the existing split AR glasses system being unable to perform complex image data processing operations well. Therefore, there is an urgent need for a data processing method to improve the data processing ability of the split AR glasses system and enhance the intelligence of the split AR glasses system.

[0061] In order to solve the problems existing in the existing decoding process, in the embodiments of the present application, that is, in the embodiments of the present application, the AR terminal in the AR system can transmit the acquired original image data to the control terminal, and then use the original image data to replace the image data collected by the control terminal. Then, the original image data can be input into the first ISP module in the control terminal for data processing. Since the performance and capabilities of the first ISP module are superior to those of the second ISP module, by using the first ISP module to process the data of the AR terminal, the data processing ability of the AR system can be effectively improved, and the intelligence of the AR system can be enhanced.

[0062] Next, the technical solutions in the embodiments of the present application will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present application.

[0063] An embodiment of the present application provides a data processing method, which can be applied to the control terminal in the AR system. Figure 1 Schematic diagram of the implementation process of the data processing method Figure 1 , as Figure 1 shown, in the embodiments of the present application, the method for the control terminal in the AR system to process data may include the following steps:

[0064] Step 101: After establishing a connection with the AR terminal in the AR system, send a shooting instruction to the AR terminal; wherein, the shooting instruction carries a target format.

[0065] In the embodiments of the present application, after the control terminal in the AR system establishes a connection with the AR terminal, it can first send a shooting instruction to the AR terminal, so that the AR terminal can perform shooting processing based on the shooting instruction.

[0066] It should be noted that in the embodiments of the present application, the system may include at least one control terminal and at least one AR terminal. Specifically, the system may be a split-type AR system, and the control terminal in the AR system may supply power to the AR terminal through the connection with the AR terminal.

[0067] It can be understood that in the embodiments of the present application, a physical connection may be established between the control terminal and the AR terminal through a connection line and an interface, or a wireless connection may be established through a wireless transmission module, and the present application does not make specific limitations.

[0068] Exemplarily, in the embodiments of the present application, Figure 2 is a schematic diagram of an AR system Figure 1 , such as Figure 2 shown, the AR system 10 is a split-type AR glasses system, the control terminal 20 in the split-type AR glasses system is a mobile phone, and the AR terminal 30 is AR glasses. Among them, the mobile phone supplies power to the AR glasses through a wired connection with the AR glasses.

[0069] Further, in the embodiments of the present application, the control terminal in the AR system may be any device with communication and storage functions, such as: tablet computer, mobile phone, e-reader, remote control, personal computer (Personal Computer, PC), notebook computer, vehicle-mounted device, Internet TV, wearable device, personal digital assistant (Personal Digital Assistant, PDA), portable media player (Portable Media Player, PMP), navigation device and other devices.

[0070] It can be understood that in the embodiments of the present application, the control terminal in the AR system may be configured with a power module, and among them, the power module of the control terminal may supply power to the AR terminal through a connection line or through a wireless transmission module.

[0071] It should be noted that in the embodiments of the present application, the control terminal may send a connection request to the AR terminal, receive the connection response returned by the AR terminal, and then establish a connection with the AR terminal through the connection response; the control terminal may also receive the connection request sent by the AR terminal, send a connection response to the AR terminal, and then establish a connection with the AR terminal through the connection response.

[0072] Further, in the embodiments of the present application, after the control terminal establishes a connection with the AR terminal, it may send a shooting instruction to the AR terminal, where the shooting instruction is used to instruct the AR terminal to perform shooting processing. Specifically, the shooting instruction may carry a target format.

[0073] It should be noted that, in the embodiments of the present application, the target format is used to restrict the format of the image data collected by the AR terminal through the second imaging device. Specifically, common data output formats of an imaging device (such as a Camera) include: RAW DATA format, YUV format, and RGB format. Among them, the RAW DATA format is the original record of the voltage levels when a Charge-coupled Device (CCD) or a Complementary Metal Oxide Semiconductor (CMOS) converts an optical signal into an electrical signal, and it is obtained by simply digitizing the image data without any processing, that is, the electrical signal directly obtained by the imaging element.

[0074] Generally, for a shooting module that supports the YUV format or the RGB format, an ISP is generally integrated on the shooting module. The original image data after A / D conversion is processed by the ISP to generate the YUV standard format. For a shooting module that supports the RAW DATA format, due to the requirements of the photosensitive area, an ISP is not integrated in the shooting module to maximize the area of the photosensitive area and improve the photo quality.

[0075] It can be understood that, in the embodiments of the present application, the format of the input data during the ISP processing is the RAW DATA format. Therefore, the target format in the present application is used to restrict the format of the image data sent by the AR terminal to the control terminal to the RAW DATA format, even if the AR terminal can directly send the original image data collected through the second imaging device to the control terminal.

[0076] Furthermore, in the embodiments of the present application, the control terminal may also be configured with a first imaging device and a first ISP module. Among them, the first imaging device is used to collect image data, and the first ISP module is used to process the image data.

[0077] Exemplarily, in the present application, Figure 3 is a schematic diagram of the control terminal, as Figure 3 shown, the control terminal 20 may include a first imaging device 21, a first ISP module 22, and a power module 23.

[0078] Step 102, receive the original image data sent by the AR terminal; wherein, the original image conforms to the target format.

[0079] In the embodiments of the present application, after sending a shooting instruction to the AR terminal, the control terminal may further receive the original image data that conforms to the target format and is sent by the AR terminal. Among them, the original image data is collected by the AR terminal through the second imaging device.

[0080] It should be noted that in the embodiments of the present application, the original image data is the image data in RAW DATA format directly output by the second shooting device after the AR terminal calls the second shooting device for shooting and processing.

[0081] In the embodiments of the present application, further, Figure 4 Schematic diagram of the implementation process of the data processing method Figure 2 , such as Figure 4 shown, before receiving the original image data sent by the AR terminal, that is, before step 102, the method for the control terminal to perform data processing may further include the following steps:

[0082] Step 105: Receive the processing request sent by the AR terminal.

[0083] Step 106: Respond to the processing request and send a processing response to the AR terminal; wherein, the processing response is used to obtain the original image data.

[0084] In the embodiments of the present application, after the control terminal sends the shooting instruction to the AR terminal, it may first receive the processing request sent by the AR terminal, and then respond to the processing request and send a processing response to the AR terminal, so as to instruct the AR terminal to send the original image data.

[0085] It should be noted that in the embodiments of the present application, the processing request is sent by the AR terminal and is used to request the control terminal to process the data by means of the first ISP module configured by the control terminal.

[0086] It can be understood that in the embodiments of the present application, after the control terminal receives the processing request, it may first determine the current status and tasks. Specifically, if the control terminal is using the first shooting device or the first ISP module, that is, the first terminal is performing shooting processing or image data processing, then it can be considered that the first ISP module cannot be called to process the original image data sent by the AR terminal. If the control terminal is not using the first shooting device and the first ISP module, that is, the first terminal is not performing shooting processing and image data processing, then it can be considered that the first ISP module is allowed to process the original image data sent by the AR terminal.

[0087] Further, in the embodiments of the present application, after the control terminal determines that the first ISP module is allowed to be called based on the current status and tasks, it can respond to the processing request and send a processing response for obtaining the original image data to the AR terminal.

[0088] Correspondingly, in the present application, after the AR terminal receives the processing response sent by the control terminal, it can send the original image data to the control terminal.

[0089] In an embodiment of the present application, further, Figure 5 is a schematic implementation process of the data processing method Figure 3 , as Figure 5 shown, after establishing a connection with the AR terminal in the AR system and sending a shooting instruction to the AR terminal, that is, after step 101, the method for the control terminal to perform data processing may further include the following steps:

[0090] Step 107, receive the processed data corresponding to the original image data sent by the AR terminal; wherein, the processed data is obtained by the AR terminal calling the second ISP module to perform data processing.

[0091] In an embodiment of the present application, after the control terminal establishes a connection with the AR terminal in the AR system and sends a shooting instruction to the AR terminal, it can also directly receive the processed data corresponding to the original image data sent by the AR terminal.

[0092] It should be noted that, in an embodiment of the present application, the processed data may be obtained by the AR terminal calling the configured second ISP module to perform data processing on the original image data. Among them, the processed data may be image data in YUV format or RGB format.

[0093] Further, in an embodiment of the present application, since the AR terminal is also configured with a second ISP module capable of performing image data processing, therefore, after the AR terminal calls the second shooting device to collect the original image data, it can choose to directly call the second ISP module to perform data processing on the original image data, or choose to send the original image data to the control terminal to perform data processing on the original image data by means of the first ISP module configured with higher performance of the control terminal.

[0094] It can be understood that, in an embodiment of the present application, for image processing services with simple functions or short time consumption, the second ISP module configured by the AR terminal can complete them well. Therefore, the AR terminal can choose not to perform data processing by means of the first ISP module configured by the control terminal, but directly call the second ISP module to process the collected original image information.

[0095] Correspondingly, in the present application, for image processing services with complex functions or long time consumption, the performance and capabilities of the second ISP module configured by the AR terminal cannot complete them well. Therefore, the AR terminal can choose to send the original image data to the control terminal, so as to complete the processing of the original image data by means of the first ISP module of the control terminal.

[0096] Step 103, call the first shooting device to collect image data based on the target format and the shooting parameters corresponding to the AR terminal.

[0097] In an embodiment of the present application, after the control terminal sends a shooting instruction to the AR terminal, it can also call the first shooting device to collect image data based on the target format and the shooting parameters corresponding to the AR terminal.

[0098] Further, in an embodiment of the present application, the control terminal can activate the configured first shooting device, and then, according to the target format and the shooting parameters corresponding to the AR terminal for shooting the original image data, perform image acquisition processing through the first shooting device to obtain the corresponding image data.

[0099] It should be noted that, in an embodiment of the present application, since the control terminal controls the first shooting device to collect the image data based on the target format, the image data also meets the requirements of the target format, that is, the image data is the data in RAW DATA format directly output by the first shooting device.

[0100] It can be understood that, in an embodiment of the present application, since the control terminal controls the first shooting device to collect the image data based on the shooting parameters corresponding to the AR terminal, the image data collected by the control terminal through the first shooting device has the same attributes as the original image data received from the AR terminal.

[0101] Further, in an embodiment of the present application, the shooting parameters can be the relevant parameters used by the second shooting device when the AR terminal collects the original image data. Exemplarily, the shooting parameters can include resolution, clarity, brightness, contrast, saturation, focal length, field of view, aperture, gain, exposure time, white balance, etc.

[0102] It should be noted that, in an embodiment of the present application, the shooting parameters corresponding to the AR terminal can be preset by the control terminal and sent to the AR terminal. Specifically, before calling the first shooting device to collect image data based on the target format and the shooting parameters corresponding to the AR terminal, the control terminal can first set the shooting parameters. Correspondingly, the shooting instruction sent by the control terminal to the AR terminal can carry the shooting parameters.

[0103] It should be noted that, in an embodiment of the present application, the shooting parameters corresponding to the AR terminal can also be set by the AR terminal and sent to the control terminal. Specifically, before calling the first shooting device to collect image data based on the target format and the shooting parameters corresponding to the AR terminal, the control terminal can receive the shooting parameters sent by the AR terminal.

[0104] Further, in the embodiments of the present application, after the control terminal collects image data by invoking the first imaging device based on the target format and the shooting parameters corresponding to the AR terminal, the image data can be stored in the first storage space. Generally, after the first imaging device in the control terminal outputs the image data, the image data can be first stored in the first storage space, and then in the subsequent ISP processing process, it can be copied to the first ISP module through the memory.

[0105] It should be noted that, in the embodiments of the present application, the methods proposed in the above steps 102 and 103 can be executed by the control terminal simultaneously or sequentially, and the present application does not make a specific limitation.

[0106] Step 104: Replace the image data with the original image data, and invoke the first ISP module to perform data processing to obtain the processed data.

[0107] In the embodiments of the present application, after the control terminal receives the original image data sent by the AR terminal and simultaneously collects image data by invoking the first imaging device based on the target format and the shooting parameters corresponding to the AR terminal, the image data can be replaced with the original image data, and then the first ISP module can be invoked to perform data processing, and finally the processed data can be obtained.

[0108] Further, in the embodiments of the present application, when the control terminal invokes the first ISP module to process the original image data, it is necessary to first replace the image data collected by the first imaging device with the original image data.

[0109] Figure 6 is the ISP processing flow chart. As Figure 6 shown, when performing ISP processing on the image data collected by the Camera Sensor through the ISP module, the Camera Sensor can first transfer the image data to the ISP module, and then the ISP module performs ISP processing on the image data, and finally generates preview data or encoded photo data. Among them, the ISP module only accepts the image data transmitted by the Camera Sensor because the ISP module and the Camera Sensor are connected through a physical MIPI interface. That is to say, the path for the ISP module to receive image data is unique, and it can only obtain image data through the Camera Sensor. Therefore, the ISP module cannot be used alone and needs to be used in cooperation with the Camera Sensor.

[0110] Generally, the image data in the Raw Data format output by the Camera Sensor can be copied to the ISP module through the memory.

[0111] Further, in the embodiments of the present application, when the control terminal replaces the image data with the original image data and calls the first ISP module for data processing to obtain the processed data, it may first store the original image data in the first storage space, thereby completing the replacement process of the image data collected by the first shooting device that has been stored; then, the control terminal may control the first ISP module to read the original image data from the first storage space; finally, the control terminal may control the first ISP module to perform image processing according to the original image data and the shooting parameters, and thus the processed data can be obtained.

[0112] It should be noted that, in the embodiments of the present application, the processed data may be image data in YUV format or RGB format.

[0113] Further, in the embodiments of the present application, after the control terminal replaces the image data with the original image data and calls the first ISP module for data processing to obtain the processed data, it may further use the processed data to execute the image data service corresponding to the AR terminal.

[0114] In summary, for the data processing method proposed in the present application, through the above steps 101 to 107, the control terminal can use the configured first ISP module to process the original image data collected by the AR terminal through the second shooting device, thereby improving the performance of data processing and enhancing the quality of the data processing effect. Specifically, in the present application, when the power consumption of the second ISP module of the AR terminal is relatively high, the control terminal can use the first ISP module to process the original image data collected by the AR terminal, thereby balancing the overall power consumption of the AR system and avoiding the control terminal from crashing due to excessive power consumption load.

[0115] It should be noted that the data processing method proposed in the present application can be a general solution, not limited to the Qualcomm platform, and can also be used in hardware platforms such as MTK and Spreadtrum.

[0116] Further, in the embodiments of the present application, the control terminal can also process offline Camera data, such as photos taken by other devices or the control terminal. By performing replacement processing on the image data, subsequent ISP processing can be completed to generate photo data.

[0117] It can be understood that the data processing method proposed in the present application has strong scalability and is not limited to the split AR system. It can also be extended to other similar split devices. At the same time, the data processed by the control terminal is not limited to the image data processed by calling the ISP module, and other modules can also be called to process other data with high complexity or high power consumption. For example, a Graphics Processing Unit (GPU) and other devices can be coordinated to perform data processing.

[0118] An embodiment of the present application provides a data processing method. After a control terminal establishes a connection with an AR terminal in an AR system, it sends a shooting instruction to the AR terminal; wherein, the shooting instruction carries a target format; receives the original image data sent by the AR terminal; wherein, the original image conforms to the target format; based on the target format and the shooting parameters corresponding to the AR terminal, calls a first shooting device to collect image data; replaces the image data with the original image data, and calls a first ISP module to perform data processing to obtain processed data. That is to say, in the embodiment of the present application, the AR terminal in the AR system can transmit the collected original image data to the control terminal, then use the original image data to replace the image data collected by the control terminal, and then the original image data can be input into the first ISP module in the control terminal for data processing. Since the performance and capabilities of the first ISP module are superior to those of the second ISP module, therefore, by using the first ISP module to process the data of the AR terminal, the data processing ability of the AR system can be effectively improved, and the intelligence of the AR system can be enhanced.

[0119] Based on the above embodiment, in another embodiment of the present application, the proposed data processing method of the present application can be applied to an AR terminal in an AR system. Figure 7 Schematic diagram of the implementation process of the data processing method Figure 4 , such as Figure 7 shown, the method for an AR terminal in an AR system to perform data processing may include the following steps:

[0120] Step 201, after establishing a connection with a control terminal in the AR system, receive the shooting instruction sent by the control terminal; wherein, the shooting instruction carries a target format.

[0121] In the embodiment of the present application, after the AR terminal establishes a connection with the control terminal in the AR system, it can receive the shooting instruction sent by the control terminal; wherein, the shooting instruction carries a target format.

[0122] It can be understood that, in the embodiment of the present application, the control terminal and the AR terminal can establish a physical connection through a connection line and an interface, or can establish a wireless connection through a wireless transmission module, and the present application does not make specific limitations.

[0123] Exemplarily, in the embodiment of the present application, Figure 8 Schematic diagram of the AR system Figure 2 , such as Figure 8 shown, the AR system 10 is a split-type AR glasses system, the control terminal 20 in the split-type AR glasses system is a mobile phone, and the AR terminal 30 is AR glasses. Among them, the mobile phone powers the AR glasses through a wireless connection with the AR glasses.

[0124] Further, in the embodiments of the present application, Figure 9 is a schematic structural diagram of an AR glasses. As Figure 9 shown, the AR glasses 40 may include a main body 41. Among them, the main body 41 may include a frame 411, lenses 412, a touch part 413, and a control board 414.

[0125] Specifically, in the present application, the touch part 413 may be arranged on the frame 411, so that different touch gestures can be generated by touching.

[0126] It can be understood that in the present application, the control board 414 may be provided with various functional units such as a detection device, a control module, etc. For example, the control board 414 may be provided with a control unit, a barometric pressure and temperature sensor, a geomagnetic sensor, a light intensity sensor, a nine-axis sensor, a Global Positioning System (GPS) device, a microphone, an audio module, a display module, a camera module, and a communication module.

[0127] Further, in the embodiments of the present application, through the above various functional units, the control board 414 can detect different types of data and information in the surrounding environment. For example, the AR glasses 40 can collect the altitude, barometric pressure, and temperature information of the user's environment periodically / according to a control signal through the barometric pressure and temperature sensor; the AR glasses 40 can periodically detect the orientation of the user's direction through the geomagnetic sensor; the AR glasses 40 can periodically collect, or collect according to a control signal, the daylighting information of the user's environment through the light intensity sensor; the AR glasses 40 can periodically detect, or detect according to a control signal, to obtain human motion information through the nine-axis sensor; the AR glasses 40 can periodically detect, or detect according to a control signal, to obtain the position information of the user through the GPS device; the AR glasses 40 can periodically collect, or collect according to a control signal, the voice information of the user through the microphone; the AR glasses 40 can collect scene information according to a control signal through the camera module; the AR glasses 40 can play audio data or recognize voice according to a control signal through the audio module; the AR glasses 40 can perform corresponding displays according to a control signal through the display module; the AR glasses 40 can perform information interaction with the outside world according to a control signal through the communication module.

[0128] Further, in the embodiments of the present application, after the AR terminal establishes a connection with the control terminal, it can receive a shooting instruction sent by the control terminal, where the shooting instruction is used to instruct the AR terminal to perform shooting processing. Specifically, the shooting instruction may carry a target format.

[0129] It should be noted that, in the embodiments of the present application, the target format is used to limit the format of the image data collected by the AR terminal through the second imaging device. The format of the input data in the ISP processing process is the RAW DATA format. Therefore, the target format in the present application is used to limit the format of the image data sent by the AR terminal to the control terminal to the RAW DATA format, so that the AR terminal can directly send the original image data collected through the second imaging device to the control terminal.

[0130] Furthermore, in the embodiments of the present application, the AR terminal may be configured with a second imaging device and a second ISP module. Among them, the second imaging device is used for collecting image data, and the second ISP module is used for processing image data.

[0131] Exemplarily, in the present application, Figure 10 is a schematic diagram of the AR terminal. As Figure 10 shown, the AR terminal 30 may include a second imaging device 31 and a second ISP module 32.

[0132] Step 202: Determine the shooting parameters, and based on the target format and the shooting parameters, call the second imaging device to collect the original image data; wherein, the original image conforms to the target format.

[0133] In the embodiments of the present application, after receiving the shooting instruction carrying the target format sent by the control terminal, the AR terminal may first determine the shooting parameters, and then based on the target format and the shooting parameters, call the second imaging device to collect the original image data.

[0134] It should be noted that, in the embodiments of the present application, the original image data is the RAW DATA format image data directly output by the second imaging device after the AR terminal calls the second imaging device for shooting processing.

[0135] Furthermore, in the embodiments of the present application, the shooting parameters may be the relevant parameters used by the second imaging device when the AR terminal collects the original image data. Exemplarily, the shooting parameters may include resolution, clarity, brightness, contrast, saturation, focal length, field of view, aperture, gain, exposure time, white balance, etc.

[0136] It should be noted that, in the embodiments of the present application, the shooting parameters corresponding to the AR terminal may be preset by the control terminal and sent to the AR terminal. Specifically, the control terminal may first set the shooting parameters. Correspondingly, the shooting instruction sent by the control terminal to the AR terminal may carry the shooting parameters, that is, the AR terminal obtains the shooting parameters carried by the shooting instruction.

[0137] It should be noted that in the embodiments of the present application, the shooting parameters corresponding to the AR terminal can also be set by the AR terminal and sent to the control terminal. Specifically, after determining the shooting parameters, the shooting parameters can also be sent to the control terminal.

[0138] Step 203: Send the original image data to the control terminal to complete data processing through the first ISP module of the control terminal.

[0139] In the embodiments of the present application, after the AR terminal determines the shooting parameters, and based on the target format and shooting parameters, calls the second shooting device to collect the original image data, the original image data can be sent to the control terminal, so that the data processing can be completed through the first ISP module of the control terminal.

[0140] In the embodiments of the present application, further, Figure 11 Schematic diagram of the implementation process of the data processing method Figure 5 , such as Figure 11 As shown, after determining the shooting parameters, and based on the target format and shooting parameters, calling the second shooting device to collect the original image data, that is, after step 202, the method for the AR terminal to perform data processing may further include the following steps:

[0141] Step 204: Send a processing request to the control terminal.

[0142] Step 205: Receive the processing response returned by the control terminal; wherein, the processing response is used to obtain the original image data.

[0143] In the embodiments of the present application, after the second terminal determines the shooting parameters, and based on the target format and shooting parameters, calls the second shooting device to collect the original image data, it can first send a processing request to the control terminal, and then receive the processing response sent by the control terminal in response to the processing request, so as to further send the original image data to the control terminal.

[0144] It should be noted that in the embodiments of the present application, the processing request is sent by the AR terminal and is used to request the control terminal to perform data processing by means of the first ISP module configured by the control terminal.

[0145] It can be understood that in the embodiments of the present application, after receiving a processing request, the control terminal can first determine the current status and tasks. Specifically, if the control terminal is using the first shooting device or the first ISP module, that is, the first terminal is performing shooting processing or image data processing, then it can be considered that the first ISP module cannot be called to process the original image data sent by the AR terminal. If the control terminal is not using the first shooting device and the first ISP module, that is, the first terminal is not performing shooting processing and image data processing, then it can be considered that the first ISP module is allowed to process the original image data sent by the AR terminal.

[0146] Further, in the embodiments of the present application, after the control terminal determines that the first ISP module is allowed to be called based on the current status and tasks, it can respond to the processing request and send a response for obtaining the original image data processing to the AR terminal.

[0147] Correspondingly, in the present application, after the AR terminal receives the processing response sent by the control terminal, it can send the original image data to the control terminal.

[0148] In the embodiments of the present application, further, Figure 12 Schematic diagram of the implementation process for the data processing method Figure 6 , as Figure 12 shown, after determining the shooting parameters and calling the second shooting device to collect the original image data based on the target format and shooting parameters, that is, after step 202, the method for the AR terminal to perform data processing may further include the following steps:

[0149] Step 206: Based on the original image data, call the second ISP module to perform data processing to obtain processed data.

[0150] Step 207: Send the processed data to the control terminal.

[0151] In the embodiments of the present application, after the AR terminal calls the second shooting device to collect the original image data based on the target format and shooting parameters, it can also directly call the second ISP module to perform data processing based on the original image data, so as to obtain processed data corresponding to the original image data, and then it can send the processed data to the control terminal.

[0152] It should be noted that in the embodiments of the present application, the processed data can be obtained by the AR terminal calling the configured second ISP module to perform data processing on the original image data. Among them, the processed data can be image data in YUV format or RGB format.

[0153] Furthermore, in the embodiments of the present application, since the AR terminal is also configured with a second ISP module capable of performing image data processing, after the AR terminal calls the second imaging device to acquire the original image data, it can choose to directly call the second ISP module to process the original image data, or it can choose to send the original image data to the control terminal to process the original image data by means of the first ISP module configured in the control terminal with higher performance.

[0154] It can be understood that in the embodiments of the present application, for image processing services with simple functions or short time consumption, the second ISP module configured in the AR terminal can complete them well. Therefore, the AR terminal can choose not to process the data by means of the first ISP module configured in the control terminal, but directly call the second ISP module to process the acquired original image information.

[0155] Correspondingly, in the present application, for image processing services with complex functions or long time consumption, the performance and capabilities of the second ISP module configured in the AR terminal cannot complete them well. Therefore, the AR terminal can choose to send the original image data to the control terminal, so that the processing of the original image data can be completed by means of the first ISP module of the control terminal.

[0156] In summary, in the data processing method proposed in the present application, through the above steps 201 to 207, the AR terminal can send the original image data to the control terminal, so that the first ISP module configured in the control terminal can be used to process the original image data collected by the second imaging device, thereby improving the performance of data processing and enhancing the quality of data processing results. Specifically, in the present application, when the power consumption of the second ISP module of the AR terminal is relatively high, the control terminal can use the first ISP module to process the original image data collected by the AR terminal, thereby balancing the overall power consumption of the AR system and avoiding the control terminal from crashing due to excessive power consumption load.

[0157] It should be noted that the data processing method proposed in the present application can be extended to other split devices, such as dash cams, etc. Other devices can borrow the processing capabilities of powerful ISP and other processors in the control terminal to cooperate in business processing.

[0158] It should be noted that the data processing method proposed in the present application can also be applied to scenarios where the AR terminal uses other processing modules of the control terminal, that is, not limited to using the ISP to process image data. The AR terminal can also use other processing modules such as the GPU and Neural-network Processing Unit (NPU) of the control terminal to process data.

[0159] It can be understood that the data processing method proposed in this application has strong scalability and is not limited to the split AR system. It can also be extended to other similar split devices. At the same time, the data processed by the control terminal is not limited to the image data processed by calling the ISP module. Other data with high complexity or high power consumption can also be processed by calling other modules. For example, coordinating devices such as the GPU to perform data processing.

[0160] An embodiment of this application provides a data processing method. After the AR terminal establishes a connection with the control terminal in the AR system, it receives a shooting instruction sent by the control terminal. The shooting instruction carries a target format. Determine the shooting parameters, and based on the target format and the shooting parameters, call the second shooting device to collect the original image data. The original image conforms to the target format. Send the original image data to the control terminal to complete data processing through the first ISP module of the control terminal. That is to say, in the embodiment of this application, the AR terminal in the AR system can transmit the collected original image data to the control terminal, then use the original image data to replace the image data collected by the control terminal, and then input the original image data into the first ISP module in the control terminal for data processing. Since the performance and capabilities of the first ISP module are superior to those of the second ISP module, using the first ISP module to process the data of the AR terminal can effectively improve the data processing ability of the AR system and enhance the intelligence of the AR system.

[0161] Based on the above embodiment, in another embodiment of this application, the proposed data processing method is applied to the AR system. Specifically, the AR system may include a control terminal and an AR terminal. Figure 13 Schematic diagram of the implementation process of the data processing method Figure 7 , such as Figure 13 shown, the method for the control terminal and the AR terminal in the AR system to perform data processing may include the following steps:

[0162] Step 301: After the control terminal establishes a connection with the AR terminal in the AR system, send a shooting instruction to the AR terminal. The shooting instruction carries a target format.

[0163] Step 302: The AR terminal determines the shooting parameters, and based on the target format and the shooting parameters, calls the second shooting device to collect the original image data. The original image conforms to the target format.

[0164] Step 303: The control terminal receives the original image data sent by the AR terminal. The original image conforms to the target format.

[0165] Step 304: Based on the target format and the shooting parameters corresponding to the AR terminal, call the first shooting device to collect image data.

[0166] Step 305: Replace the image data with the original image data, and call the first ISP module to process the data to obtain the processed data.

[0167] Figure 14 Schematic of the implementation process of the data processing method Figure 8 , such as Figure 14 shown, the method for the control terminal and the AR terminal in the AR system to process data may further include the following steps:

[0168] Step 301: After establishing a connection with the AR terminal in the AR system, the control terminal sends a shooting instruction to the AR terminal; wherein, the shooting instruction carries the target format.

[0169] Step 302: The AR terminal determines the shooting parameters, and based on the target format and the shooting parameters, calls the second shooting device to collect the original image data; wherein, the original image conforms to the target format.

[0170] Step 306: The AR terminal sends a processing request to the control terminal.

[0171] Step 307: The control terminal responds to the processing request and sends a processing response to the AR terminal; wherein, the processing response is used to obtain the original image data.

[0172] Step 303: The control terminal receives the original image data sent by the AR terminal; wherein, the original image conforms to the target format.

[0173] Step 304: Based on the target format and the shooting parameters corresponding to the AR terminal, call the first shooting device to collect the image data.

[0174] Step 305: Replace the image data with the original image data, and call the first ISP module to process the data to obtain the processed data.

[0175] In the embodiment of the present application, after the control terminal sends the shooting instruction to the AR terminal, it may first receive the processing request sent by the AR terminal, and then respond to the processing request and send a processing response to the AR terminal, so as to instruct the AR terminal to send the original image data.

[0176] It should be noted that, in the embodiment of the present application, the processing request is sent by the AR terminal and is used to request the control terminal to process the data with the first ISP module configured by the control terminal.

[0177] It can be understood that in the embodiments of the present application, after receiving a processing request, the control terminal can first determine the current status and tasks. Specifically, if the control terminal is using the first shooting device or the first ISP module, that is, the first terminal is performing shooting processing or image data processing, then it can be considered that the first ISP module cannot be called to process the raw image data sent by the AR terminal. If the control terminal is not using the first shooting device and the first ISP module, that is, the first terminal is not performing shooting processing and image data processing, then it can be considered that the first ISP module is allowed to process the raw image data sent by the AR terminal.

[0178] Further, in the embodiments of the present application, after the control terminal determines that the first ISP module is allowed to be called based on the current status and tasks, it can respond to the processing request and send a response for obtaining raw image data processing to the AR terminal.

[0179] Correspondingly, in the present application, after the AR terminal receives the processing response sent by the control terminal, it can send the raw image data to the control terminal.

[0180] Figure 15 Schematic diagram of the implementation process of the data processing method Figure 9 , such as Figure 15 shown, the method for the control terminal and the AR terminal in the AR system to perform data processing may further include the following steps:

[0181] Step 301: After establishing a connection with the AR terminal in the AR system, the control terminal sends a shooting instruction to the AR terminal; wherein, the shooting instruction carries the target format.

[0182] Step 302: The AR terminal determines the shooting parameters, and based on the target format and the shooting parameters, calls the second shooting device to collect raw image data; wherein, the raw image conforms to the target format.

[0183] Step 308: The AR terminal performs data processing on the raw image data by calling the second ISP module to obtain processed data.

[0184] Step 309: The AR terminal sends the processed data to the control terminal.

[0185] In the embodiments of the present application, after the AR terminal calls the second shooting device to collect raw image data based on the target format and the shooting parameters, it can directly perform data processing on the raw image data by calling the second ISP module, so as to obtain processed data corresponding to the raw image data, and then can send the processed data to the control terminal.

[0186] It should be noted that in the embodiments of the present application, the processed data can be obtained by the AR terminal calling the configured second ISP module to process the original image data. Among them, the processed data can be image data in YUV format or RGB format.

[0187] Furthermore, in the embodiments of the present application, since the AR terminal is also configured with a second ISP module capable of processing image data, after the AR terminal calls the second imaging device to collect the original image data, it can choose to directly call the second ISP module to process the original image data, or choose to send the original image data to the control terminal to process the original image data by means of the first ISP module configured with higher performance in the control terminal.

[0188] It can be understood that in the embodiments of the present application, for image processing services with simple functions or short time consumption, the second ISP module configured by the AR terminal can complete them well. Therefore, the AR terminal can choose not to process the data by means of the first ISP module configured by the control terminal, but directly call the second ISP module to process the collected original image information.

[0189] Correspondingly, in the present application, for image processing services with complex functions or long time consumption, the performance and capabilities of the second ISP module configured by the AR terminal cannot complete them well. Therefore, the AR terminal can choose to send the original image data to the control terminal, so that the processing of the original image data can be completed by means of the first ISP module of the control terminal.

[0190] In summary, in the present application, the original image data collected by the AR terminal in the AR system can be transmitted to the control terminal through USB, and then the second ISP module of the control terminal can be called for processing to obtain the processed data, and finally the corresponding service processing can be carried out using the processed data.

[0191] That is to say, in the data processing method proposed in this application, the control terminal can determine shooting parameters such as the resolution of the data output by the AR terminal and the target format. Among them, the format of the original image data output by the AR terminal must meet the requirements for ISP processing by the first ISP module. Specifically, the control terminal can also send a shooting instruction to the AR terminal. The AR terminal calls the second shooting device to collect the original image data. At the same time, the control terminal can also activate the first shooting device and set the specific parameters of the first shooting device according to the shooting parameters corresponding to the AR terminal. After the AR terminal outputs the original image data to the control terminal through USB, the control terminal can replace the image data collected by the first shooting module with the original image data. Specifically, the control terminal can store the original image data in the first storage space and use the technology of in-memory data copying to replace the image data collected by the control terminal. Then, the replaced original image data can be input into the first ISP module and undergo ISP processing.

[0192] It should be noted that in this application, the processed data obtained by calling the first ISP module to perform ISP processing on the original image data is the image data corresponding to the AR terminal. This processed data can be used for business processing corresponding to the AR terminal, such as data synthesis, effect processing, etc.

[0193] It can be understood that in this application, during the process of calling the first ISP module to perform ISP processing on the original image data, since the first ISP module and the first shooting device of the control terminal are occupied, the shooting function and the image processing function of the control terminal cannot be used.

[0194] It should be noted that in this application, when the control terminal calls the first shooting device, there is no need for preview output. Therefore, no preview drawing action is performed, and there will be no situation where the camera preview is displayed.

[0195] That is to say, the data processing method proposed in this application can use the original image data collected by the second shooting device of the AR terminal to replace the image data collected by the first shooting device of the control terminal, and use the first ISP module of the control terminal to process the original image data, thereby ensuring the performance and quality of the original image data of the AR terminal.

[0196] It should be noted that in this application, the control terminal sets and calls the specific parameters of the first shooting module using the shooting parameters corresponding to the AR terminal, which can ensure that the data attributes and formats collected by the shooting devices of the control terminal and the AR terminal are consistent.

[0197] Further, in the embodiments of the present application, the control terminal may adopt the memory copy technology to copy the original image data corresponding to the first terminal to the memory space of the data corresponding to the first shooting device of the control terminal, thereby replacing the image data collected by the first shooting device. Furthermore, in the subsequent ISP processing of the first ISP module, the processing of the original image data of the AR terminal can be realized.

[0198] An embodiment of the present application provides a data processing method. The AR terminal in the AR system can transmit the acquired original image data to the control terminal, then use the original image data to perform replacement processing on the image data collected by the control terminal, and then the original image data can be input into the first ISP module in the control terminal for data processing. Since the performance and capabilities of the first ISP module are superior to those of the second ISP module, by using the first ISP module to process the data of the AR terminal, the data processing ability of the AR system can be effectively improved, and the intelligence of the AR system can be enhanced.

[0199] Based on the above embodiments, in another embodiment of the present application, Figure 16 Schematic diagram of the composition structure of the control terminal Figure 1 , as Figure 16 shown, the control terminal 20 proposed in the embodiment of the present application may include: a first sending unit 24, a first receiving unit 25, a first collecting unit 26, a replacement unit 27, and a processing unit 28.

[0200] The first sending unit 24 is configured to send a shooting instruction to the AR terminal after establishing a connection with the AR terminal in the AR system; wherein, the shooting instruction carries a target format.

[0201] The first receiving unit 25 is configured to receive the original image data sent by the AR terminal; wherein, the original image conforms to the target format.

[0202] The first collecting unit 26 is configured to call the first shooting device to collect image data based on the target format and the shooting parameters corresponding to the AR terminal.

[0203] The replacement unit 27 is configured to replace the image data with the original image data.

[0204] The processing unit 28 is configured to call the first ISP module to perform data processing to obtain processed data.

[0205] In the embodiments of the present application, further, Figure 17 Schematic diagram of the composition structure of the control terminal Figure 2 , as Figure 17As shown, the control terminal 20 proposed in the embodiment of the present application may further include a first processor 29, a first memory 210 storing executable instructions of the first processor 29. Further, the control terminal 20 may further include a first communication interface 211, and a first bus 212 for connecting the first processor 29, the first memory 210, and the first communication interface 211.

[0206] In the embodiment of the present application, the control terminal 20 may further include a first memory 210, and the first memory 210 may be connected to the first processor 29. Among them, the first memory 210 is used to store executable program codes, and the program codes include computer operation instructions. The first memory 210 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0207] In the embodiment of the present application, the first bus 212 is used to connect the first communication interface 211, the first processor 29, and the first memory 210 and for mutual communication between these devices.

[0208] In the embodiment of the present application, the first memory 210 is used to store instructions and data.

[0209] Further, in the embodiment of the present application, the above-mentioned first processor 29 is used to send a shooting instruction to the AR terminal after establishing a connection with the AR terminal in the AR system; wherein, the shooting instruction carries a target format; receive the original image data sent by the AR terminal; wherein, the original image conforms to the target format; call the first shooting device to collect image data based on the target format and the shooting parameters corresponding to the AR terminal; replace the image data with the original image data, and call the first ISP module to perform data processing to obtain processed data.

[0210] Based on the above embodiment, in another embodiment of the present application, Figure 18 Schematic diagram of the composition structure of the AR terminal Figure 1 As Figure 18 shown, the AR terminal 30 proposed in the embodiment of the present application may include: a second receiving unit 33, a determining unit 34, a second collecting unit 35, a second sending unit 36.

[0211] The second receiving unit 33 is used to receive the shooting instruction sent by the control terminal after establishing a connection with the control terminal in the AR system; wherein, the shooting instruction carries a target format.

[0212] The determining unit 34 is used to determine shooting parameters.

[0213] The second acquisition unit 35 is configured to call a second photographing device to acquire original image data based on the target format and the photographing parameters; wherein, the original image conforms to the target format.

[0214] The second sending unit 36 is configured to send the original image data to the control terminal, so that the data processing is completed by the first ISP module of the control terminal.

[0215] In an embodiment of the present application, further, Figure 19 Schematic diagram of the composition structure of the AR terminal Figure 2 , such as Figure 19 shown, the AR terminal 30 proposed in the embodiment of the present application may further include a second processor 37, a second memory 38 storing executable instructions of the second processor 37. Further, the AR terminal 30 may further include a second communication interface 39, and a second bus 310 for connecting the second processor 37, the second memory 38, and the second communication interface 39.

[0216] In an embodiment of the present application, the AR terminal 30 may further include a second memory 38, and the second memory 38 may be connected to the second processor 37. Wherein, the second memory 38 is used to store executable program codes, and the program codes include computer operation instructions. The second memory 38 may include a high-speed RAM memory, and may also include a non-volatile memory, for example, at least two disk memories.

[0217] In an embodiment of the present application, the second bus 310 is used to connect the second communication interface 39, the second processor 37, and the second memory 38 and for mutual communication between these devices.

[0218] In an embodiment of the present application, the second memory 38 is used to store instructions and data.

[0219] Further, in an embodiment of the present application, the above-mentioned second processor 37 is configured to receive a photographing instruction sent by the control terminal after establishing a connection with the control terminal in the AR system; wherein, the photographing instruction carries the target format; determine the photographing parameters, and based on the target format and the photographing parameters, call a second photographing device to acquire original image data; wherein, the original image conforms to the target format; send the original image data to the control terminal, so that the data processing is completed by the first ISP module of the control terminal.

[0220] Based on the above embodiment, in another embodiment of the present application, Figure 20 Schematic diagram of the composition structure of the AR system Figure 1 , such as Figure 20As shown in the figure, the AR system 10 proposed in the embodiments of the present application may include: a control terminal 20 and an AR terminal 30. Among them, the control terminal 20 is configured with a first shooting device 21 and a first ISP module 22, and the AR terminal 30 is configured with a second shooting device 31 and a second ISP module 32.

[0221] Further, in the embodiments of the present application, the control terminal 20 calls the first shooting device 21 and the first ISP module 22 to send a shooting instruction to the AR terminal after establishing a connection with the AR terminal in the AR system; wherein, the shooting instruction carries a target format; receive the original image data sent by the AR terminal; wherein, the original image conforms to the target format; based on the target format and the shooting parameters corresponding to the AR terminal, call the first shooting device to collect image data; replace the image data with the original image data, and call the first ISP module to perform data processing to obtain processed data.

[0222] Further, in the embodiments of the present application, the AR terminal 30 calls the second shooting device 31 and the second ISP module 32 to receive the shooting instruction sent by the control terminal after establishing a connection with the control terminal in the AR system; wherein, the shooting instruction carries a target format; determine the shooting parameters, and based on the target format and the shooting parameters, call the second shooting device to collect the original image data; wherein, the original image conforms to the target format; send the original image data to the control terminal to complete data processing through the first ISP module of the control terminal.

[0223] In the embodiments of the present application, further, Figure 21 Schematic diagram of the composition structure of the AR system Figure 2 , as Figure 21 shown, the AR system 10 proposed in the embodiments of the present application may further include a third processor 11, a third memory 12 storing executable instructions of the third processor 11. Further, the AR system 10 may further include a third communication interface 13 and a third bus 14 for connecting the third processor 11, the third memory 12, and the third communication interface 13.

[0224] In the embodiments of the present application, the AR system 10 may further include a third memory 12, and the third memory 12 may be connected to the third processor 11. Among them, the third memory 12 is used to store executable program codes, and the program codes include computer operation instructions. The third memory 12 may include a high-speed RAM memory and may also include non-volatile memory, for example, at least two disk memories.

[0225] In an embodiment of the present application, the third bus 14 is used to connect the third communication interface 13, the third processor 11, and the third memory 12 and enable mutual communication between these devices.

[0226] In an embodiment of the present application, the third memory 12 is used to store instructions and data.

[0227] The embodiments of the present application provide a control terminal, an AR terminal, and an AR system. After the control terminal establishes a connection with the AR terminal in the AR system, it sends a shooting instruction to the AR terminal; wherein, the shooting instruction carries a target format; it receives the original image data sent by the AR terminal; wherein, the original image conforms to the target format; based on the target format and the shooting parameters corresponding to the AR terminal, it calls the first shooting device to collect image data; replaces the image data with the original image data, and calls the first ISP module to perform data processing to obtain processed data. After the AR terminal in the AR system establishes a connection with the control terminal in the AR system, it receives the shooting instruction sent by the control terminal; wherein, the shooting instruction carries a target format; determines the shooting parameters, and based on the target format and the shooting parameters, calls the second shooting device to collect the original image data; wherein, the original image conforms to the target format; sends the original image data to the control terminal to complete data processing through the first ISP module of the control terminal. That is to say, in the embodiments of the present application, the AR terminal in the AR system can transmit the collected original image data to the control terminal, then use the original image data to replace the image data collected by the control terminal, and then input the original image data into the first ISP module in the control terminal for data processing. Since the performance and capabilities of the first ISP module are superior to those of the second ISP module, by using the first ISP module to process the data of the AR terminal, the data processing ability of the AR system can be effectively improved, and the intelligence of the AR system can be enhanced.

[0228] In addition, each functional module in this embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0229] When the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer AR system (which can be a personal computer, a server, or a network AR system, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0230] An embodiment of the present application provides a computer-readable storage medium, on which a program is stored, and when the program is executed by a processor, it implements the data processing method as described above.

[0231] Specifically, the program instructions corresponding to a data processing method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to a data processing method in the storage medium are read or executed by an electronic device, the following steps are included:

[0232] After establishing a connection with the AR terminal in the AR system, send a shooting instruction to the AR terminal; wherein, the shooting instruction carries a target format.

[0233] Receive the original image data sent by the AR terminal; wherein, the original image conforms to the target format.

[0234] Based on the target format and the shooting parameters corresponding to the AR terminal, call the first shooting device to collect image data.

[0235] Replace the image data with the original image data, and call the first ISP module to perform data processing to obtain processed data.

[0236] Specifically, the program instructions corresponding to a data processing method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to a data processing method in the storage medium are read or executed by an electronic device, the following steps are further included:

[0237] After establishing a connection with the control terminal in the AR system, receive the shooting instruction sent by the control terminal; wherein, the shooting instruction carries a target format.

[0238] Determine the shooting parameters, and based on the target format and the shooting parameters, call the second shooting device to collect the original image data; wherein, the original image conforms to the target format;

[0239] Send the original image data to the control terminal to complete data processing through the first ISP module of the control terminal.

[0240] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0241] The present application is described with reference to the schematic flowcharts and / or block diagrams of the methods, AR systems (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the schematic flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of the processes and / or blocks in the schematic flowcharts and / or block diagrams can also be implemented. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing AR systems to generate a machine, so that the instructions executed by the processors of the computer or other programmable data processing AR systems generate means for implementing the specified functions in one or more processes or a combination of multiple processes and / or blocks in the schematic flow Figure 1 one process or multiple processes and / or blocks Figure 1 means for implementing the specified functions in one or more blocks or a combination of multiple blocks.

[0242] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing AR system to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implement the specified functions in one or more processes or a combination of multiple processes and / or blocks in the schematic flow Figure 1 one process or multiple processes and / or blocks Figure 1 means for implementing the specified functions in one or more blocks or a combination of multiple blocks.

[0243] These computer program instructions can also be loaded onto a computer or other programmable data processing AR system, so that a series of operation steps are executed on the computer or other programmable AR system to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable AR system provide steps for implementing the specified functions in one or more processes or a combination of multiple processes and / or blocks in the schematic flow Figure 1 one process or multiple processes and / or blocks Figure 1 means for implementing the specified functions in one or more blocks or a combination of multiple blocks.

[0244] As described above, it is only the preferred embodiment of the present application and is not intended to limit the protection scope of the present application.

Claims

1. A data processing method, characterized in that, The described data processing method is applied to a control terminal in a split system. The control terminal supplies power to the split device in the split system through a connection with the split device. The method includes: After establishing a connection with the split device, send a shooting instruction to the split device; When the image processing service meets a preset condition, receive the processed data sent by the split device and corresponding to the original image data collected by the split device; wherein, the processed data is obtained by the split device invoking the second image signal processor (ISP) module in the split device for data processing; When the image processing service does not meet the preset condition, receive the original image data sent by the split device; invoke a first shooting device to collect image data; replace the image data with the original image data, and invoke the first ISP module for data processing to obtain processed data.

2. The method according to claim 1, wherein The shooting instruction carries a target format; the original image conforms to the target format; Correspondingly, the invoking the first shooting device to collect image data includes: Based on the target format and the shooting parameters corresponding to the split device, invoke the first shooting device to collect image data.

3. The method according to claim 1 or 2, characterized in that, Before receiving the original image data sent by the split device, the method further includes: Receive a processing request sent by the split device; In response to the processing request, send a processing response to the split device; wherein, the processing response is used to obtain the original image data.

4. The method according to claim 2, wherein Before invoking the first shooting device to collect image data, the method further includes: Set the shooting parameters; Correspondingly, the shooting instruction carries the shooting parameters.

5. The method according to claim 2, wherein Before invoking the first shooting device to collect image data, the method further includes: Receive the shooting parameters sent by the split device.

6. The method according to claim 2, wherein After invoking the first shooting device to collect image data, the method further includes: Store the image data in a first storage space.

7. The method according to claim 6, characterized in that, The replacing the image data with the original image data and invoking the first ISP module for data processing to obtain processed data includes: Store the original image data in the first storage space to replace the image data; Control the first ISP module to read the original image data from the first storage space; Control the first ISP module to perform image processing according to the original image data and the shooting parameters to obtain the processed data.

8. The method according to claim 1 or 2, characterized in that, After the replacing the image data with the original image data and invoking the first ISP module for data processing to obtain processed data, the method further includes: Execute the image data service corresponding to the split device by using the processed data.

9. A data processing method, characterized in that, The data processing method is applied to a split device in a split system. The split device is powered by a control terminal in the split system through a connection with the split device. The method includes: After establishing a connection with the control terminal in the split system, receive the shooting instruction sent by the control terminal; Invoke the second imaging device to collect the original image data; When the image processing service meets the preset conditions, based on the original image data, invoke the second ISP module to process the data and obtain the processed data; send the processed data to the control terminal; When the image processing service does not meet the preset conditions, send the original image data to the control terminal to complete data processing through the first ISP module of the control terminal.

10. The method according to claim 9, wherein The imaging instruction carries the target format; the original image conforms to the target format; Correspondingly, invoking the second imaging device to collect the original image data includes: Determine the imaging parameters, and based on the target format and the imaging parameters, invoke the second imaging device to collect the original image data.

11. The method according to claim 9 or 10, characterized in that, After invoking the second imaging device to collect the original image data, the method further includes: Send a processing request to the control terminal; Receive the processing response returned by the control terminal; where the original image data is obtained.

12. The method according to claim 10, wherein Determining the imaging parameters includes: Obtain the imaging parameters carried by the imaging instruction.

13. The method according to claim 10, wherein After determining the imaging parameters, the method further includes: Send the imaging parameters to the control terminal.

14. A control terminal, characterized in that, The control terminal in the split system powers the split device through the connection with the split device in the split system. The control terminal includes: a first sending unit, a first receiving unit, a first acquisition unit, a replacement unit, and a processing unit. The first sending unit is configured to send an imaging instruction to the split device after establishing a connection with the split device in the split system; The first receiving unit is configured to receive the processed data corresponding to the original image data collected by the split device sent by the split device when the image processing service meets the preset conditions; where the processed data is obtained by the split device invoking the second image signal processor (ISP) module in the split device for data processing; The first receiving unit is further configured to receive the original image data sent by the split device when the image processing service does not meet the preset conditions; The first acquisition unit is configured to invoke the first imaging device to collect image data; The replacement unit is configured to replace the image data with the original image data; The processing unit is configured to invoke the first ISP module to process the data and obtain the processed data.

15. A control terminal, characterized in that, The control terminal includes a first processor and a first memory storing executable instructions of the first processor. When the instructions are executed by the first processor, the method described in any one of claims 1-8 is implemented.

16. A split device, characterized in that, The split device in the split system is powered by the control terminal in the split system through the connection with the split device. The split device includes: a second receiving unit, a second acquisition unit, and a second sending unit. The second receiving unit is configured to receive the imaging instruction sent by the control terminal after establishing a connection with the control terminal in the split system; The second acquisition unit is configured to call a second imaging device to acquire original image data; The second sending unit is configured to, when an image processing service meets a preset condition, based on the original image data, call a second ISP module to perform data processing, and after obtaining processed data, send the processed data to the control terminal; The second sending unit is further configured to, when the image processing service does not meet the preset condition, send the original image data to the control terminal so that the first ISP module of the control terminal completes data processing.

17. A split device, characterized in that, The split device includes a second processor and a second memory storing executable instructions of the second processor. When the instructions are executed by the second processor, the method according to any one of claims 9-13 is implemented.

18. A split system, characterized in that, The split system includes: a control terminal and a split device. The control terminal supplies power to the split device through a connection with the split device. Wherein, the control terminal is configured with a first imaging device and a first ISP module, and the split device is configured with a second imaging device and a second ISP module. The control terminal calls the first imaging device and the first ISP module to implement the method according to any one of claims 1-8, and the split device calls the second imaging device and the second ISP module to implement the method according to any one of claims 9-13.

19. A split-type system, characterized in that, The split system includes a third processor and a third memory storing executable instructions of the third processor. When the instructions are executed by the third processor, the method according to any one of claims 1-13 is implemented.

20. A computer-readable storage medium, on which a program is stored and applied to a split system, characterized in that When the program is executed by a first processor, the method according to any one of claims 1-8 is implemented. When the program is executed by a second processor, the method according to any one of claims 9-13 is implemented.

Images