Abnormal processing method, device and collaborative working system in image data transmission process

Abstract

The present application provides an exception handling method, device, and collaborative working system for an image data transmission process. The method sets the network packets generated by the second electronic device to carry packet identification information and status identification information, so that the first electronic device that receives these network packets can determine whether the network packet is an abnormal feedback packet or an abnormal packet based on the packet identification information and status identification information without parsing the network packets, that is, without knowing the specific content carried in the data body of the network packets. In this way, it can be known in advance that the image data requested this time cannot be displayed, and then countermeasures can be taken, such as rebuilding the image data channel and reacquiring the image data.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to an exception handling method, device and collaborative working system for an image data transmission process. Background Art

[0002] With the advancement of communication technology, data transmission, such as image data transmission, can be carried out between any two electronic devices based on business needs. Considering that image data is usually large, to speed up transmission and avoid network congestion, the image data of a large photo, such as 100K to 20M, is usually broken down into multiple image data packets. These packets are then placed in a queue to be transmitted, and subsequently retrieved from the queue for transmission.

[0003] However, with current image data transmission, the receiver cannot determine whether the received data packets are abnormal without first parsing the image data packets. Therefore, the receiver must wait for the sender to send all the image data packets and then parse each packet to determine whether the image data it carries is normal. This not only prevents the other device from restoring the image data but also wastes device and network resources. Summary of the Invention

[0004] In order to solve the above technical problems, the present application provides an exception handling method, device and collaborative working system for the image data transmission process, which aims to enable the electronic device to know in advance that the image data requested cannot be displayed without parsing the data body of the received data packet, so that the image data channel can be rebuilt in time and the image data can be retrieved.

[0005] In a first aspect, the present application provides an exception handling method for an image data transmission process. The method applies a first electronic device, comprising: receiving a network packet sent by a second electronic device, the network packet carrying packet identification information and status identification information; when the packet identification information indicates that the network packet is an abnormal feedback packet, parsing the data body of the abnormal feedback packet, extracting the error code carried in the abnormal feedback packet, and reporting the error code to a target application; when the packet identification information indicates that the network packet is an image data packet, determining whether the image data packet is normal based on the status identification information; when it is determined that the image data packet is abnormal based on the status identification information, deleting the parsed image data, clearing the cache, and reporting the cause of the abnormality to the target application.

[0006] The first electronic device and the second electronic device may be electronic devices of the same type, such as any one or more of mobile phones, tablets, etc., which have powerful processing capabilities.

[0007] The types of the first electronic device and the second electronic device may also be different. For example, the first electronic device may be an electronic device with powerful processing capabilities such as a mobile phone or a tablet, and the second electronic device may be an Internet of Things device.

[0008] For example, in a scenario where the second electronic device is an IoT device, it may be a desk lamp, for example.

[0009] For ease of explanation, this application takes the first electronic device as an electronic device with powerful processing capabilities such as a mobile phone or tablet as an example, and the second electronic device as an Internet of Things device such as a desk lamp as an example.

[0010] Among them, when the network packet is an image data packet, it can be specifically divided into a first packet, a middle packet and a tail packet.

[0011] For example, the packet identification information of the first packet is represented by "10" mentioned below, the packet identification information of the middle packet is represented by "00" mentioned below, and the packet identification information of the last packet is represented by "01" mentioned below.

[0012] Exemplarily, the packet identification information of the abnormal feedback packet is represented by “11” mentioned below.

[0013] For example, the status identification information indicating that the network packet is normal is represented by "0" mentioned below, and the abnormal status is represented by "1".

[0014] Regarding the data structure of the first packet, middle packet, and tail packet in different status identification information, excluding the data structure of the data header, the structure of its identification field and data body can be found in Figures 18a to 20b , I will not go into details here.

[0015] Regarding the data structure of the exception feedback packet excluding the data header, its identification field and data body structure can be found in Figure 21 , I will not go into details here.

[0016] Among them, the operations of parsing network packets, deleting the parsed image data, and reporting abnormalities are implemented by the network packet assembly module below, and the operations of receiving network packets and clearing the cache are implemented by the network packet collection module below.

[0017] Therefore, the method sets the network packets generated by the second electronic device to carry packet identification information and status identification information, so that the first electronic device that receives these network packets does not need to parse the network packets, that is, it does not need to know the specific content carried in the data body of the network packets. It can determine whether the network packet is an abnormal feedback packet or an abnormal packet based on the packet identification information and status identification information, so that it can know in advance that the image data requested this time cannot be displayed, and then extract and take countermeasures, such as rebuilding the image data channel, re-acquiring image data, etc.

[0018] According to the first aspect, the network packet includes a data header, an identification field, and a data body; the data header occupies 12 bytes, the identification field occupies 1 byte; the combined content of the binary data corresponding to the 0th and 1st frames in the identification field is the packet identification information, and the binary data corresponding to the 3rd frame in the identification field is the status identification information; the data body is used to store image data or error codes. In this way, by setting the network packet to follow the above data structure, when the network packet is received, the first electronic device directly identifies the content in frames 0 to 3 in the byte where the identification field is located, and can quickly and accurately determine the attributes of the current network packet, such as whether it is an abnormal feedback packet or an image data packet, and whether the image data packet is normal.

[0019] According to the first aspect, or any implementation of the first aspect above, when the packet identification information indicates that the network packet is an exception feedback packet, the data body occupies 128 bytes; the error code is filled in bytes 0 to 3 of the data body. In this way, when the network packet is an exception feedback packet, the data body is set to only 128 bytes, thereby reducing transmission and parsing pressure, thereby quickly reporting the exception.

[0020] According to the first aspect, or any implementation of the first aspect above, when the packet identification information indicates that the network packet is an abnormal feedback packet, the data body of the abnormal feedback packet is parsed, and the error code carried in the abnormal feedback packet is extracted, including: parsing the data body of the abnormal feedback packet and extracting the error code from bytes 0 to 3 of the data body. In this way, when the network packet is an abnormal feedback packet, the content is directly extracted from bytes 0 to 3 after parsing the data body, without paying attention to the content of other fields, which is convenient and fast.

[0021] According to the first aspect, or any implementation of the first aspect above, when the packet identification information indicates that the network packet is an image data packet, and the image data packet is the first packet, the data body includes a first data portion and a second data portion; the first data portion is a reserved field, occupying 128 bytes, for carrying extended information; the second data portion is used to fill in the image data. Thus, by distinguishing between the reserved field and the image data field, different content can be obtained from different portions according to business needs during subsequent parsing.

[0022] According to the first aspect, or any implementation of the first aspect above, the image data packet is the first packet; the method also includes: when it is determined that the image data packet is normal according to the status identification information, parsing the image data packet and extracting the image data from the second data part.

[0023] According to the first aspect, or any implementation of the first aspect above, when the packet identification information indicates that the network packet is an image data packet, and the image data packet is an intermediate packet or a tail packet, the data body is filled with image data starting from 0 bytes.

[0024] According to the first aspect, or any implementation of the first aspect above, the image data packet is an intermediate packet or a tail packet; the method also includes: when it is determined that the image data packet is normal based on the status identification information, the image data packet is parsed and the image data is extracted starting from byte 0 of the data body.

[0025] According to the first aspect, or any implementation of the first aspect above, the method also includes: when receiving a new intermediate packet, determining whether the extracted image data is greater than a set threshold; when the extracted image data is greater than the set threshold, marking the new intermediate packet as over-limit and discarding the new intermediate packet; when receiving the last packet or a new first packet, deleting the extracted image data, clearing the cache, and reporting the cause of the exception to the target application.

[0026] According to the first aspect, or any implementation of the first aspect above, upon receiving a tail packet or a new first packet, the extracted image data is deleted and the cache is cleared, including: upon receiving a tail packet, deleting the extracted image data and clearing all cached image data packets; upon receiving a new first packet, deleting the extracted image data and clearing all cached image data packets except the new first packet. In this way, different cache clearing strategies are set based on the attributes of the received network packet, such as whether it is a tail packet or a first packet, thereby avoiding the deletion of network packets that need to be retained.

[0027] According to the first aspect, or any implementation of the first aspect above, the method also includes: starting a timer when sending a photo request to the second electronic device; during the timer timing process, receiving a network packet sent by the second electronic device, and processing the network packet according to the packet identification information and status identification information in the network packet; during the processing of the network packet, when the timer timing reaches the set timeout period, destroying the thread processing the network packet, and sending a destruction instruction to the second electronic device, so that the second electronic device destroys the image data channel between it and the first electronic device.

[0028] The network packet is processed according to the packet identification information and the status identification information in the network packet, for example:

[0029] When the packet identification information in the network packet indicates that the network packet is an abnormal feedback packet, the data body of the abnormal feedback packet is parsed, the error code carried in the abnormal feedback packet is extracted, and the error code is reported to the target application; when the packet identification information in the network packet indicates that the network packet is an image data packet, whether the image data packet is normal is determined based on the status identification information in the network packet; when it is determined that the image data packet is abnormal based on the status identification information, the parsed image data is deleted, the cache is cleared, and the cause of the abnormality is reported to the target application; when it is determined that the image data packet is normal based on the status identification information, the image data packet is parsed and the image data carried in the image data packet is extracted.

[0030] In this way, by automatically destroying the thread processing the network packet and sending a destruction instruction to the second electronic device, the second electronic device destroys the image data channel between the second electronic device and the first electronic device, thereby reducing the resource occupation of the tablet.

[0031] In a second aspect, the present application provides an electronic device. The electronic device includes: a memory and a processor, the memory and the processor being coupled; the memory storing program instructions, which, when executed by the processor, cause the electronic device to execute instructions of the method of the first aspect or any possible implementation of the first aspect.

[0032] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0033] In a third aspect, the present application provides a collaborative work system. The system includes: a first electronic device for executing the method instructions in the first aspect or any possible implementation of the first aspect, and a second electronic device, wherein the second electronic device is provided with a camera, the camera is used to collect image data, and the target application in the first electronic device is bound to the second electronic device; the first electronic device is used to: register a virtual camera corresponding to the camera in the system, and send a photo request to the second electronic device by calling the virtual camera; the second electronic device is used to: call the camera to capture image data according to the photo request of the first electronic device, and send the image data to the target application of the first electronic device for preview display.

[0034] According to the third aspect, the second electronic device is an Internet of Things device.

[0035] According to the third aspect, or any implementation of the third aspect above, the IoT device is a desk lamp, and the camera is used to collect image data downward.

[0036] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0037] In a fourth aspect, the present application provides a computer-readable medium for storing a computer program, wherein the computer program includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0038] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0039] In a fifth aspect, the present application provides a computer program product, comprising a computer program including instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0040] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0041] In a sixth aspect, the present application provides a chip comprising a processing circuit and a transceiver pin. The transceiver pin and the processing circuit communicate with each other via an internal connection path, and the processing circuit executes the method of the first aspect or any possible implementation of the first aspect to control the receive pin to receive a signal and to control the transmit pin to send a signal.

[0042] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1a-Figure 1b An application scenario is shown as an example;

[0044] Figure 2a is a schematic diagram of the hardware structure of an electronic device shown as an example;

[0045] Figure 2b is a schematic diagram of the software structure of an electronic device shown as an example;

[0046] Figure 3a Schematic diagram of the hardware structure of an IoT device shown as an example;

[0047] Figure 3b Schematic diagram of the software structure of an IoT device shown as an example;

[0048] Figure 4a A schematic diagram of module interaction provided in an embodiment of the present application;

[0049] Figure 4b A schematic diagram of module interaction provided in an embodiment of the present application;

[0050] Figure 5a-5b An application scenario is shown as an example;

[0051] Figure 6a A schematic diagram of module interaction provided in an embodiment of the present application;

[0052] Figure 6b A schematic diagram of module interaction provided in an embodiment of the present application;

[0053] Figure 7a-7b An application scenario is shown as an example;

[0054] Figure 8 A schematic diagram of module interaction provided in an embodiment of the present application;

[0055] Figure 9a-9c An application scenario is shown as an example;

[0056] Figure 10 A schematic diagram of module interaction provided in an embodiment of the present application

[0057] Figure 11a-Figure 11b An application scenario is shown as an example;

[0058] Figure 12 This is a schematic diagram illustrating an exemplary embodiment of image data transmission between a tablet and a desk lamp;

[0059] Figure 13 FIG1 is a schematic diagram showing the data structure of the entire image data packet;

[0060] Figure 14 is a schematic diagram showing the structure of a data header in an exemplary image data packet;

[0061] Figure 15 is a schematic diagram of the data structure of an exemplary image data packet;

[0062] Figure 16 Schematic diagram of the structure of the RTP data header in an exemplary image data packet;

[0063] Figure 17 Schematic diagram of the structure of the identification field in an exemplary image data packet;

[0064] Figure 18a Schematic diagram of the structure of an image data packet when the first packet is a normal packet;

[0065] Figure 18b Schematic diagram of the structure when the first packet of the image data packet is an abnormal packet;

[0066] Figure 19a This is a schematic structural diagram of an example of an image data packet in which the intermediate packet is a normal packet;

[0067] Figure 19b This is a schematic structural diagram showing an example of an image data packet in which the intermediate packet is an abnormal packet;

[0068] Figure 20a Schematic diagram of the structure when the tail packet of the image data packet is a normal packet;

[0069] Figure 20b Schematic diagram of the structure when the tail packet of the image data packet is an abnormal packet;

[0070] Figure 21 is a schematic diagram of the structure of an abnormal feedback package shown as an example;

[0071] Figure 22 This is a schematic diagram of modules involved in image data processing on a desk lamp side and a tablet side;

[0072] Figure 23 Schematic diagram of an exception handling method when the desk lamp side cannot obtain image data;

[0073] Figure 24 Schematic diagram of an exception handling method when an exception occurs on the desk lamp side during transmission;

[0074] Figure 25 This is a schematic diagram illustrating an exception handling method when the image data packets received by the tablet side exceed a set threshold;

[0075] Figure 26This is a schematic diagram illustrating an exception handling method when the time it takes for a tablet to receive an image data packet exceeds a set time. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0077] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0078] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0079] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0080] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0081] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0082] With the development of the internet, online education has become increasingly popular, and users (such as students) are increasingly demanding it. In some scenarios, when encountering unfamiliar words, students can look up the relevant explanations online; in other scenarios, online reading of textbook content facilitates students' learning and pronunciation; and in still other scenarios, students need to submit their homework online. Therefore, how to meet users' online education needs based on smart devices is a challenge that needs to be addressed.

[0083] Currently, for online education, users typically use smart learning machines that combine both camera and display functions. These devices require cameras or mirrors in specialized locations to capture images of books, making them less versatile and user-friendly. Furthermore, devices with both camera and display capabilities require robust hardware and system support, resulting in high device costs. Consequently, the challenge remains to provide users with a superior online education experience based on smart devices, while enhancing their versatility and usability and reducing the cost of online education.

[0084] A collaborative work system provided by an embodiment of the present application can be applied to online education scenarios. The system working system includes an electronic device and a desk lamp that establish a communication connection. The desk lamp is provided with a camera that can be used to shoot books downward. The electronic device calls the camera of the desk lamp to capture images, and combines them with the platform's online education resources to meet the user's online education needs. Among them, the electronic device can be a tablet computer or a mobile phone. In addition to online education scenarios, electronic devices and desk lamps can also serve users based on their respective basic functions (i.e., communication functions and lighting functions). In this way, the system working system can create a better online education experience for users based on two smart devices commonly used by users, with strong versatility and ease of use. In addition, since tablet computers or mobile phones have long been essential products for every household, and the cost of a desk lamp device with only a shooting function will be lower than that of a device with both shooting and display functions, the user's online education cost is greatly reduced.

[0085] The following uses a tablet as an example to explain the technical solution provided by this application.

[0086] Figure 1a An example application scenario is shown. Figure 1aAs shown, the collaborative work system includes a tablet 100 and a desk lamp 200 that establish a communication connection. The desk lamp 200 includes a camera 201 for capturing downward images, for example, by shooting downward at the text or pictures in a book. An educational app (Application) is installed on the tablet 100. The educational app can use the camera 201 of the desk lamp 200 to capture images and provide users with various online education functions based on the images captured by the camera 201 of the desk lamp 200, such as online word search, online reading, and online homework submission.

[0087] Although the tablet also has a front camera and a rear camera, no matter which camera is used to take pictures of the book, the user needs to hold the tablet and point the camera at the book, which not only makes the shooting unstable, but also affects the user's finger reading and point reading operations, and cannot provide users with a good online education experience. Figure 1a As shown, both the tablet 100 and the desk lamp 200 can be placed in fixed positions. The tablet 100 uses the camera 201 of the desk lamp 200 to shoot a book. The stable shooting image makes the content recognition success rate high, and the user can also flexibly perform finger reading and point reading operations in the book. Therefore, the linkage between the tablet and the desk lamp can provide users with a better online education experience.

[0088] like Figure 1b As shown, tablet 100 and lamp 200 can perform near-field communication or far-field communication. Near-field communication can be accomplished through devices such as routers to exchange information between devices, while far-field communication can be accomplished through cloud servers to exchange information between devices. For example, tablet 100 and lamp 200 can implement near-field communication based on Wi-Fi (wireless fidelity) network protocols.

[0089] like Figure 2a The figure shows the structure of the electronic device 100. Optionally, the electronic device 100 may be a terminal, which may also be called a terminal device. The terminal may be a cellular phone or a tablet computer, etc., which is not limited in this application. It should be noted that the structure of the electronic device 100 may be applicable to Figure 1a~Figure 1b It should be understood that Figure 2a The illustrated electronic device 100 is merely one example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration of components. Figure 2a The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0090] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0091] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0092] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0093] The processor 110 may further include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0094] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0095] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0096] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0097] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0098] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0099] The mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0100] The wireless communication module 160 can provide wireless communication solutions applied to the electronic device 100, including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc.

[0101] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0102] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0103] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0104] In the embodiment of the present application, the display screen 194 can display a shooting preview interface and a photo image interface, etc. It should be noted that, in the embodiment of the present application, the shooting preview interface refers to an interface where a user can view images captured in real time by the desk lamp camera through the display screen 194 .

[0105] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0106] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, for example, enabling the electronic device 100 to implement the collaborative working method in the embodiment of the present application. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0107] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0108] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0109] Speaker 170A, also known as a "horn," is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music or make hands-free calls through speaker 170A. In some embodiments, electronic device 100 can be provided with multiple speakers 170A.

[0110] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0111] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0112] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0113] The pressure sensor is used to sense pressure signals and convert the pressure signals into electrical signals. In some embodiments, the pressure sensor can be provided on the display screen 194. The electronic device 100 can also calculate the touch position based on the detection signal of the pressure sensor.

[0114] A touch sensor, also known as a "touch panel," can be mounted on display screen 194. The touch sensor and display screen 194 together form a touch screen, also known as a "touch screen." The touch sensor detects touch operations applied to or near the touch sensor. The touch sensor can communicate the detected touch operations to the application processor to determine the type of touch event.

[0115] Keys 190 include a power button (also known as a power button), a volume button, and the like. Keys 190 may be mechanical or touch-sensitive. Electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of electronic device 100.

[0116] Motor 191 can generate vibration alerts. This can be used for incoming call vibration alerts or touch vibration feedback. For example, different touch operations (such as taking photos, playing audio, etc.) can correspond to different vibration feedback effects.

[0117] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0118] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0119] Figure 2b It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0120] The layered architecture of electronic device 100 divides the software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into four layers: application layer, application framework layer, system layer, HAL layer (Hardware Abstract Layer), and kernel layer.

[0121] The application layer can include a series of application packages.

[0122] like Figure 2b As shown, the application package may include call, video, Bluetooth, camera, WLAN, education applications, device manager applications, etc. The application package may also include calendar, map, navigation, music, short message applications, etc.

[0123] Among them, educational applications can be used to provide users with online educational functions, such as online word recognition, online reading, online homework submission, etc.

[0124] In some examples, a device manager app can be used to bind IoT devices such as a desk lamp. In some examples, an education app can be used to bind IoT devices such as a desk lamp.

[0125] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0126] like Figure 2b As shown, the application framework layer may include camera services, authentication services, hardware virtualization services, device management services, transmission management services, etc.

[0127] The camera service can be used to call the camera (including the front camera and / or the rear camera) in response to a request from an application.

[0128] In an embodiment of the present application, the camera service can be used to call the virtual camera on the electronic device side, that is, to call the camera in the IOT device, in response to a request from an application.

[0129] Authentication services are used to provide security permission management capabilities.

[0130] Hardware virtualization services can be used to establish a logical channel between the electronic device side (i.e., the central device side) and the IoT device side, providing the ability to virtualize the camera.

[0131] Device management services can be used for IoT device discovery and management, providing far-field (i.e., cloud-based) IoT device information and near-field (i.e., nearby connectable) IoT device information for applications such as educational applications.

[0132] Transmission management services can be used to establish physical transmission channels and provide data transmission capabilities.

[0133] In addition, it can also include window managers, content providers, view systems, phone managers, resource managers, notification managers, etc.

[0134] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0135] The content provider is used to store and obtain data and make the data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures. The phone manager is used to provide communication functions for the electronic device 100. For example, the management of call status (including connecting, hanging up, etc.). The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction.

[0136] The system library and runtime layer (i.e., the system layer) includes the system library and the Android runtime.

[0137] The Android Runtime consists of core libraries and a virtual machine. The Android Runtime is responsible for scheduling and management of the Android system.

[0138] The core library consists of two parts: one containing the Java language's callable functions and the other the Android core library. The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0139] In the embodiment of the present application, Android Runtime also includes a virtual camera adaptation layer that provides the ability to register a virtual camera.

[0140] The system library in the system layer can include multiple functional modules, such as multimedia platform, graphics and image processing library, codec, etc.

[0141] The multimedia platform can be used for multimedia management, supporting playback and recording of a variety of common audio and video formats, as well as static image files. The multimedia platform can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0142] The graphics and image processing library can be used to implement graphics drawing, image rendering, compositing, and layer processing.

[0143] Codecs can be used to implement encoding and decoding operations on audio data and video data.

[0144] The HAL layer is the interface layer between the operating system kernel and the hardware circuit. The HAL layer includes but is not limited to: audio HAL, sensor HAL, modem HAL, camera HAL, and virtual camera HAL.

[0145] The audio HAL is used to process audio streams, for example, by performing noise reduction and directional enhancement. The camera HAL is used to process image streams corresponding to the camera on the electronic device side, and the virtual camera HAL is used to process image streams corresponding to the virtual camera registered on the electronic device side, that is, to process image streams captured by the camera on the IoT device side.

[0146] The kernel layer is the layer between hardware and software. It includes at least the display driver, camera area, audio driver, network driver (such as Wi-Fi driver), CPU driver, USB driver, storage driver, and printer driver. Hardware includes at least the processor, display, and Wi-Fi module.

[0147] It is understandable that Figure 2bThe layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0148] like Figure 3a The schematic diagram of the hardware structure of the Internet of Things device 200 is shown. It should be noted that the schematic diagram of the structure of the Internet of Things device 200 can be applied to Figure 1a~Figure 1b It should be understood that Figure 3a The illustrated IoT device 200 is merely one example of an electronic device, and the IoT device 200 may have more or fewer components than shown, may combine two or more components, or may have a different configuration of components. Figure 3a The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0149] The IoT device 200 may include: a processor 210, a camera 201, a wireless communication module 202, a memory 203, an audio module 204, a USB interface 205, a charging management module 206, a power management module 207, a battery 208, a lighting device 209, a button 211, etc.

[0150] The processor 210 may include one or more processing units, for example, the processor 210 may include a GPU, an ISP, a controller, a memory, a video codec, etc. The different processing units may be independent devices or integrated into one or more processors.

[0151] The controller can be the nerve center and command center of the IoT device 200. The controller can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

[0152] Camera 201 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. Taking a desk lamp as an example, the camera 201 can be set on the desk lamp stand to capture images downward.

[0153] The IoT device 200 can implement a shooting function through an ISP, a camera 201, a video codec, a GPU, and the like.

[0154] The ISP processes data fed back by camera 201. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 201.

[0155] Wireless communication module 202 can provide wireless communication solutions for IoT device 200, including WLAN (e.g., Wi-Fi network) and Bluetooth (BT). In some embodiments, the antenna of IoT device 200 is coupled to wireless communication module 202, enabling IoT device 200 to communicate with the network and other devices via wireless communication technologies.

[0156] The memory 203 can be used to store computer-executable program code, which includes instructions. The processor 210 executes the instructions stored in the memory 203 to execute various functional applications and data processing of the IoT device 200, for example, enabling the IoT device 200 to implement the collaborative working method in the embodiment of the present application.

[0157] The IoT device 200 can implement audio functions, such as music playback, through the audio module 204, the speaker 212, etc.

[0158] The USB interface 205 is an interface that complies with USB standards, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 205 can be used to connect a charger to charge the IoT device 200, and can also be used to transfer data between the IoT device 200 and peripheral devices.

[0159] The charging management module 206 is configured to receive charging input from a charger. While charging the battery 208 , the charging management module 206 can also provide power to the IoT device 200 through the power management module 207 .

[0160] The power management module 207 is used to connect the battery 208, the charging management module 206, and the processor 210. The power management module 207 receives input from the battery 208 and / or the charging management module 206 to provide power to the processor 210, the memory 203, the camera 201, the wireless communication module 202, and the lighting device 209.

[0161] The buttons 211 include a power button (or power button) and the like.

[0162] The software system of the Internet of Things device 200 may adopt a layered architecture or other architectures, etc. In the embodiment of the present application, the layered architecture is used as an example to illustrate the software structure of the Internet of Things device 200.

[0163] Figure 3b This is a software structure block diagram of the Internet of Things device 200 according to an embodiment of the present application.

[0164] The layered architecture of IoT device 200 divides the software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the system of IoT device 200 is divided into three layers: the application layer, the application framework layer, the system layer, and the kernel layer, from top to bottom.

[0165] like Figure 3b As shown, the application layer may include device application services, which may be understood as system-level applications. The device application services are started after the system of the Internet of Things device 200 is started.

[0166] like Figure 3b As shown, the application framework layer may include device interconnection services, hardware abstraction services, resource managers, etc.

[0167] Device interconnection services can be used to establish physical transmission channels, provide data transmission capabilities, and manage the startup switches of hardware abstraction services.

[0168] The hardware abstraction service can be used to establish a logical channel between the electronic device side (i.e., the central device side) and the IoT device, provide the ability to virtualize the camera, and provide an open camera interface for the IoT device.

[0169] The resource manager can provide various resources to applications.

[0170] like Figure 3b As shown, the system layer may include a multimedia platform, a graphics and image processing library, a codec, a device adaptation module, etc.

[0171] The multimedia platform can be used for multimedia management and supports a variety of commonly used audio, video, and static image files. The multimedia platform can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0172] The graphics and image processing library can be used to implement graphics drawing, image rendering, compositing, and layer processing.

[0173] Codecs can be used to implement encoding and decoding operations on audio data and video data.

[0174] The device adaptation module can implement the interface of hardware abstraction service, provide device information and capability query, and provide functions for performing related operations on the IoT device side, such as opening the camera, taking photos, previewing, etc.

[0175] It is understandable that, in order to implement the collaborative working method in the embodiment of the present application, the electronic device 100 and the Internet of Things device 200 include hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0176] like Figure 4a The following is a schematic diagram of the interaction between modules. Figure 4a , the embodiment of the present application provides a method process for collaborative operation of a tablet and a desk lamp, specifically including:

[0177] 0. Device service initialization phase

[0178] S0.1: In response to a user operation, the device application service of the desk lamp is started and the device interconnection service is loaded.

[0179] For example, the user operation can be turning on the power of a desk lamp. In response to the user operation, the desk lamp system starts up, the device application service starts up, and the device interconnection service is loaded. The device interconnection service can be used to establish a physical transmission channel between the tablet and the desk lamp to provide data transmission capabilities.

[0180] S0.2, the device interconnection service of the desk lamp loads the hardware abstraction service.

[0181] The device interconnection service can also control the startup of the hardware abstraction service. For example, after the device interconnection service is started, the device interconnection service can load the hardware abstraction service as a plug-in. The hardware abstraction service can be used to establish a logical channel between the tablet and the desk lamp, providing virtualized camera capabilities and an open interface for the desk lamp camera.

[0182] Reference Figure 4bAs shown in the module interaction diagram, the hardware abstraction service may include at least a basic component and a camera component. In the device service initialization phase, the device interconnection service first loads the basic component and initializes the basic component. After the basic component is initialized, information can be exchanged with the device adaptation module of the desk lamp to obtain device information and virtualization capability information. Exemplarily, the device information includes but is not limited to the device name, device identification, device type, etc. Exemplarily, the virtualization capability information includes but is not limited to whether a virtualized camera is supported, whether a virtualized microphone is supported, etc.

[0183] Among them, the desk lamp has the ability to support virtualized cameras, which can be understood as the desk lamp's camera allowing other electronic devices (such as tablets) to call it, that is, it can be understood as the desk lamp's camera allowing it to be used as a virtual camera for other electronic devices.

[0184] After the basic component obtains the lamp's device information and capability information, if the lamp supports virtualized cameras, it loads the camera component to provide virtualized camera capabilities. At this point, the basic component prepares to establish a negotiation channel with the tablet to negotiate network connection information (including but not limited to IP addresses and ports). During this process, the basic component creates a session service (SessionServer) and sends the session name (Session Name) of the session service to the device interconnection service, allowing the tablet's transmission management service to establish a negotiation channel with the lamp's device interconnection service.

[0185] 1. Device discovery phase

[0186] S1.1, in response to a user operation, the tablet's education APP sends a device discovery instruction to the device management service.

[0187] The user operation may be an operation in which the user clicks on a function option in the education APP that requires calling a virtual camera. For example, the user operation may be an operation in which the user clicks on a reading function, a word search function, an assignment function, or a photo taking function in the education APP.

[0188] The tablet's education app receives a user action and, in response, sends a device discovery instruction to the tablet's device management service. The device discovery instruction is used to instruct a search for IoT devices that can establish a connection with the tablet. For example, the device discovery instruction may include, but is not limited to, the instruction type and the type of device to be discovered. In this embodiment, the device discovery instruction is specifically used to search for a desk lamp that can establish a connection with the tablet.

[0189] S1.2, the device management service in the tablet calls the authentication service to authenticate the education app and obtains the authentication result of the education app.

[0190] After receiving the device discovery command, the device management service can obtain the name (or identifier) ​​of the education app based on the existing Android system mechanism and authenticate the education app based on the name. The device management service can then call the authentication service to authenticate the education app and obtain the authentication result.

[0191] In this embodiment, the tablet application framework layer also provides a device management API corresponding to the device management service, as well as a hardware virtualization API corresponding to the hardware virtualization service. To implement the technical solution provided by this embodiment, the education app must register on the relevant platform (such as the platform provided by the tablet manufacturer), adapt to the frameworks of the device management service, hardware virtualization service, and transmission management service, and apply for permissions to the device management API and hardware virtualization API.

[0192] Exemplarily, the authentication service accesses the authentication server to authenticate the education APP through the authentication server, including but not limited to verifying whether it is registered on the relevant platform, whether it is adapted to the relevant framework, and whether relevant API permissions have been applied for.

[0193] As another example, the authentication service can authenticate the education APP based on the local whitelist.

[0194] The authentication service obtains the authentication result (authentication success or authentication failure) of the education app and sends the authentication result to the device management service.

[0195] S1.3, the device management service in the tablet sends a device search instruction to the transmission management service when the education APP is successfully authenticated.

[0196] If the education app successfully authenticates, the device management service sends a device search instruction to the transmission management service. The device search instruction may include, but is not limited to, the instruction type, the device type to be searched, and the search method. Exemplary search methods include, but are not limited to, near-field device scanning and obtaining device information from a cloud server. In this implementation, the device type to be searched is a desk lamp.

[0197] S1.4, the transmission management service in the tablet obtains a list of near- and far-field devices according to the device search instruction, and sends the list of near- and far-field devices to the device management service.

[0198] The far-field and near-field device lists include both the far-field device list and the near-field device list. The far-field devices listed in the far-field device list are registered devices obtained from the cloud server, while the near-field devices listed in the near-field device list are devices scanned via near-field communication. Device information in both the far-field and near-field device lists includes, but is not limited to, the device name, device ID, and device type.

[0199] When the transmission management service receives a device search instruction, it performs relevant device search operations based on the device type to be searched and the search method carried in the device search instruction, such as performing a near-field device scanning operation and obtaining relevant device information in the cloud server, obtaining a far-field device list and a near-field device list, and sending the far-field device list and the near-field device list to the device management service.

[0200] S1.5, the device management service in the tablet filters devices based on the far-field and near-field device lists, and reports the filtered device information to the education app.

[0201] The device management service filters devices based on the far-field device list and the near-field device list, determines the information of desk lamps that can be linked with the tablet, and sends it to the education APP. Among them, the device management service can perform an intersection operation on the far-field device list and the near-field device list, filtering out desk lamps that only exist in the far-field device list or only exist in the near-field device list, and treating desk lamps that exist in both the far-field device list and the near-field device list as desk lamp information that can be linked with the tablet. In this way, the device management service can filter out desk lamps that are not registered in the cloud server, and can also filter out desk lamps that cannot communicate with the tablet in the near field.

[0202] In another optional embodiment, regardless of whether the tablet and the desk lamp are in the same local area network, the tablet's transmission management service can obtain a communication device list and a registered device list based on a device search instruction. The devices included in the communication device list are devices scanned via near-field communication or far-field communication, and the devices included in the registered device list are registered devices obtained from the cloud server. The near-field devices included in the near-field device list are devices scanned via near-field communication. In the communication device list and the registered device list, device information includes, but is not limited to, device name, device ID, device type, etc.

[0203] The transmission management service in the tablet sends the communication device list and the registered device list to the device management service. The device management service filters the devices based on the communication device list and the registered device list, and reports the filtered device information to the education APP. Among them, the device management service can perform an intersection operation on the communication device list and the registered device list, filtering out desk lamps that only exist in the communication device list or only exist in the registered device list, and using desk lamps that exist in both the communication device list and the registered device list as desk lamp information that can be linked with the tablet. In this way, the device management service can filter out desk lamps that are not registered in the cloud server, and can also filter out desk lamps that cannot perform near-field communication with the tablet.

[0204] 2. Virtual camera enabling stage

[0205] S2.1, the tablet's education APP determines the desk lamp to be linked.

[0206] The device management service can filter out one or more desk lamps that can be linked to the tablet. If there is only one desk lamp, the education app will default it to the lamp to be linked. If there are multiple desk lamps, the education app will display a list of lamps to be linked to the user for selection. In response to the user's selection, the education app will select the lamp selected by the user as the lamp to be linked.

[0207] It should be pointed out that the step of the education APP determining the desk lamp to be linked can also be divided into the device discovery stage, which is not limited in this embodiment.

[0208] In step S2.2, the tablet's educational app performs device verification and device connection on the desk lamp, and obtains the ability of the desk lamp to support a virtualized camera.

[0209] S2.3, the tablet's education app sends a virtual camera enablement request to the hardware virtualization service.

[0210] After the tablet's educational app detects that the desk lamp supports virtualized cameras, it sends a virtual camera enablement request to the hardware virtualization service. This request instructs the hardware virtualization service to register a virtual camera with the virtual camera HAL. The request may include, but is not limited to, the request type, device name, device identifier, device type, and virtual camera identifier.

[0211] S2.4, the hardware virtualization service of the tablet registers the virtual camera with the virtual camera HAL.

[0212] After receiving the virtual camera enable request, the hardware virtualization service registers the corresponding virtual camera with the virtual camera HAL according to the virtual camera enable request.

[0213] S2.5, after the virtual camera registration is completed, the tablet's virtual camera HAL sends a virtual camera enabling success indication to the education APP.

[0214] The following combination Figure 4b The module interaction diagram shown in the figure provides a detailed explanation of the process of virtual camera enabling stage. Figure 4b ,The process of the virtual camera enabling phase mainly includes the ,device verification sub-phase (S301-S309), the device connection sub-phase (S310-S321), the ,device service capability request sub-phase (S322-S325) and the virtual camera enabling sub-phase (S326-S331).

[0215] Reference Figure 4b The process of the virtual camera enabling stage includes the following steps:

[0216] S301: The education APP in the tablet sends a virtual camera enable instruction to the hardware virtualization API.

[0217] The virtual camera enabling instruction is used to instruct to enable the virtual camera, and the virtual camera enabling instruction may include but is not limited to an instruction type, a device name, a device identifier, and a device type.

[0218] S302: After receiving the virtual camera enable instruction, the hardware virtualization API in the tablet sends a device verification instruction to the interface scheduling module of the hardware virtualization service.

[0219] The device verification instruction is used to instruct to verify the device information carried in the virtual camera enable instruction. The device verification instruction may include but is not limited to the instruction type, device name, device identifier, and device type.

[0220] S303: The interface scheduling module of the hardware virtualization service in the tablet sends an APP authentication instruction to the authority management module of the hardware virtualization service.

[0221] After receiving the device verification instruction, the hardware virtualization service's interface scheduling module first sends an app authentication instruction to the hardware virtualization service's permission management module to authenticate the app that initiated the virtual camera enable instruction. The app authentication instruction may include, but is not limited to, the name of the app.

[0222] S304: The rights management module of the hardware virtualization service in the tablet performs APP authentication on the education APP.

[0223] For example, the rights management module can access the authentication server to authenticate the education app through the authentication server, including but not limited to verifying whether it is registered on the relevant platform, whether it is adapted to the relevant framework, and whether it has applied for relevant API permissions. The rights management module can access the authentication server through the authentication service, which is not limited in this embodiment.

[0224] S305 , when the rights management module of the hardware virtualization service in the tablet successfully authenticates the education APP, it sends an authentication success indication to the interface scheduling module.

[0225] After the permission management module obtains the authentication result of the education APP, if the education APP authentication is successful, it sends an authentication success indication to the interface scheduling module; if the education APP authentication fails, it sends an authentication failure indication to the hardware virtualization API, so that the hardware virtualization API returns an indication information that the APP has no permission to the education APP based on the authentication failure indication.

[0226] S306 , when the interface scheduling module of the hardware virtualization service in the tablet determines that the education APP authentication is successful, it sends a device verification instruction to the device management module.

[0227] After receiving the successful authentication indication and confirming that the education app has been successfully authenticated, the hardware virtualization service's interface scheduling module sends a device verification instruction to the device management module. The device verification instruction is used to verify the status of the linked device, specifically, in this embodiment, the status of the linked desk lamp. Exemplarily, the device verification instruction may include, but is not limited to, the instruction type, device name, device identifier, and device type.

[0228] S307 , the device management module of the hardware virtualization service in the tablet sends a device information query instruction to the device introduction module of the device management service.

[0229] The device profile module of the device management service stores the information of the currently online devices.

[0230] After receiving the device verification instruction, the device management module of the hardware virtualization service sends a device information query instruction to the device introduction module of the device management service. The device information query instruction may include but is not limited to the device name, device identifier, and device type.

[0231] S308 , the device introduction module of the device management service in the tablet returns the device information to the device management module of the hardware virtualization service.

[0232] If the device profile module of the device management service finds the corresponding device according to the device information query instruction, it returns the device information to the device management module of the hardware virtualization service. The returned device information may include but is not limited to the device name, device identifier, device type and online status.

[0233] If the device profile module of the device management service fails to find the corresponding device according to the device information query instruction, it returns a null value to the device management module of the hardware virtualization service to indicate that the corresponding device was not found. At this time, the device management module of the hardware virtualization service can send a device verification failure indication to the hardware virtualization API, so that the hardware virtualization API returns an indication of device verification failure to the education app based on the device verification failure indication.

[0234] S309 , after receiving the device information, the device management module of the hardware virtualization service in the tablet sends a device verification success indication to the hardware virtualization API.

[0235] The device management module of the hardware virtualization service receives the device information returned by the device introduction module of the device management service, and then sends a device verification success indication to the hardware virtualization API to indicate that the table lamp to be linked has been successfully verified.

[0236] S310: The hardware virtualization API in the tablet sends a device connection request to the device management module of the hardware virtualization service.

[0237] After confirming that the desk lamp to be linked has been successfully verified, the hardware virtualization API sends a device connection request to the device management module of the hardware virtualization service. The device connection request is used to instruct the establishment of a network connection with the device to be linked. In this embodiment, it specifically instructs the establishment of a network connection with the desk lamp to be linked. The device connection request may include, but is not limited to, the request type, device name, device identifier, and device type.

[0238] S311 , after receiving the device connection request, the device management module of the hardware virtualization service performs a negotiation channel establishment preparation operation and sends a negotiation channel opening request to the transmission management service.

[0239] After receiving the device connection request, the device management module of the hardware virtualization service prepares a negotiation channel. This process creates a Session Server and sends the Session Name of the Session Service to the Transmission Management Service. After preparing the negotiation channel, the module sends a negotiation channel open request to the Transmission Management Service. This request indicates the establishment of a negotiation channel and may include, but is not limited to, the peer device identifier (i.e., the lamp identifier) ​​and the Session Name.

[0240] In this embodiment, the negotiation channel opening request is initiated by the tablet side, that is, the tablet needs to establish a connection with the desk lamp. In this case, the desk lamp can be understood as the server, and the tablet can be understood as the client that needs to access the server.

[0241] S312: The transmission management service in the tablet establishes a negotiation channel with the device interconnection service in the desk lamp.

[0242] After receiving the negotiation channel opening request, the transmission management service interacts with the device interconnection service in the desk lamp based on the Session Name to establish a negotiation channel. Establishing the negotiation channel can specifically involve creating a session and determining a session identifier.

[0243] S313, the device interconnection service in the desk lamp sends an indication of successful negotiation channel establishment to the camera component in the hardware abstraction service.

[0244] After the negotiation channel is established, the device interconnection service in the desk lamp sends a successful negotiation channel establishment indication to the camera component in the hardware abstraction service, indicating that the negotiation channel is established and that a device needs to be connected. This successful negotiation channel establishment indication may include, but is not limited to, device information (i.e., tablet device information) and a session identifier.

[0245] S314 , the transmission management service in the tablet sends an indication of successful establishment of a negotiation channel to the device management module of the hardware virtualization service.

[0246] After the negotiation channel is established, the tablet's transport management service sends a successful negotiation channel establishment indication to the device management module of the hardware virtualization service, indicating the successful negotiation channel establishment and the presence of a device requiring a connection. This successful negotiation channel establishment indication may include, but is not limited to, device information requiring a connection (i.e., the lamp device information) and a session identifier.

[0247] This embodiment does not limit the execution order of S313 and S314.

[0248] S315 , the device management module of the hardware virtualization service in the tablet sends a device information negotiation request to the camera component of the hardware abstraction service in the desk lamp based on the negotiation channel.

[0249] The device negotiation request may include but is not limited to device information (such as device name, device identification, device type, etc.) and a control channel connection request.

[0250] S316 , after receiving the device negotiation request, the camera component of the hardware abstraction service in the desk lamp prepares a control channel and returns device negotiation information to the device management module of the hardware virtualization service in the tablet.

[0251] After the camera component of the hardware abstraction service in the desk lamp receives the device negotiation request, it parses the device negotiation request to obtain the device information of the other end and records it. It then determines an IP address and port to be monitored based on the control channel connection request, adds the IP address and port to the device negotiation information, and returns it to the device management module of the hardware virtualization service in the tablet.

[0252] It should be noted that the device negotiation request and the device negotiation information are transmitted based on the established negotiation channel.

[0253] S317 , after receiving the returned device negotiation information, the device management module of the hardware virtualization service in the tablet closes the negotiation channel.

[0254] Closing the negotiation channel may specifically be closing a session. When the device management module of the hardware virtualization service in the tablet receives the returned device negotiation information, the session ends and the device management module of the hardware virtualization service may close the corresponding session according to the session identifier.

[0255] S318 , the device management module of the hardware virtualization service in the tablet sends a control channel opening request to the transmission management service.

[0256] A control channel open request is used to establish a network communication connection with the lamp. The control channel open request may include, but is not limited to, the communication protocol, source IP address, source port number, destination IP address, and destination port number. The destination IP address and destination port number are the IP address and port number monitored by the camera component of the hardware abstraction service in the lamp.

[0257] S319, the transmission management service in the tablet connects to the control channel of the camera component of the hardware abstraction service in the desk lamp, and sends a control channel successful connection indication to the device management module of the hardware virtualization service in the tablet.

[0258] The tablet's transmission management service receives the control channel open request and, based on the information in the control channel open request, establishes a control channel connection with the lamp, effectively establishing a network communication connection between the tablet and the lamp. Furthermore, the tablet's hardware virtualization service's device management module and the lamp's hardware abstraction service's camera component can communicate over this control channel.

[0259] After the control channel is successfully established, the transport management service in the tablet sends a control channel successful connection indication to the device management module of the hardware virtualization service in the tablet. The control channel successful connection indication may include but is not limited to a connection success flag and control channel related information.

[0260] S321 , the device management module of the hardware virtualization service in the tablet sends a device connection success indication to the hardware virtualization API.

[0261] The device connection success indication may include but is not limited to a connection success identifier and connected device information.

[0262] S322: The hardware virtualization API in the tablet sends a device capability request to the device management module of the hardware virtualization service.

[0263] After receiving a successful device connection indication, the hardware virtualization API sends a device capability request to the hardware virtualization service's device management module. This device capability request can be used to request virtualization capability information for the peer device (i.e., the desk lamp). For example, virtualized device capability information includes, but is not limited to, support for virtualized cameras and microphones.

[0264] S323, the device management module of the hardware virtualization service in the tablet sends a device capability request to the camera component of the hardware abstraction service in the desk lamp in the control channel.

[0265] S324 , the camera component of the hardware abstraction service in the desk lamp returns device capability information to the device management module of the hardware virtualization service in the tablet in the control channel.

[0266] In this embodiment, the returned device capability information of the desk lamp may at least include the capability of supporting a virtualized camera and the camera identifier of the desk lamp.

[0267] S325 , the device management module of the hardware virtualization service in the tablet sends the device capability information to the hardware virtualization API.

[0268] The device management module of the hardware virtualization service in the tablet sends the received device capability information to the hardware virtualization API, so that the hardware virtualization API knows whether the desk lamp has the capability to support the virtualized camera.

[0269] S326: The hardware virtualization API in the tablet sends a virtual camera enablement request to the device management module of the hardware virtualization service.

[0270] The hardware virtualization API in the tablet learns that the desk lamp supports virtualized cameras and sends a virtual camera enablement request to the device management module of the hardware virtualization service. The virtual camera enablement request may include, but is not limited to, the request type and the camera ID of the desk lamp.

[0271] S327 , the device management module of the hardware virtualization service in the tablet registers the virtual camera in the virtual camera HAL.

[0272] After receiving the virtual camera enable request, the device management module of the hardware virtualization service sends a virtual camera registration request to the virtual camera HAL. The virtual camera registration request may include, but is not limited to, the request type and the lamp's camera ID. Upon receiving the virtual camera registration request, the virtual camera HAL registers a virtual camera driver for the lamp's camera within the virtual camera HAL, assigns a camera ID (i.e., a virtual camera ID) to the lamp's camera, and registers the camera ID with the system. This establishes a mapping between the lamp's camera and the virtual camera within the virtual camera HAL.

[0273] S328 , the device management module of the hardware virtualization service in the tablet sends a service status update indication to the camera component of the hardware abstraction service in the desk lamp.

[0274] A service status update indication is used to instruct the camera component of the hardware abstract service in the desk lamp to update its virtualized service status. The virtualized service status can include occupied or unoccupied, or alternatively, registered or unregistered. Exemplarily, the service status update indication may include, but is not limited to, device information of the peer device (i.e., the desk lamp), a hardware identifier (such as the desk lamp camera identifier), and the virtualized service status corresponding to the hardware identifier.

[0275] S329, the camera component of the hardware abstract service in the desk lamp updates the service status according to the service status update instruction.

[0276] When the service status update indication indicates that the virtualized service status corresponding to the desk lamp camera is occupied (or registered), the camera component updates the virtualized service status corresponding to its desk lamp camera to occupied (or registered).

[0277] S330: The device management module of the hardware virtualization service in the tablet sends a virtual camera enabling success indication to the hardware virtualization API.

[0278] The virtual camera enabling success indication may include but is not limited to an enabling success identifier (or a virtualization success identifier), the camera identifier of the desk lamp, and the camera ID corresponding to the virtual camera (or the camera ID corresponding to the desk lamp camera).

[0279] This embodiment does not limit the execution order of S328 and S330.

[0280] S331: The hardware virtualization API in the tablet sends a virtual camera enabling success indication to the education app.

[0281] 3. Virtual camera preview access stage

[0282] S3.1, the hardware virtualization API in the tablet sends a virtual camera access instruction to the camera service.

[0283] A virtual camera access instruction refers to an instruction for invoking a virtual camera. The virtual camera access instruction may include, but is not limited to, an instruction type, a virtual camera ID, and camera configuration parameters, including, but not limited to, camera resolution and acquisition frame rate.

[0284] S3.2, the camera service in the tablet sends an image preview request to the virtual camera HAL according to the virtual camera access instruction.

[0285] After receiving the virtual camera access instruction, the camera service generates a corresponding image preview request based on the virtual camera ID and sends it to the virtual camera HAL. The image preview request is used to request a preview of the image data stream. For example, the image preview request may include, but is not limited to, a request identifier, a virtual camera ID, and camera configuration parameters.

[0286] S3.3, the virtual camera HAL in the tablet sends an image preview request to the hardware virtualization service.

[0287] After receiving the image preview request, the virtual camera HAL determines the matching virtualized hardware identifier based on the virtual camera ID carried in the image request. In this embodiment, the virtual camera HAL determines the linked desk lamp camera based on the virtual camera ID and the mapping relationship between the virtual camera ID and the desk lamp camera. Based on the determined virtualized hardware identifier, the virtual camera HAL generates a corresponding image request and sends it to the hardware virtualization service. Exemplarily, the image preview request may include, but is not limited to, a request identifier, device information (i.e., desk lamp information), a virtualized hardware identifier (i.e., desk lamp camera identifier), and camera configuration parameters.

[0288] S3.4, the hardware virtualization service in the tablet sends an image preview request to the transmission management service.

[0289] The hardware virtualization service sends an image preview request to the transmission management service. The image preview request may include, but is not limited to, a request identifier, device information (i.e., lamp information), a virtualized hardware identifier (i.e., lamp camera identifier), and camera configuration parameters.

[0290] When the tablet's hardware virtualization service sends an image preview request to the transmission management service, if it finds that a data channel with the desk lamp is not established, it generates a data channel establishment request and sends it to the transmission management service. This data channel establishment request is used to instruct data transmission between the desk lamp and the tablet. The data channel establishment request may include, but is not limited to, a session identifier, connection information, and data encoding and decoding mode.

[0291] The tablet's transmission management service receives the data channel establishment request and, based on the information in the data channel establishment request, establishes a data channel connection with the lamp. This establishes a data channel between the tablet and the lamp. The tablet's transmission management service and the lamp's device interconnection service can then use this data channel to transmit various data, including but not limited to image data.

[0292] After the data channel is successfully established, the tablet's transport management service sends a successful data channel connection indication to the tablet's hardware virtualization service, and the lamp's device interconnection service sends a successful data channel connection indication to the camera component in the hardware abstraction service. The successful data channel connection indication may include, but is not limited to, a successful connection flag and data channel-related information.

[0293] S3.5: The transmission management service in the tablet transmits an image preview request to the device interconnection service in the desk lamp.

[0294] The transmission management service in the tablet determines the corresponding control channel according to the device information carried in the image preview request, and transmits the image preview request to the device interconnection service of the desk lamp through the control channel.

[0295] S3.6: The device interconnection service in the desk lamp sends an image preview request to the camera driver.

[0296] After receiving the image preview request, the device interconnection service in the desk lamp determines the corresponding hardware driver (in this embodiment, the camera driver) according to the virtualized hardware identifier, and sends the corresponding image preview request to the camera driver.

[0297] In S3.7, the camera driver in the desk lamp drives the camera to capture images and transmits the preview image data to the hardware virtualization service of the tablet through the data channel.

[0298] The camera driver turns on the camera and drives it to capture an image using the camera configuration parameters carried in the image preview request. This generates a preview image data stream and sends it to the device interconnection service via the hardware abstraction service. The device interconnection service then continuously transmits the preview image data stream to the tablet's hardware virtualization service via the data channel. The details of the preview image data stream's packetization, assembly, and encoding / decoding are not detailed here.

[0299] S3.8, the hardware virtualization service in the tablet sends the preview image data to the virtual camera HAL.

[0300] The hardware virtualization service continuously receives the preview image data stream and sends the preview image data stream to the virtual camera HAL.

[0301] S3.9, the virtual camera HAL in the tablet sends the preview image data to the camera service.

[0302] At this time, the virtual camera HAL continues to obtain the preview image data collected by the desk lamp camera and continues to send the preview image data to the camera service.

[0303] S3.10, the camera service in the tablet sends the preview image data to the education app.

[0304] S3.11, the educational app on the tablet displays a preview image.

[0305] After the education app receives the preview image data stream through the camera service, it can display the preview image in the corresponding interface.

[0306] 4. Virtual camera photo taking stage

[0307] S4.1, in response to the received user operation, the education APP in the tablet sends a photo-taking request to the hardware virtualization service.

[0308] For example, a user action might be clicking a photo option. In response to this action, the tablet's education app sends a photo request to the hardware virtualization service. The photo request may include, but is not limited to, the image sequence number, device information (i.e., the lamp), the virtualized hardware identifier (i.e., the lamp camera identifier), and camera configuration parameters. Camera configuration parameters include, but are not limited to, image resolution.

[0309] The photo request may also carry a task identifier to ensure orderly management of multiple photo tasks.

[0310] S4.2, the hardware virtualization service in the tablet sends a photo-taking request to the transmission management service.

[0311] S4.3, the transmission management service in the tablet transmits the photo request to the device interconnection service of the desk lamp.

[0312] The transmission management service in the tablet determines the corresponding control channel according to the device information carried in the image preview request, and transmits the image preview request to the device interconnection service of the desk lamp through the control channel.

[0313] S4.4: The device interconnection service in the desk lamp sends a photo request to the camera driver.

[0314] After receiving the image preview request, the device interconnection service in the desk lamp determines the corresponding hardware driver (in this embodiment, the camera driver) according to the virtualized hardware identifier, and sends the corresponding photo request to the camera driver.

[0315] In S4.5, the camera driver in the desk lamp drives the camera to capture images and transmits the captured image data to the hardware virtualization service of the tablet through a data channel.

[0316] The camera driver drives the camera to capture an image using the camera configuration parameters carried in the photo request, obtains the captured image data, and sends the captured image data to the device interconnection service through the hardware abstraction service. The device interconnection service then continuously transmits the captured image data to the tablet's hardware virtualization service through the data channel. The image data packetization, grouping, encoding, and decoding processes are not detailed here.

[0317] S4.6, the hardware virtualization service in the tablet sends the captured image data to the education app.

[0318] S4.7, the tablet's educational app displays the captured image.

[0319] After the educational APP receives the captured image through the hardware virtualization service, it can display the captured image in the corresponding interface.

[0320] In this embodiment, the virtual camera preview access phase is implemented based on Android's native camera framework, while the virtual camera capture phase is implemented based on a proprietary virtualized camera framework. This shortens the processing path involved in the virtual camera capture phase and reduces capture latency. Furthermore, because image preview is still implemented based on Android's native camera framework, educational apps require fewer modifications to adapt to the technical solutions provided by this embodiment.

[0321] It should be noted that the stage divisions in the above process are merely illustrative and are not intended to limit this embodiment of the present application. Furthermore, after the virtual camera preview access stage is executed and the preview image is displayed on the tablet, the real-time preview image display process and the virtual camera photo capture process can be executed simultaneously. Any details not fully explained in the above process can be referred to existing technologies and will not be elaborated upon here.

[0322] Figure 4a The communication architecture of the collaborative work system is given in the paper. This communication architecture is used to complete the tablet's management of the virtual camera (i.e., the desk lamp camera), the control command interaction between the tablet and the desk lamp, and the return and processing of image data.

[0323] It's important to note that instructions and requests transmitted across devices (i.e., between the tablet and the desk lamp) require encapsulation based on communication protocols and parameter sequences, which are not detailed in this embodiment. The tablet's hardware virtualization service also manages the lifecycle of preview image streams and captured images by dynamically allocating and destroying memory.

[0324] In addition, it should be pointed out that before executing the collaborative working method provided in this embodiment, the education APP needs to be bound to the desk lamp and the desk lamp needs to be registered in the cloud server.

[0325] An embodiment of the present application provides a framework solution for an Android system device to use the camera of an external device to take photos. This solution can not only be applied in educational scenarios, but also is applicable to other devices equipped with cameras. These devices can share their camera capabilities with Android system devices such as mobile phones and tablets to achieve interconnection and interoperability between Android system devices and these devices.

[0326] Figure 5a-5b An example application scenario is shown. Figure 5a As shown in (1), the tablet displays an interface 401, which displays multiple application icons. The user clicks on the education application icon 4011. In response to the received user operation, the tablet opens the education application and displays the education application interface. Figure 5a As shown in (2). Figure 5a As shown in (2), the tablet displays an education application interface 402. Various function options of the education application are displayed on the interface 402, including but not limited to a word lookup function, a point reading function, an assignment function, a photo taking function, etc. In order to use the photo taking function, the user clicks on the photo taking function option 4021. In response to the user operation, the tablet executes the process of the device discovery phase, the virtual camera service enabling phase, and the virtual camera preview access phase.

[0327] Among them, in the device discovery phase, if the number of desk lamps that can be linked with the tablet is filtered by the tablet's device management service is one, the tablet automatically executes the virtual camera service enabling phase and the virtual camera preview access phase process, and displays, for example Figure 5b The interface shown in (1). In the device discovery phase, if the number of desk lamps that can be linked to the tablet is filtered by the tablet's device management service is multiple, the tablet displays a desk lamp selection interface. Exemplarily, a list of desk lamps to be linked is displayed on the desk lamp selection interface, and the user can perform a selection operation. In response to the user's selection operation, the educational application determines a desk lamp to be linked, and continues to execute the virtual camera service enabling phase and the virtual camera preview access phase process to display, for example Figure 5b The interface shown in (1).

[0328] like Figure 5b As shown in (1), the interface 403 displays an image preview window 4031 and a photo taking option. The image preview window 4031 displays a preview image captured by the lamp camera in real time. At this time, if the user clicks the photo taking option, in response to the user operation, the tablet executes the process of the virtual camera photo taking stage, and displays, for example Figure 5b The interface shown in (2) is shown in the figure. Figure 5b(2) The interface 404 shown displays the image captured by the lamp camera in the image preview window. At this time, if the user clicks the confirmation option 4041, the tablet responds to the user operation, saves the captured image, and continues to display, for example Figure 5b The preview interface shown in (1) is shown in the figure. If the user clicks the cancel option, the tablet responds to the user operation and can display, for example, Figure 5b The preview interface shown in (1) is shown in the figure.

[0329] It should be pointed out that if Figure 5b (2) The interface shown is merely an example. The image captured by the desk lamp camera may not be displayed in the image preview window 4041, but may be displayed in other areas of the interface. The image preview window 4041 continues to display the preview image captured in real time by the desk lamp camera. This application does not impose any restrictions on this.

[0330] The collaborative working method provided in this embodiment mainly describes a low-cost technical solution for implementing online education functions based on a combination of a tablet device and a desk lamp device. The following describes the technical solution provided in this embodiment in combination with several different functions involved in online education.

[0331] Scene 1

[0332] Reference Figure 1a The following application scenario illustrates the technical solution using the word lookup function as an example. When students encounter an unfamiliar word, they can point their finger at the word's location. The lamp's camera then captures the image, which the tablet then uses to identify the word. After the online word lookup is complete, the tablet provides feedback on the meaning of the word, such as displaying and announcing the word's meaning on the screen.

[0333] like Figure 6a The following is a schematic diagram of the interaction between modules. Figure 6a , the embodiment of the present application provides a method process for collaborative operation of a tablet and a desk lamp, specifically including:

[0334] S501, in response to the user clicking on the word search function, the tablet and the desk lamp execute the processes of the device discovery phase, the virtual camera enabling phase and the virtual camera preview access phase, and the tablet displays the preview interface.

[0335] Among them, the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0336] It's important to note that during the virtual camera preview access phase, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service, along with camera configuration parameters. These parameters may include, but are not limited to, image resolution and frame rate. The desk lamp camera configures itself according to the received configuration parameters and captures preview image data at the corresponding resolution and frame rate.

[0337] In this scenario, the tablet needs to accurately recognize the preview image to determine the text content pointed to by the user. Therefore, this scenario requires high image quality for the preview image. For example, the image resolution can be set to 1080P. This ensures a high success rate for word lookup.

[0338] S502: The education APP in the tablet performs finger recognition on the preview image.

[0339] For example, the educational APP may perform finger recognition on each preview image received, or may periodically perform finger recognition on the latest preview image received, which is not limited in this embodiment.

[0340] For example, an image recognition algorithm may be integrated into an education APP to implement image recognition operations, and the education APP may also call an image recognition service to perform image recognition operations, which is not limited in this embodiment.

[0341] Regarding the image recognition algorithm, reference may be made to existing technologies, and this embodiment will not be described in detail here.

[0342] S503 , in response to the user pointing to a word with a finger, the education APP in the tablet recognizes the user's finger in the preview image.

[0343] When a user points his finger at a word in a book, the desk lamp camera can capture a preview image of the word pointed at by the finger, and then the education app in the tablet can recognize the user's finger in the preview image.

[0344] Among them, when the educational APP performs finger recognition on the preview image, if a finger is recognized, the position information of the finger in the preview image, such as coordinate information, can be obtained.

[0345] S504 , the education APP in the tablet determines a ROI (region of interest) image according to the position of the finger in the preview image.

[0346] After the educational app recognizes the user's finger in the preview image, it can determine the ROI image based on the finger's position in the preview image. Specifically, the educational app can determine the ROI information based on the finger's coordinates in the preview image. The ROI information includes, but is not limited to, the center coordinates and the region (such as width and height). Furthermore, the educational app can crop the ROI image from the preview image based on the ROI information.

[0347] S505: The education APP in the tablet accurately recognizes the ROI image and determines the new words to be explained.

[0348] For example, an image recognition algorithm may be integrated into the education APP to accurately identify the ROI image. The education APP may also call an image recognition service to accurately identify the ROI image to determine the new words to be explained. This embodiment does not limit this.

[0349] Regarding the image recognition algorithm, reference may be made to existing technologies, and this embodiment will not be described in detail here.

[0350] S506, the education APP in the tablet searches for the new word to be explained and displays the meaning of the new word to be explained.

[0351] After determining the new word to be explained, the education app can perform an online word search or a word search in a database to obtain the definition of the new word to be explained. Furthermore, the education app can display the definition of the new word to be explained for the user to review. Furthermore, the education app can also read the displayed definition of the new word aloud, which is not limited in this embodiment.

[0352] Similarly, users can also use a pointing tool (or indicator tool) such as a reading pen to perform a pointing operation, which is not limited in this embodiment. Accordingly, the education app performs a pointing tool recognition on the preview image to determine whether the user intends to look up a word, and determines the ROI image based on the position information of the pointing tool in the preview image.

[0353] Similarly, users can use a finger or a pointing tool like a reading pen to point to an image in a book. Accordingly, the education app determines the ROI image based on the position of the finger or reading pen, performs image content recognition on the ROI image, displays the corresponding interpretation of the image, and can also read the displayed interpretation aloud. This scenario is not further described in this embodiment.

[0354] Figure 1a 、 Figure 7a-7b An example application scenario is shown. Figure 7aAs shown in (1), the tablet displays an education APP interface 701, which displays various function options of the education application, including but not limited to a word search function, a point reading function, a homework function, a photo taking function, etc. The user clicks on the word search function option 7011. In response to the user operation, the tablet executes the process of the device discovery phase, the virtual camera service enabling phase, and the virtual camera preview access phase.

[0355] Among them, in the device discovery phase, if the number of desk lamps that can be linked with the tablet is filtered by the tablet's device management service is one, the tablet automatically executes the virtual camera service enabling phase and the virtual camera preview access phase process, and displays, for example Figure 7a The interface shown in (2). In the device discovery phase, if the number of desk lamps that can be linked to the tablet is filtered by the tablet's device management service is multiple, the tablet displays a desk lamp selection interface. Exemplarily, a list of desk lamps to be linked is displayed on the desk lamp selection interface, and the user can perform a selection operation. In response to the user's selection operation, the educational application determines a desk lamp to be linked, and continues to execute the virtual camera service enabling phase and the virtual camera preview access phase process to display, for example Figure 7a The interface shown in (2) in the figure.

[0356] like Figure 7a As shown in (2), the interface 702 displays an image preview window 7021 and a word search function operation diagram 7022. The image preview window 7021 displays a preview image captured in real time by the lamp camera. The user can refer to the word search function operation diagram 7022 to perform a word pointing or image pointing operation to trigger the word search function. The education APP performs word pointing or image pointing recognition based on the preview image. Figure 7a In (2), when the user points to a word in the book, the desk lamp camera captures a preview image of the finger pointing to the word, and the preview image of the finger pointing to the word is displayed in the image preview window 7021. Then, the education APP can recognize the user's finger in the preview image and determine the position information of the finger in the preview image, such as coordinate information. The education APP determines the ROI image based on the position of the finger in the preview image, and accurately recognizes the ROI image to determine the new word to be explained. After the education APP finds the meaning of the new word, it displays the corresponding new word meaning on the interface, which can be referred to Figure 7b shown.

[0357] However, in order to ensure the success rate of word lookup in the above process, the desk lamp camera needs to continuously return a high-resolution (e.g., 1080p) preview image stream, which places high bandwidth requirements on 4-8 Mbps. This also increases the hardware chip requirements of the desk lamp, which in turn increases its cost.

[0358] To enable collaborative word search functionality between a tablet and a desk lamp while reducing the hardware cost of the desk lamp, this embodiment also provides a technical solution. Because finger recognition, word pointing tool recognition, and position recognition in images do not require high image resolution, during the virtual camera preview access phase, the desk lamp's camera continuously returns a low-resolution (e.g., 480p) preview image stream. If the educational app recognizes a user's word or image pointing operation based on the preview image, the desk lamp is triggered to capture a high-resolution (e.g., 1080p) image, allowing the educational app to accurately identify the new word or image to be explained based on this high-resolution image. This low-resolution preview image stream, continuously returned by the desk lamp's camera, requires minimal bandwidth, requiring only 0.5-1 Mbps. Additional bandwidth is only required to transmit the high-resolution image. Consequently, this technical solution not only reduces the hardware chip requirements and costs of the desk lamp, but also ensures a high word search success rate.

[0359] like Figure 6b The following is a schematic diagram of the interaction between modules. Figure 6b , the embodiment of the present application provides a method process for collaborative operation of a tablet and a desk lamp, specifically including:

[0360] S601, in response to the user clicking on the word search function, the tablet and the desk lamp execute the processes of the device discovery phase, the virtual camera enabling phase and the virtual camera preview access phase, and the tablet displays the preview interface.

[0361] Among them, the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0362] It's important to note that during the virtual camera preview access phase, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service, along with camera configuration parameters. These parameters may include, but are not limited to, image resolution and frame rate. The desk lamp camera configures itself according to the received configuration parameters and captures preview image data at the corresponding resolution and frame rate.

[0363] In this scenario, to reduce the bandwidth consumed by the preview image stream, the desk lamp camera can capture preview images at a low resolution during the virtual camera preview access phase. For example, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service with first configuration parameters, including a first image resolution (e.g., 480p). The desk lamp camera then configures itself according to the received configuration parameters and captures preview image data at the first image resolution and the corresponding image capture frame rate.

[0364] S602: The education APP in the tablet performs finger recognition on the preview image.

[0365] S603 , in response to the user pointing to a word with a finger, the education APP in the tablet recognizes the user's finger in the preview image.

[0366] S604: The education APP in the tablet determines ROI information according to the position of the finger in the preview image, and generates a photo request according to the ROI information.

[0367] The ROI information refers to information used to determine the ROI, which may include but is not limited to center point coordinates and region range (such as width and height information).

[0368] S605: The tablet transmits the photo-taking request to the desk lamp.

[0369] The photo request may include, but is not limited to, the virtual camera ID corresponding to the desk lamp camera, second configuration parameters of the desk lamp camera, and ROI information. The second configuration parameters may include, but are not limited to, a second image resolution, which is higher than the first image resolution, such as setting the second image resolution to 1080P. This allows the educational app to accurately identify the new word or image to be explained based on this high-resolution image.

[0370] S606 , the camera of the desk lamp is configured according to the second configuration parameters carried in the photo request, captures an image according to the second image resolution, and sends the captured image to the hardware abstraction service.

[0371] S607: The hardware abstraction service determines the ROI image according to the ROI information.

[0372] The hardware abstraction service can crop the ROI image from the captured image based on the ROI information.

[0373] In an optional embodiment, the photo request includes, but is not limited to, the second configuration parameters, but does not include ROI information. The tablet then transmits the photo request to the desk lamp. The lamp's camera is configured according to the second configuration parameters in the photo request, captures an image at the second image resolution, and returns the captured image to the education app on the tablet. The education app can then determine the ROI image based on the ROI information, for example, by cropping the ROI image from the captured image based on the ROI information.

[0374] S608, the desk lamp transmits the ROI image to the education APP in the tablet.

[0375] Compared with the desk lamp directly returning the high-resolution captured image to the education APP in the tablet, returning the cropped ROI image to the education APP in the tablet can also reduce the amount of data transmission and reduce bandwidth usage.

[0376] S609: The education APP in the tablet accurately recognizes the ROI image and determines the new words to be explained.

[0377] S610, the education APP on the tablet performs a word lookup operation on the new word to be explained, and displays the definition of the new word to be explained.

[0378] For details not explained in detail in this process, please refer to the previous text and will not be repeated here.

[0379] Similarly, users can also use pointing tools such as a reading pen to perform word pointing operations, which is not limited in this embodiment. Accordingly, the education app recognizes the pointing tool on the preview image to determine whether the user intends to look up a word, and determines the ROI information based on the position information of the pointing tool in the preview image.

[0380] Similarly, users can use a finger or a pointing tool like a reading pen to point to an image in a book. Accordingly, the education app determines the ROI information based on the position of the finger or reading pen, and then uses the ROI information to determine the ROI image in the captured image. It then performs image content recognition on the ROI, displays the corresponding interpretation of the image, and can also read the displayed image interpretation aloud. This scenario is not further described in this embodiment.

[0381] For the application scenarios of this process, please refer to Figure 1a 、 Figure 7a-7b The application scenario shown. Figure 7a In (2), when the user points to a word in the book, the desk lamp camera captures a preview image of the finger pointing to the word, and the preview image of the finger pointing to the word is displayed in the image preview window 7021. Then, the education APP can recognize the user's finger in the preview image, determine the ROI information, and generate a photo request based on the ROI information and the image high resolution, triggering the desk lamp camera to capture the image. The desk lamp camera captures the image based on the image high resolution, and the desk lamp side crops the high resolution captured image based on the ROI information to obtain the ROI image, and returns the ROI image to the education APP in the tablet. The education APP accurately recognizes the ROI image and determines the new word to be explained. After the education APP finds the meaning of the new word, the corresponding new word meaning is displayed on the interface, which can be referred to Figure 7b shown.

[0382] Scene 2

[0383] Reference Figure 1a The following application scenario illustrates the technical solution using the homework function as an example. When students need to submit their homework online, they can click "Photo" in the education app, use the desk lamp camera to capture an image of the homework, and then upload the image to the database through the education app.

[0384] like Figure 8 The following is a schematic diagram of the interaction between modules. Figure 8 , the embodiment of the present application provides a method process for collaborative operation of a tablet and a desk lamp, specifically including:

[0385] S801, in response to the user clicking on the homework function, the education APP in the tablet displays a homework submission list.

[0386] It should be noted that the homework submission list refers to a list of multiple homework submission options. One homework submission option corresponds to one homework. Figure 9a The middle interface is shown as 704. The homework options may be divided by subject or by time, which is not limited in this embodiment.

[0387] If the education app's homework function only requires submitting homework images for one course, the app won't display the homework submission list. In this case, in response to the user clicking the homework function, the tablet and desk lamp execute the device discovery phase, virtual camera enablement phase, and virtual camera preview access phase, with the tablet displaying the preview interface.

[0388] S802, in response to the user clicking the submit job option, the tablet and the desk lamp execute the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase, and the tablet displays the preview interface.

[0389] Among them, the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0390] It's important to note that during the virtual camera preview access phase, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service, along with camera configuration parameters. These parameters may include, but are not limited to, image resolution and frame rate. The desk lamp camera configures itself according to the received configuration parameters and captures preview image data at the corresponding resolution and frame rate.

[0391] In this scenario, to reduce the bandwidth consumed by the preview image stream, the desk lamp camera can capture preview images at a low resolution during the virtual camera preview access phase. For example, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service with first configuration parameters, including a first image resolution (e.g., 480p). The desk lamp camera then configures itself according to the received configuration parameters and captures preview image data at the first image resolution and the corresponding image capture frame rate.

[0392] S803: In response to the user clicking the photo option, the education APP in the tablet generates a photo request.

[0393] When the user places homework or books within the collection area of ​​the desk lamp camera, the user can click the photo option to trigger the desk lamp camera to capture the homework image.

[0394] The photo request may include, but is not limited to, the virtual camera ID corresponding to the desk lamp camera and second configuration parameters of the desk lamp camera. The second configuration parameters may include, but are not limited to, a second image resolution, which is higher than the first image resolution, such as setting the second image resolution to 1080P. This allows the education app to upload high-resolution images of homework.

[0395] S804: The tablet transmits the photo-taking request to the desk lamp.

[0396] S805: The camera of the desk lamp is configured according to the second configuration parameters carried in the photo taking request, and takes an image according to the second image resolution.

[0397] S806, the desk lamp transmits the captured image to the education APP on the tablet.

[0398] S807, the educational APP on the tablet displays the captured image.

[0399] The tablet's education app receives the image of the homework captured by the desk lamp's camera and displays it. If the user is satisfied with the captured image, they can click Submit to upload it to the database. If they are not satisfied with the captured image, they can click Take Photo again to trigger the desk lamp's camera to retake the image.

[0400] S808 , in response to the user clicking the submit option, the education APP in the tablet uploads the captured image to the database.

[0401] For details not explained in detail in this process, please refer to the previous text and will not be repeated here.

[0402] It should be noted that the homework images mentioned above are merely illustrative examples. Users can click the photo option to trigger the desk lamp's camera to capture other images. After the desk lamp sends the captured images to the education app on the tablet, the education app can then upload the received images to the corresponding database.

[0403] Figure 1a 、 9a - Figure 9c An example application scenario is shown. Figure 9aAs shown in (1), the tablet displays an education APP interface 701, and various function options of the education application are displayed on the interface 701, including but not limited to a word search function, a point reading function, a homework function, a photo taking function, etc. The user clicks on the word search function option 7012, and in response to the user operation, the education APP in the tablet displays a homework submission list, which can be referred to Figure 9a As shown in (2). Figure 9a In the job submission list interface 704 shown in (2), multiple job submission options are displayed (such as Submit Job 1, Submit Job 2, Submit Job 3, Submit Job 4, etc.), and different job submission options correspond to different jobs. For example, if the user needs to upload a job image for Submit Job 4, the user clicks Submit Job 4 option 7042. In response to the user operation, the tablet can display the following Figure 9b The job submission interface 705 shown in (1) is shown in FIG.

[0404] Continuing to refer to (1) in 9b, in the homework submission interface 705, an image preview window 7041, a photo option 7051 and a submission option 7052 are displayed. Among them, the image preview window 7041 displays the preview image captured in real time by the desk lamp camera. When the user places homework or books, etc. in the acquisition area of ​​the desk lamp camera, the user can click the photo option 7051 to trigger the desk lamp camera to capture the homework image. In response to the user operation, the education APP generates a photo request and sends it to the desk lamp side to call the desk lamp camera to capture the homework image. Among them, the desk lamp camera captures the homework image with high resolution according to the image carried in the photo request, and returns the captured homework image to the education APP in the tablet to be displayed through the education APP. Please refer to the following. Figure 9b Interface 706 shown in (2).

[0405] Continuing to refer to (2) in 9b, the interface 706 displays an image preview window 7041, a photo option 7051, a submit option 7052, and a job image 7061 captured by the desk lamp camera. Among them, a close option 7062 is also displayed on the job image 7061. If the user is not satisfied with the job image 7061, the user can click the close option 7062, and the job image 7061 will no longer be displayed on the interface. At this time, the user can click the photo option 7051 to trigger the desk lamp camera to re-record the job image. If the user is satisfied with the job image 7061, the user can click the submit option 7052. In response to the user operation, the tablet can display the following Figure 9cThe pending confirmation interface 707 is shown. In the pending confirmation interface 707, a homework image 7061 to be submitted and a confirmation window 7071 are displayed. If the user clicks the confirmation option 7072 in the confirmation submission window 7071, the education APP responds to the user operation and uploads the homework image 7061 to the database. If the user clicks the cancel option 7073 in the confirmation submission window 7071, the education APP responds to the user operation and may display the following information: Figure 9b The interface shown in (1) is waiting for the user to click the photo option 7051 to trigger the desk lamp camera to retake the work image.

[0406] It should be noted that, referring to (2) in 9b, if the user is satisfied with the homework image 7061, he can click the submit option 7052. In response to the user operation, the education APP may no longer display the Figure 9c The interface shown directly uploads the job image 7061 to the database. This embodiment does not limit this.

[0407] Scenario 3

[0408] Reference Figure 1a The following application scenario illustrates the technical solution using the touch-to-read feature (also known as finger-reading) as an example. When a student wants to read aloud content from a book through an educational app, the desk lamp camera can capture the book image in real time. The educational app can then load the corresponding book content based on the image and determine the desired content based on the student's finger position or page turning.

[0409] like Figure 10 The following is a schematic diagram of the interaction between modules. Figure 10 , the embodiment of the present application provides a method process for collaborative operation of a tablet and a desk lamp, specifically including:

[0410] S901, in response to the user clicking the point-to-read function, the tablet and the desk lamp execute the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase, and the tablet displays the preview interface.

[0411] Among them, the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0412] It's important to note that during the virtual camera preview access phase, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service, along with camera configuration parameters. These parameters may include, but are not limited to, image resolution and frame rate. The desk lamp camera configures itself according to the received configuration parameters and captures preview image data at the corresponding resolution and frame rate.

[0413] In this scenario, to reduce the bandwidth consumed by the preview image stream, the desk lamp camera can capture preview images at a low resolution during the virtual camera preview access phase. For example, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service with first configuration parameters, including a first image resolution (e.g., 480p). The desk lamp camera then configures itself according to the received configuration parameters and captures preview image data at the first image resolution and the corresponding image capture frame rate.

[0414] S902: The education APP in the tablet recognizes the preview image and determines the title of the book.

[0415] For example, the education APP may perform book information recognition for each preview image received, or may periodically perform book information recognition for the latest preview image received, which is not limited in this embodiment.

[0416] For example, an image recognition algorithm may be integrated into an education APP to implement image recognition operations, and the education APP may also call an image recognition service to perform image recognition operations, which is not limited in this embodiment.

[0417] Regarding the image recognition algorithm, reference may be made to existing technologies, and this embodiment will not be described in detail here.

[0418] S903, the education APP in the tablet searches the database according to the book title and loads the book content corresponding to the book title.

[0419] If the tablet education app searches the database based on the book name and determines different versions of the book, it can display the corresponding book list for the user to choose. Then, in response to the user's selection of a certain version of the book, the education app loads the content corresponding to that version of the book.

[0420] S904 , in response to the user turning pages or finger-pointing reading operation, the education APP in the tablet recognizes the preview image, determines the paragraph to be read aloud, and reads the corresponding paragraph aloud.

[0421] After loading the book content, the educational app can identify the pointing tool of the finger or reading pen command for each preview image received, or can periodically identify the pointing tool of the finger or reading pen command for the most recently received preview image, which is not limited in this embodiment. In response to the user's point reading operation, the educational app can identify the book page number and the user's click location information, such as coordinate information, based on the preview image. Then, based on the book page number and the user's click location information, it can determine the paragraph to be read aloud in the loaded book content and read the corresponding paragraph aloud.

[0422] After loading the book content, the education app can also perform page turning recognition based on the preview image stream. In response to the user's page turning operation, the education app can determine the paragraph to be read aloud in the loaded book content based on the recognized book page number and read the corresponding paragraph aloud.

[0423] For details not explained in detail in this process, please refer to the previous text and will not be repeated here.

[0424] Figure 1a 、 Figure 11a-Figure 11b An example application scenario is shown. Figure 11a As shown in (1), the tablet displays an education APP interface 701, which displays various function options of the education application, including but not limited to a word lookup function, a point reading function, a homework function, a photo taking function, etc. The user clicks on the point reading function option 7013. In response to the user operation, the tablet executes the process of the device discovery phase, the virtual camera service enabling phase, and the virtual camera preview access phase.

[0425] Among them, in the device discovery phase, if the number of desk lamps that can be linked with the tablet is filtered by the tablet's device management service is one, the tablet automatically executes the virtual camera service enabling phase and the virtual camera preview access phase process, and displays, for example Figure 11a The interface shown in (2). In the device discovery phase, if the number of desk lamps that can be linked to the tablet is filtered by the tablet's device management service is multiple, the tablet displays a desk lamp selection interface. Exemplarily, a list of desk lamps to be linked is displayed on the desk lamp selection interface, and the user can perform a selection operation. In response to the user's selection operation, the educational application determines a desk lamp to be linked, and continues to execute the virtual camera service enabling phase and the virtual camera preview access phase process to display, for example Figure 11a The interface shown in (2) in the figure.

[0426] like Figure 11a As shown in (2), an image preview window 7081 is displayed in the interface 708. The image preview window 7081 displays the preview image captured by the desk lamp camera in real time. The education APP recognizes the preview image to determine the book name. After recognizing the book name, the education APP searches the database according to the book name. If the corresponding book is retrieved, the book content loading operation is executed, which can be referred to Figure 11b As shown in (1).

[0427] Continue to refer to Figure 11bAs shown in (1), the interface 709 displays an image preview window 7081, the recognized book name 7091, and a book content loading progress indicator 7092. After the book content is loaded, the education APP can recognize the user's point-to-read operation or page-turning operation based on the preview image. Take the user performing the page-turning operation as an example, refer to Figure 11b In the interface 710 shown in (2), the image preview window 7081 can display the user's page turning action. In response to the user's operation, the education app recognizes the book page number based on the preview image. Furthermore, the education app can determine the paragraph to be read aloud in the loaded book content based on the recognized book page number and read the corresponding paragraph aloud.

[0428] In the collaborative work method provided in the embodiment of this application, a professional online education experience can be combined using a home tablet and a desk lamp with a camera. The desk lamp's camera is used in conjunction with the tablet's education app to complete scenarios that require photography, such as student finger lookup, homework submission, and book reading.

[0429] When the electronic device 100 (using a tablet as an example) and the IoT device 200 (using a desk lamp as an example) based on the aforementioned hardware and software structures are used in conjunction with each other in the learning scenarios described above, the data transmitted from the desk lamp to the tablet will vary depending on the application. For example, in the point-and-read scenario described above, the desk lamp transmits preview stream data captured by the camera to the tablet, that is, a continuously transmitted video data stream; in the word-checking and homework submission scenarios described above, the desk lamp transmits image data captured by the camera to the tablet.

[0430] Specifically in practical applications, when the desk lamp transmits preview stream data and image data to the tablet, it can be based on the same data packaging principle, with a set size as the unit, to encapsulate the currently generated preview stream data into a preview stream data packet, and encapsulate a part of the entire image data into an image data packet, and then transmit it to the tablet side based on the Real-time Transport Protocol (RTP). For preview stream data, this method does not involve the packaging process, so the preview stream data packet can be directly processed and displayed, that is, there is no distinction between the first packet, the last packet, and the middle packet. For image data, the final displayed photo must be determined according to the positional relationship between the image data packets, and the packaging operation is triggered according to the last packet. However, in the current image data transmission method, the data body of the image data packet must be parsed to determine whether the image data packet is the first packet.

[0431] In addition, even if the data body of the image data packet is parsed, it cannot be determined whether the image data packet is the tail packet, that is, the packet grouping cannot be triggered quickly and accurately.

[0432] In view of this, the present application provides an image data transmission method, which aims to quickly and accurately determine whether the image data packet is the first packet, the middle packet, or the last packet without parsing the data body of the received image data packet, so that when the received image data packet is identified as the last packet, the data packetization operation can be performed.

[0433] See also Figure 12 , which illustrates the specific process of image data transmission between a tablet and a desk lamp:

[0434] S1001: The tablet generates a photo request (including parameter information that the photo needs to meet) in response to a user's operation.

[0435] It is understandable that from the above description, the transmission of image data is only involved in the word search scenario and the homework submission scenario. Therefore, the tablet responds to the user's operation behavior, specifically when the user triggers the operation of the word search function or the homework submission function.

[0436] For example, in some implementations, the operation performed by the user is, for example, clicking on a control corresponding to a point-to-read function / scene displayed in the current interface of the tablet, or clicking on a control corresponding to a submitting job function / scene.

[0437] For example, in other implementations, the user performs an operation such as controlling the tablet to respond through voice commands, thereby selecting a word lookup function or submitting an assignment function.

[0438] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0439] Furthermore, as described above, the image data resolution requirements for word lookup and assignment submission are higher than those for preview stream data in point reading scenarios. Therefore, when generating a photo request in response to a user's action, it must at least include the parameters that the captured photo must meet, such as resolution.

[0440] S1002: The tablet sends a photo-taking request to the desk lamp.

[0441] After generating the photo request required above, the tablet can send the photo request to the desk lamp through a control channel negotiated with the desk lamp, for example.

[0442] S1003, the desk lamp detects whether the camera is abnormal.

[0443] In actual applications, the camera of the desk lamp may not be able to take pictures due to certain factors. Therefore, in order to minimize unnecessary processing, such as parsing of photo requests and judgment of parameter information, it is possible to first detect whether the camera of the desk lamp is abnormal in the current scene, that is, whether it can be used normally.

[0444] Correspondingly, if the camera is abnormal, the desk lamp can generate an abnormal feedback package. For the sake of convenience, the abnormal feedback package fed back when the camera is abnormal is called abnormal feedback package 1, and the abnormal feedback package fed back when the parameter is abnormal is called abnormal feedback package 2, and step S1004 is executed.

[0445] On the contrary, when the camera is normally available, the parameter information carried in the photo request is extracted, and step S1006 is executed.

[0446] S1004: The desk lamp sends an abnormality feedback packet 1 to the tablet.

[0447] Exemplarily, in some implementations, the abnormality feedback package 1 may be transmitted to the tablet via an image data channel negotiated with the tablet.

[0448] S1005: Respond according to the abnormal cause in abnormal feedback package 1.

[0449] For example, in some implementations, to facilitate the tablet's understanding of the cause of the exception in Exception Feedback Packet 1, an error code identifying the exception may be included in the data header of Exception Feedback Packet 1. This allows the tablet to quickly determine the cause of the exception based on the data header without having to parse the data itself, and to respond accordingly. For example, the tablet may display a prompt in the user interface to inform the user that a desk lamp's camera has experienced an abnormality, allowing the user to check the lamp's camera according to the prompt.

[0450] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0451] S1006: The desk lamp extracts parameter information from the photo request.

[0452] Specifically, when the camera of the desk lamp is normally available, the desk lamp can extract parameter information in the photo request sent by the tablet, such as resolution information of the photo to be taken.

[0453] S1007, the desk lamp detects whether parameter information is abnormal.

[0454] Specifically, in this embodiment, the detection of parameter information by the desk lamp is, for example, to determine whether the resolution information is supported by the camera.

[0455] Accordingly, if it is not supported, it is determined that the parameter is abnormal and step S1008 needs to be executed; if it is supported, it is determined that the parameter information is normal and step S1010 can be executed.

[0456] S1008: The desk lamp sends an exception feedback packet 2 to the tablet.

[0457] Similarly, the abnormal feedback package 2 can be transmitted to the tablet through the image data channel negotiated with the tablet.

[0458] In addition, it should be noted that in actual applications, the desk lamp side can also generate other abnormal feedback packets based on other abnormal reasons, such as network transmission abnormalities, and feed them back to the tablet side. The above are only two specific abnormal scenarios and are not intended to be a specific limitation of this embodiment.

[0459] S1009: Respond according to the abnormal cause in abnormal feedback package 2.

[0460] For example, the error code carried in Exception Feedback Packet 2 can also be stored in the data header of Exception Feedback Packet 2. This allows the tablet to quickly determine the cause of the exception based on the data header without having to parse the data body, and then respond accordingly. For example, the tablet can regenerate a photo request based on the resolution supported by the desk lamp and send the regenerated photo request to the desk lamp.

[0461] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0462] S1010: The desk lamp adjusts the shooting parameters of the camera according to the parameter information, and uses the adjusted camera to shoot to obtain an image data packet.

[0463] It should be noted that the image data packet in this embodiment specifically includes image data (data body) of a whole photo that needs to be displayed on the tablet side, and a data header allocated to the image data.

[0464] In order to better understand the generation process of the above-mentioned image data packet, the following Figure 13 Provide explanation.

[0465] See also Figure 13 For example, using the adjusted camera to capture a photo with a size of 10,000 bytes to be displayed on the tablet, after obtaining the image data, based on the image data packet generation standard of this embodiment, the image data is used as a data body, and a 128-byte data header is added before the data body to generate the image data packet corresponding to the entire photo.

[0466] As you can understand, bytes start at 0, so the total size of the image data packet is 10,128 bytes: a 128-byte header and a 10,000-byte body. Bytes 0 to 127 correspond to the header, and bytes 128 to 10,127 correspond to the 10,000-byte image data.

[0467] Specifically in this embodiment, the data header in the image data packet includes two parts. One part is used to record whether the image data is normal, which is called the data normal identification field in this embodiment, and the other part is reserved for the expansion of subsequent services, which is called the extension field in this embodiment. In order to better understand the data header of this structure, let's combine Figure 14 Provide explanation.

[0468] See also Figure 14 In the 128 bytes of the data header, bytes 0 to 3 are used to record whether the image data is normal, that is, the data normality identification field, and bytes 4 to 127 are reserved for subsequent business expansion, that is, the extension field.

[0469] For example, in some implementations, it can be agreed that the user uses "0" to indicate that the data is normal, and uses "1" to indicate that the data is abnormal. That is, if the content in byte 0 is "0", it indicates that the image data in the data body is normal data (essentially the binary data of the photo that needs to be returned to the tablet), that is, the corresponding photo can be restored after processing by the tablet. If the content in byte 0 is "1", it indicates that the image data in the data body is abnormal data, for example, it may be an abnormal character string.

[0470] For example, in some implementations, it may be agreed that different exception character strings correspond to different exception causes, which facilitates locating the exception later and resolving the exception.

[0471] In addition, it should be noted that in some implementations, the extended field can be further divided according to business needs. For example, for one-to-many scenarios, that is, a desk lamp needs to transmit image data to multiple tablets and mobile devices, it can be agreed that the photo request carries the unique identifier of the photo to be taken, and when the desk lamp generates the above-mentioned image data packet, it can add an identifier that identifies the uniqueness of the photo in the 4-32 bytes of the extended field.

[0472] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0473] S1011, the desk lamp divides the data in the image data packet into N small packets according to the set size to obtain N small packets of image data (each small packet of image data includes a data header; an identification field that records whether the small packet of image data is the first packet, the middle packet, or the last packet; and a data body that stores the image data).

[0474] It is understandable that the size of the image data packet can be dynamically set according to the bandwidth resources of the transmission channel, the current network quality, etc., or can be set to a fixed value. The specific implementation method is not limited in this embodiment.

[0475] For the sake of convenience, this embodiment takes the size of 1394 bytes as an example, and the image data packet is 10128 bytes as mentioned above. Figure 15 The process of dividing an image data packet into multiple image data packets is described.

[0476] See also Figure 15 Taking the divided image data packets as an example, which are subsequently transmitted using the RTP protocol, the data structure of the divided image data packets in this embodiment includes three parts: an RTP data header, an identification field, and a data body.

[0477] Continue to see Figure 15 For any image data packet, bytes 0 to 11 are used as the data header, byte 12 is used as the identification field, and bytes 13 to 1393 are used as the data body. In other words, bytes 13 to 1393 of each image data packet are used to store the data in the image data packet. Therefore, each image data packet only stores 1381 bytes of data from the image data packet.

[0478] In addition, it should be noted that, in the image data packet as the first packet, the data body also includes the 128-byte data header in the image data packet, so in the image data packet of the first packet, such as Figure 15 The image data actually stored in the image data packet 1 is 1253 bytes of content, and the subsequent image data packets do not need to include the 128-byte data header in the image data packet, so it is 1381 bytes. Based on this, for an image data packet of 10128 bytes, it can be split into 8 image data packets with the above structure. The image data packet 8 as the tail packet is less than 1394 bytes in size, specifically a 12-byte RTP data header + a 1-byte identification field + 589 bytes of image data, that is, the size of image data packet 8 is 601 bytes.

[0479] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0480] See also Figure 16 , exemplarily shows the specific format of the RTP data header of each image data packet. Figure 16 As shown in the figure, byte 0 specifically stores the current version number (V), byte 1 specifically stores the data type (T) of the image data, bytes 2 and 3 specifically store the sequence number (SEQNUM), bytes 4 to 7 specifically store the timestamp (RTPTIME), and bytes 8 to 11 specifically store the synchronization source identifier (SSRC).

[0481] It should be noted that the data types mentioned above may specifically be Portable Network Graphics (PGN) type, Portable Network Graphics (JPG) type, etc., which are not listed here one by one and are not limited in this embodiment.

[0482] In addition, regarding the serial number mentioned above, in actual application, the serial number will be increased by 1 every time the desk lamp sends an image data packet.

[0483] Furthermore, the timestamp mentioned above specifically reflects the time when the first byte of the image data in the data body of the image data packet was captured. This allows the image data packets to be sequentially assembled based on the timestamp to restore the entire photo image data packet, allowing the tablet to process the photo image data and display it on the user interface.

[0484] In addition, the synchronization source identifier mentioned above is specifically used to identify the uniqueness of each image data packet, thereby avoiding repeated sending of the same image data packet, which would cause redundancy in the cache queue on the tablet side.

[0485] See also Figure 17 , which exemplarily shows the specific format of the identification field of each image data packet. Figure 17 As shown in the figure, the 12-byte frames 0 and 1 specifically store the packet identification information that identifies whether the current image data packet is the first packet, the middle packet, or the last packet. The 2-byte frame specifically stores the status identifier that identifies whether the current image data packet is normal. The remaining 3 to 7 frames are reserved for subsequent expansion according to business needs.

[0486] That is, in the image data packet provided in this embodiment, which includes an RTP data header, an identification field, and a data body structure, the identification field is specifically used by the tablet for packet assembly processing.

[0487] Regarding the packet identification information mentioned above, in some implementations, it can be agreed to use "10" (the position of frame 0 is "1", and the position of frame 1 is "0") to indicate that the current image data packet is the first packet, that is, it includes the first frame of image data, and when assembling the packet, it is necessary to start from the first frame of image data in the image data packet.

[0488] For example, it can also be agreed that "01" (the position of frame 0 is "0", and the position of frame 1 is "1") indicates that the current image data packet is the last packet, that is, it includes the last frame of the image data. When assembling the packets, the image data packet is the last packet of the photo that needs to be obtained this time, and the package assembly operation is triggered after receiving the packet.

[0489] For example, it can also be agreed that "00" (the position of frame 0 is "0", and the position of frame 1 is "0") is used to indicate that the current image data packet is an intermediate packet.

[0490] In addition, it should be noted that, considering that in actual applications, when image data of multiple photos need to be transmitted, the data of the previous and next photos may be located in the same image data packet. For example, the last frame of the previous photo and the first frame of the next photo are located in the same image data packet. The user can agree that "11" (the position of frame 0 is "1", and the position of frame 1 is "1") indicates that the current image data packet includes the last frame of the previous photo and the first frame of the next photo.

[0491] Regarding the status identification field, that is, the content filled in the 2 frames of 12 bytes can be divided into "0" and "1". When it is filled with "0", it means that the current image data packet is normal. When it is filled with "1", it means that the current image data packet is abnormal.

[0492] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0493] In order to better understand the two data structures of normal and abnormal when the image data packet is the first packet, the two data structures of normal and abnormal when the image data packet is the middle packet, the two data structures of normal and abnormal when the image data packet is the last packet, and the data structure of the abnormal feedback packet (such as the abnormal feedback packet 1 and abnormal feedback packet 2 mentioned above), the following is combined with Figures 18a to 20b Provide explanation.

[0494] See also Figure 18a and Figure 18bFor the first packet data structure, the identification field's frame 0 position is filled with "1," and the frame 1 position is filled with "0." That is, when the packet identification information corresponding to frames 0-1 is "10," the current image data packet is the first packet. As can be seen from the above description, in this embodiment, a status flag of "0" identifies the image data packet as a normal packet, and a status flag of "1" identifies the image data packet as an abnormal packet.

[0495] Based on this, when the 3 frame positions of the identification field are filled with "0" (such as Figure 18a ), it indicates that the current image data packet is a normal packet, that is, for a normal first packet, the content filled in its identification field is as follows Figure 18a As shown, in addition to filling the actual image data, the data body also reserves 128 bytes for filling Figure 14 The data in the header is shown.

[0496] For example, when the 3rd frame position of the identification field is filled with "1" (such as Figure 18b ), it indicates that the current image data packet is an abnormal packet, that is, for the abnormal first packet, the content filled in its identification field is as follows Figure 18b As shown, in addition to filling the actual image data, the data body also reserves 128 bytes for filling Figure 14 The data in the header is shown.

[0497] See also Figure 19a and Figure 19b For the intermediate packet data structure, the identification field's frame 0 position is filled with "0," and the frame 1 position is filled with "0." That is, when the packet identification information corresponding to frames 0-1 is "00," the current image data packet is an intermediate packet. As can be seen from the above description, in this embodiment, a status flag of "0" identifies the image data packet as a normal packet, and a status flag of "1" identifies the image data packet as an abnormal packet.

[0498] Based on this, when the 3-frame position of the identification field is filled with "0" (such as Figure 19a ), it indicates that the current image data packet is a normal packet, that is, for a normal intermediate packet, the content filled in its identification field is as follows Figure 19a As shown, every byte in the data body is used to fill the actual image data.

[0499] For example, when the 3rd frame position of the identification field is filled with "1" (such as Figure 19b ), it indicates that the current image data packet is an abnormal packet, that is, for an abnormal intermediate packet, the content filled in its identification field is as follows Figure 19b As shown, every byte in the data body is used to fill the actual image data.

[0500] See also Figure 20a and Figure 20b In the tail packet data structure, the identification field's frame 0 position is filled with "0" and the frame 1 position is filled with "1." That is, when the packet identification information corresponding to frames 0 and 1 is "01," the current image data packet is the tail packet. As can be seen from the above description, in this embodiment, a status flag of "0" identifies the image data packet as a normal packet, and a status flag of "1" identifies the image data packet as an abnormal packet.

[0501] Based on this, when the 3 frame positions of the identification field are filled with "0" (such as Figure 20a ), it indicates that the current image data packet is a normal packet, that is, for a normal tail packet, the content filled in its identification field is as follows Figure 20a As shown, every byte in the data body is used to fill the actual image data.

[0502] For example, when the 3rd frame position of the identification field is filled with "1" (such as Figure 20b ), it indicates that the current image data packet is an abnormal packet, that is, for the abnormal tail packet, the content filled in its identification field is as follows Figure 20b As shown, every byte in the data body is used to fill the actual image data.

[0503] In addition, it should be noted that for the tail packet, there may be insufficient remaining image data, that is, it cannot fill all the bytes in the data body. In this case, for the scenario of the specified overall size of the image data packet, there will be a remaining portion of the data body of the tail packet that is not filled with bytes of image data.

[0504] For example, in the case of the remaining bytes in the above data body, the remaining bytes may be uniformly filled with default content, such as "0".

[0505] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0506] In addition, it should be noted that based on the data structure of the image data packet given in this embodiment, the abnormal feedback package 1 sent to the tablet when an abnormality occurs in the above-mentioned camera, and the abnormal feedback package 2 sent to the tablet when the camera does not support the resolution carried in the photo request, can also be sub-packetized based on the above-mentioned sub-packetization principle.

[0507] It is understandable that since the abnormal feedback packets (abnormal feedback packet 1, abnormal feedback packet 2) are usually small, all the data can be stored in a small packet after the above-mentioned packet sub-packet processing. In this case, the packet identification information mentioned above is "11". In addition, the abnormal feedback packet has only 128 bytes in its data body. The data structure of these 128 bytes is the same as Figure 14 The structure of the data header shown in is similar to that of the data header. The corresponding bytes 0 to 3 are directly filled with error codes that indicate that the lamp cannot shoot due to camera or resolution, and fields 4 to 17 are extended fields. That is, the data structure of the abnormal feedback packet can be as follows: Figure 21 shown.

[0508] It is understandable that since the abnormal feedback packet is essentially a normal data packet, that is, a data packet that needs to be parsed by the tablet to obtain the error code in the data body, rather than directly discarded without processing, the tail "0" filled in the 2 frames of the status identification information in the identification field of the abnormal feedback packet is used to fill.

[0509] S1012, the desk lamp sends N image data packets.

[0510] Specifically, after the data in the image data packet is divided into packets according to the above data structure to obtain N image data packets that need to be sent, the desk lamp can send these N image data packets through the image data channel negotiated with the tablet.

[0511] It is understandable that if it is determined during the negotiation phase that the transmitted data needs to be encrypted, each image data packet needs to be encrypted according to the negotiated encryption method, and then the encrypted image data packet is sent to the tablet.

[0512] S1013: The tablet performs subsequent processing according to the content of the identification field in the currently received image data packet.

[0513] As you can understand, if the received image data packet is encrypted, the tablet first decrypts it according to the decryption method negotiated with the lamp to obtain the plaintext image data packet. The tablet then determines whether the packet identification information is the agreed-upon "10" by checking whether the 0th frame in the 12-byte identification field is "1" and the 1st frame is "0." This determines whether the currently received image data packet contains the first frame, i.e., the first packet.

[0514] Correspondingly, if so, the image data packet is cached in the cache queue, and then the above-mentioned processing is performed on the received image data packet. If the packet identifier is the agreed "00", it is determined to be an intermediate packet, and it is cached in the cache queue. Then the above-mentioned processing is performed on the received image data packet until the packet identification information of the received image data packet is the agreed "01" or "11", and it is cached in the cache queue, and the reception of image data packets is stopped, and the packet assembly operation is started.

[0515] For ease of understanding, the following Figures 18a to 21 Describes the operations involved in package assembly.

[0516] Still taking the above example where the image data is 10,000 bytes as an example, when performing the packet grouping operation, the tablet will take out the image data packet 1 from the cache queue, such as Figure 18a As shown, in the 12-byte identification field, the recorded packet identification information is "10", indicating that the image data packet 1 is the first packet, and the data body of the packet includes Figure 13 The data header of the entire image data packet is shown, and the first byte of the data header records the identification of whether the image data is normal. Therefore, the data body of the image data packet 1 can be parsed to extract the content recorded in byte 13, and then determine whether the image data is normal.

[0517] For example, when the identifier recorded in byte 13 is "1", according to the above description, "1" indicates that the image data is abnormal. In this case, there is no need to parse and process other image data packets, such as image data packet 2 to image data packet 8 mentioned above, and the cache queue can be directly cleared.

[0518] For example, in some implementations, after clearing the cache queue, the tablet may prompt the user on the user interface that the image data captured this time is abnormal and cannot be displayed, thereby enabling the user to trigger a photo request again to obtain image data.

[0519] For example, in other implementations, after clearing the cache queue, the tablet can also simulate user operations and automatically generate a photo request, so that the desk lamp can re-execute the above process according to the new photo request.

[0520] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0521] For example, when the identifier recorded in byte 13 is "0", according to the above description, "0" indicates that the image data is normal. In this case, other image data packets cached in the cache queue can be parsed and processed.

[0522] For example, if the content of frame 0 in the 12 bytes of the currently parsed image data packet is "0", the content of frame 1 is "0", that is, the packet identification information is "00" (such as Figure 19a As shown in ), this indicates that the image data packet is an intermediate packet, and the content filled in 2 frames of the 12 bytes is "0" (as shown in Figure 19a ), which indicates that the image data packet is normal and the data body can be parsed to obtain the image data of the intermediate packet.

[0523] For example, if the content of frame 0 in the 12 bytes of the currently parsed image data packet is "0", the content of frame 1 is "0", that is, the packet identification information is "00" (such as Figure 19a As shown in ), this indicates that the image data packet is an intermediate packet, and the content filled in 2 frames of the 12 bytes is "1" (as shown in Figure 19b ), this indicates that there is an anomaly in the image data packet. To avoid the image data in the data body being affected by the transmission anomaly, in this case, there is no need to parse and process the intermediate packet and subsequent image data packets. Instead, you can directly clear the cache queue and delete the spliced ​​image data.

[0524] For example, if the content of frame 0 in the 12 bytes of the currently parsed image data packet is "0", and the content of frame 1 is "1", that is, the packet identification information is "01" (such as Figure 20a As shown), this indicates that the image data packet is the tail packet, and the content filled in the 2 frames of the 12 bytes is "0" (as shown in Figure 20a ), which indicates that the image data packet is normal and the data body can be parsed to obtain the image data of the tail packet.

[0525] For example, if the content of frame 0 in the 12 bytes of the currently parsed image data packet is "0", and the content of frame 1 is "1", that is, the packet identification information is "01" (such as Figure 20a As shown in ), this indicates that the image data packet is the tail packet, and the content filled in 2 frames of the 12 bytes is "1" (as shown in Figure 20b ), this indicates that there is an anomaly in the image data packet. To avoid the image data in the data body being affected by the transmission anomaly, there is no need to parse the tail packet in this case. Instead, the cache queue can be cleared directly and the spliced ​​image data can be deleted.

[0526] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0527] Therefore, the image data transmission method provided by this embodiment divides the image data packet into packets according to the above-mentioned data structure before transmission, so that the tablet that receives the image data packet can quickly and accurately determine whether the current image data packet is the first packet, the middle packet, or the last packet based on the packet identification information of the identification field in each image data packet.

[0528] That is to say, based on the image data transmission method provided in this embodiment, there is no need to parse the data body of the received image data packet. The first packet, middle packet and last packet of each photo that needs to be restored can be quickly and accurately determined, thereby quickly and accurately realizing package assembly.

[0529] In addition, it can be understood that the above is an explanation of the image data transmission method provided by this application from the perspective of the two devices, the tablet and the desk lamp. In the specific implementation, the implementation of the above process needs to involve educational applications, device management services, camera services, transmission management services, hardware virtualization services, virtual camera HAL on the tablet side, and device interconnection services, hardware abstraction services, and cameras on the desk lamp side. For the specific interactions between these modules, please refer to the description of the word search scenario and homework submission scenario above, which will not be repeated here.

[0530] Furthermore, considering that in the image data transmission scenario, it may be impossible to generate image data packets for transmission due to hardware problems on the desk lamp side itself, such as the camera having an abnormality or not supporting the required resolution as mentioned in the above embodiment, it may also be due to abnormalities in the image data packets generated during the transmission process, or the image data in the transmitted image data packets may exceed the maximum limit set on the tablet side, or the tablet may not synthesize image data within the set timeout period due to various factors such as network reasons, and many other abnormalities may make it impossible for the tablet side to display the required image data.

[0531] In view of this, the present application also provides an exception handling method for the image data transmission process, which aims to enable the tablet to know in advance that the image data requested cannot be displayed without parsing the data body of the received data packet, so that the image data channel can be rebuilt in time and the image data can be acquired again.

[0532] Specifically in this embodiment, the exception handling method of the image data transmission process is implemented by the network packet collection module, network packet assembly module, data processing module and image processing module in the hardware virtualization service on the tablet side.

[0533] Regarding the processing done by the above four functional modules, for example Figure 22 shown.

[0534] See also Figure 22When the desk lamp side packets the captured image data based on the above-mentioned packetization standard and transmits it to the tablet side through the image data channel negotiated with the tablet, the network packet collection module integrated in the hardware virtualization service on the tablet side will collect the image data packets transmitted from the desk lamp side and cache them in the cache queue.

[0535] Continue to see Figure 22 After the network packet collection module collects the image data packets, it will report the collected image data packets to the network packet assembly module, which will parse and process the image data packets, and then extract the actual image data carried in the data body for splicing. After the image data in all the image data packets are spliced ​​together, the spliced ​​image data will be sent to the data processing module, which will add other necessary data, such as identification information, and then generate an access unit. The data processing module will then send the generated access unit to the image processing module, which will then perform graphical processing on the access unit according to the configuration options, and finally obtain a picture that can be displayed in the user interface.

[0536] For the specific information included in the accessunit and the specific implementation details of processing the accessunit into a visual image, please refer to the existing standards and will not be repeated here.

[0537] The following describes the exception handling method for the image data transmission process provided by this embodiment for the four exception situations mentioned above.

[0538] Abnormal handling on the desk lamp side:

[0539] See also Figure 23 For example, when the camera on the desk lamp side has an abnormality, or the camera does not support the resolution carried in the photo request, the desk lamp will generate an abnormal feedback package based on the subpackaging standards given above.

[0540] For example, in this embodiment, based on the above-mentioned subpacketization standard, the generated abnormal feedback packet specifically includes an RTP data header, an identification field, and a data body. The format of the RTP data header is as follows: Figure 16 As shown, the structure of the identification field and data body is as follows Figure 21 As shown, no further details are given here.

[0541] In addition, it should be noted that in order to enable the tablet side to be informed of the abnormal cause of each image data transmission process, different error codes can be pre-agreed to represent the corresponding abnormal cause. Specifically in this embodiment, the error code "-299999990" is used to represent the abnormality caused by the fault encountered on the desk lamp side, that is, when the desk lamp side cannot capture image data due to reasons such as the camera and resolution, or when the captured image data itself has an abnormality, the 0~4 bytes of the reserved 128-byte data header in the data body of the generated abnormal feedback packet are filled with the error code "-299999990".

[0542] In addition, it should be noted that, whether it is an abnormal feedback packet sent by the desk lamp or an image data packet, for the tablet side, they can be regarded as network packets. Therefore, the network packets to the tablet side will be cached in the cache queue by the network packet collection module first, and reported to the network packet assembly module. That is, the abnormal feedback packet of the above structure sent by the desk lamp side will be cached by the network packet collection module, and the abnormal feedback packet will be reported to the network packet assembly module, such as Figure 23 shown.

[0543] After receiving the network packet (actually an abnormality feedback packet in this embodiment) reported by the network packet collection module, the network packet assembly module extracts the identification information in the identification field.

[0544] like Figure 23 As shown, when the packet identifier extracted by the network packet assembly module from the network packet is "11", it indicates that the packet is an abnormal feedback packet. At this time, it is necessary to parse the data body of the packet and extract the error code filled in bytes 0 to 3 in the data body.

[0545] Since this exception is caused by a fault on the desk lamp side, the extracted error code is "-299999990". In this case, the data processing module and the image processing module will not participate in the processing, and the network packet assembly module will directly report the error code across threads.

[0546] Understandably, in practical applications, the network packet collection module, network packet assembly module, data processing module, and image processing module can all be managed by independent threads, enabling asynchronous operation and preventing interference between these four functional modules. Therefore, the cross-thread error code reporting mentioned above bypasses the data processing model and image processing module, directly reporting the error code to the upper-level module / application.

[0547] For example, in some implementations, the error can be reported directly to the educational application at the application layer, and then the error code or the description of the abnormal cause corresponding to the error code is displayed in the user interface corresponding to the educational application, such as there is an abnormality in the camera or resolution, please check the camera, reset the resolution, etc.

[0548] For example, in some other implementations, it can also be reported directly to the virtual camera mentioned above, which will automatically trigger the call of the virtual camera to send a photo request to the desk lamp. For reasons of resolution abnormality, the resolution carried in the regenerated photo request can be adjusted to suit the camera of the current desk lamp.

[0549] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0550] Exception handling in the desk lamp sending process:

[0551] See also Figure 24 For example, when the camera on the desk lamp side is normal, it captures image data according to the resolution carried in the photo request sent by the tablet, and sub-packets it according to the above-mentioned sub-packetization standard to obtain several image data packets. The desk lamp will then transmit the first packet, middle packet, and last packet of the image data packets to the tablet in sequence through the pre-negotiated image data channel.

[0552] Continue to see Figure 24 When the first packet sent by the desk lamp arrives at the tablet side, it will also be cached by the network packet collection module and reported to the network packet assembly module (actually the first packet).

[0553] Correspondingly, after the network packet assembly module receives the network packet reported by the network packet collection module, it will extract the identification information in the identification field.

[0554] like Figure 24 As shown, when the packet identifier extracted by the network packet assembly module from the network packet is "10", it indicates that the packet is the first packet. Then it is determined whether the status identifier is "0". If it is "0", it indicates that the packet is normal. At this time, it is necessary to parse the data body of the packet and extract the image data filled in each byte from the 128th byte of the data body backward to obtain the actual image data carried in the first packet.

[0555] It is understandable that for each image data packet sent by the desk lamp side, the network packet collection module will perform the same operation, that is, cache the received network packets, and report them to the network packet assembly module. The network packet assembly module will extract the packet identification information and determine the status of each network packet according to the above-mentioned processing logic, and then parse the data body of the network packet when the network packet is a normal packet, and extract the actual image data from the corresponding position. However, if after the above-mentioned manual inspection, the desk lamp encounters a hardware failure or network failure when subsequently transmitting the remaining image data packets, intermediate packets, and tail packets, the content of the status identification frame in the identification field of the image data packet transmitted during the failure period will be filled with "1" to inform the tablet that a failure has occurred, and other image data packets related to the image data packet need to be discarded. For ease of explanation, this embodiment takes the above-mentioned exception occurring during the transmission of the intermediate packet as an example.

[0556] Continue to see Figure 24 When the packet identifier extracted from the received network packet is "00", it indicates that the packet is an intermediate packet. Then, it is determined whether the status identifier is "0". Since the desk lamp fails during the transmission process, the status identifier is modified to "1". Therefore, the extracted status identifier is "1", indicating that the packet is abnormal. At this time, the network packet assembly module will find the spliced ​​image data, that is, the image data extracted from the first packet, and report the cause of the abnormality across processes, and notify the network packet collection module to clear the cache.

[0557] In addition, it should be noted that in some implementations, the tablet may continue to receive network packets transmitted from the lamp side before responding to the current exception. In this case, the network packet collection module continues to cache and report the cached network packets to the network packet assembly module, and the network packet assembly module can first detect whether the packet carries the first packet identifier, that is, "10". If not, it will be discarded directly and the network packet collection module will be notified to clear the cache. If there is a first packet identifier, the network packets will be collected again, and the status will be judged according to the above processing logic. The image data will be parsed and extracted for splicing, so that after all the image data are spliced, the spliced ​​image data will be reported to the data processing module for processing to obtain an access unit. Finally, the image processing module will perform graphical processing on the access unit to obtain a picture that can be displayed on the user interface.

[0558] Exception handling when the tablet receives more packets than the maximum limit:

[0559] See also Figure 25 For example, the desk lamp continuously transmits image data packets to the tablet, such as the first packet, middle packet 1, middle packet 2, and last packet. The network packet collection module still continuously caches the received network packets and reports them to the network packet assembly module.

[0560] Accordingly, if the network packet assembly module determines that the first packet and the intermediate packet 1 are both normal packets according to the processing logic given in the above embodiment, the data bodies of the first packet and the intermediate packet 1 are parsed to extract the actual image data for splicing, and after obtaining the spliced ​​image data 1, if the intermediate packet 2 is received at this time, and the spliced ​​image data 1 is already greater than the set threshold, or if the spliced ​​image data 1 is greater than the set threshold after splicing the image data carried in the intermediate packet 2, this situation indicates that the size of the image data in the image data packet received by the tablet has exceeded the set maximum limit. Even if the image data carried in the subsequent image data packets is spliced ​​with the spliced ​​image data 1, the final image data cannot be displayed normally in the user interface of the tablet. Therefore, when such an exception occurs, the network packet assembly module will not parse the intermediate packet 2 and subsequent intermediate packets to extract the image data in the data body, but will mark the intermediate packet 2 and subsequent intermediate packets as over-limit. Specifically, these intermediate packets can be marked as error states and discarded.

[0561] Continue to see Figure 25 After completing the above operations, the network packet assembly module will not report the exception across threads, nor will it notify the network packet collection module to clear the cache. Instead, it will continue to wait for the network packet reported by the network packet collection module. When the packet is determined to be the last packet or the first packet based on the packet identifier in the received network packet, the already spliced ​​spliced ​​image data 1 will be deleted, and the cause of the exception will be reported across threads, specifically that the number of received packets exceeds the maximum limit, and the network packet collection module will be notified to clear the cache.

[0562] It is understandable that since the above operation may be triggered by receiving the last packet or the first packet, when notifying the network packet collection module to clear the cache, for the case of the last packet, all network packets cached in the cache queue may be cleared; for the case of the first packet, all network packets in the cache queue except the first packet may be cleared.

[0563] Exception handling when tablet receives packet exceeding time limit:

[0564] In order to avoid the tablet from being stuck in a long wait due to not receiving the required image data for a long time after sending a photo request, resulting in the occupation of tablet resources. Usually, a timeout period will be set on the tablet side. If the image data that can be displayed has not been obtained after the timeout period, a timeout prompt will be given. For this abnormal scenario, the technical solution provided by this embodiment will, after determining the timeout, the image processing module will actively send a destruction instruction to the data processing module, so that the data processing module can automatically perform the destruction operation according to the destruction instruction. At the same time, the data processing module will also send a destruction instruction to the desk lamp side, so that the desk lamp will destroy the image data channel between it and the tablet, and stop sending image data packets to the desk lamp.

[0565] For the implementation of this exception handling method, see Figure 26 For example, after the tablet sends a photo request with a specified resolution to the desk lamp by calling the virtual camera, the image processing module automatically starts a timer. As the timer ticks, the network packet collection module caches the received network packets according to the aforementioned processing logic and reports them to the network packet assembly module. The network packet assembly module then identifies and determines the packets according to the aforementioned processing logic, and then parses and stitches the image data.

[0566] Continue to see Figure 26 The network packet collection module and the network packet assembly module repeatedly perform the above processing. At this time, if the timing of the timer started by the image processing module has reached the set timeout period, it will actively send a destruction instruction to the data processing module, so that the data processing module can automatically perform the destruction operation according to the destruction instruction. At the same time, the data processing module will also send a destruction instruction to the desk lamp side, so that the desk lamp destroys the image data channel between it and the tablet, and stops sending image data packets to the desk lamp.

[0567] In this way, before the user re-triggers the photo request, the image data channel with the desk lamp side can be interrupted in advance, and the image processing module, data processing module, network packet assembly module and network packet collection module can be destroyed, thereby reducing the resource usage of the tablet.

[0568] In addition, it can also be understood that, in order to implement the above functions, the electronic device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0569] In addition, it should be noted that the methods provided in the above embodiments implemented by electronic devices in actual application scenarios can also be performed by a chip system included in the electronic device, wherein the chip system may include a processor. The chip system can be coupled to a memory so that when the chip system is running, it calls the computer program stored in the memory to implement the steps performed by the above electronic device. The processor in the chip system can be an application processor or a processor other than an application processor.

[0570] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the methods in the above-mentioned embodiments.

[0571] In addition, an embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned related steps to implement the methods in the above-mentioned embodiments.

[0572] In addition, an embodiment of the present application also provides a chip (which may also be a component or module), which may include one or more processing circuits and one or more transceiver pins; wherein, the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment to control the receiving pin to receive the signal, so as to control the transmitting pin to send the signal.

[0573] In addition, it can be seen from the above description that the electronic device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0574] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for handling exceptions in an image data transmission process, characterized in that: Applied to a first electronic device, the method includes: receiving a network packet sent by a second electronic device, wherein the network packet carries packet identification information and status identification information; When the packet identification information indicates that the network packet is an exception feedback packet, parsing the data body of the exception feedback packet, extracting the error code carried in the exception feedback packet, and reporting the error code to the target application; When the packet identification information indicates that the network packet is an image data packet, determining whether the image data packet is normal according to the status identification information; wherein the image data packet is a first packet, an intermediate packet, or a last packet; When it is determined that the image data packet is abnormal according to the status identification information, deleting the parsed image data, clearing the cache, and reporting the abnormality cause to the target application; When it is determined that the image data packet is normal according to the status identification information, and the image data packet is an intermediate packet, it is determined whether the extracted image data is greater than a set threshold, or whether the extracted image data after being spliced ​​with the image data of the intermediate packet is greater than the set threshold; When the value is greater than the set threshold, the intermediate package is marked as over-limit and discarded; After the intermediate packet is marked as over-limit and discarded, when the last packet or a new first packet is received, the extracted image data is deleted, the cache is cleared, and the cause of the abnormality is reported to the target application.

2. The method according to claim 1, characterized in that The network packet includes a data header, an identification field and a data body; The data header occupies 12 bytes, and the identification field occupies 1 byte; The combined content of the binary data corresponding to the 0th frame and the 1st frame in the identification field is the packet identification information, and the binary data corresponding to the 3rd frame in the identification field is the status identification information; The data body is used to store the image data or the error code.

3. The method according to claim 2, characterized in that When the packet identification information indicates that the network packet is the abnormality feedback packet, the data body occupies 128 bytes; The error code is filled in bytes 0 to 3 of the data body.

4. The method according to claim 3, characterized in that When the packet identification information indicates that the network packet is an exception feedback packet, parsing the data body of the exception feedback packet and extracting the error code carried in the exception feedback packet includes: The data body of the abnormal feedback packet is parsed, and the error code is extracted from bytes 0 to 3 of the data body.

5. The method according to claim 2, characterized in that When the packet identification information indicates that the network packet is an image data packet, and the image data packet is a first packet, the data body includes a first data portion and a second data portion; The first data portion is a reserved field, occupies 128 bytes, and is used to carry extended information; The second data portion is used to fill the image data.

6. The method according to claim 5, characterized in that The image data packet is the first packet; The method further comprises: When it is determined that the image data packet is normal based on the status identification information, the image data packet is parsed and the image data is extracted from the second data portion.

7. The method according to claim 2, characterized in that When the packet identification information indicates that the network packet is an image data packet, and the image data packet is an intermediate packet or a tail packet, the data body is filled with the image data starting from 0 bytes.

8. The method according to claim 7, characterized in that The image data packet is an intermediate packet or a tail packet; The method further comprises: When it is determined that the image data packet is normal according to the status identification information, the image data packet is parsed and the image data is extracted starting from byte 0 of the data body.

9. The method according to claim 1, characterized in that When the last packet or the first packet is received, the extracted image data is deleted and the cache is cleared, including: When the tail packet is received, the extracted image data is deleted and all cached image data packets are cleared; When a new first packet is received, the extracted image data is deleted, and all cached image data packets except the new first packet are cleared.

10. The method according to any one of claims 1 to 8, characterized in that The method further comprises: When sending a photo-taking request to the second electronic device, starting a timer; During the timing of the timer, receiving the network packet sent by the second electronic device, and processing the network packet according to the packet identification information and the state identification information in the network packet; During the process of processing the network packet, when the timer reaches the set timeout period, the thread processing the network packet is destroyed, and a destruction instruction is sent to the second electronic device to enable the second electronic device to destroy the image data channel between the second electronic device and the first electronic device.

11. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the exception handling method for the image data transmission process according to any one of claims 1 to 10.

12. A collaborative work system, characterized in that: include: A first electronic device configured to execute the method for handling exceptions in an image data transmission process according to any one of claims 1 to 10, and a second electronic device, wherein the second electronic device is provided with a camera configured to collect image data, and a target application in the first electronic device is bound to the second electronic device; The first electronic device is used to: register a virtual camera corresponding to the camera in the system, and send a photo request to the second electronic device by calling the virtual camera; The second electronic device is used to: call the camera to capture image data according to the photo-taking request of the first electronic device, and send the image data to the target application of the first electronic device for preview display. The system according to claim 12 , wherein the second electronic device is an Internet of Things device.

14. The system according to claim 13, wherein the IoT device is a desk lamp, and the camera is used to collect image data downward.

15. A computer-readable storage medium comprising a computer program, characterized in that When the computer program is run on an electronic device, the electronic device is enabled to execute the method for handling exceptions in an image data transmission process according to any one of claims 1 to 10.

Images