Image data transmission method, device and collaborative working system

Abstract

The present application provides an image data transmission method, device, and collaborative working system. The method sets an identification field in each image data packet that can record packet identification information for identifying whether the image data packet is the first packet, the middle packet, or the last packet, so that an electronic device receiving these image data packets can quickly and accurately determine whether the image data packet is the first packet, the middle packet, or the last packet based on the packet identification information recorded in the identification field without parsing the image data packet, that is, without knowing the specific content carried in the second data header and the second data body. In this way, when the received image data packet is identified as the last packet, the data packetization operation can be performed.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to an image data transmission method, device, and collaborative working system. Background Art

[0002] With the advancement of communication technology, data transmission, such as image data transmission, can be carried out between any two electronic devices based on business needs. Considering that image data is usually large, to speed up transmission and avoid network congestion, the image data of a large photo, such as 100K to 20M, is usually broken down into multiple image data packets. These packets are then placed in a queue to be transmitted, and subsequently retrieved from the queue for transmission.

[0003] However, for image data, the final displayed photo must be determined based on the positional relationship between image data packets, and the last packet triggers the packet assembly operation. However, in current image data transmission methods, the data body of the image data packet must be parsed to determine whether it is the first packet. Moreover, even if the data body of the image data packet is parsed, it is still impossible to determine whether it is the last packet, which means that packet assembly cannot be triggered quickly and accurately. Summary of the Invention

[0004] In order to solve the above technical problems, the present application provides an image data transmission method, device and collaborative working system, which aims to quickly and accurately determine whether the image data packet is the first packet, the middle packet or the last packet without parsing the data body of the received image data packet. In this way, when the received image data packet is identified as the last packet, the data packetization operation can be performed.

[0005] In a first aspect, the present application provides an image data transmission method. The method is applied to an Internet of Things device, wherein the Internet of Things device is provided with a camera for downwardly collecting image data, and the Internet of Things device is bound to a target application in the electronic device, and includes: receiving a photo request sent by the electronic device, the photo request carrying an indication of the resolution of the image data captured by the camera; calling the camera to capture an image according to the resolution to obtain image data; processing the image data into an image data packet according to a first data packet standard; packetizing the image data packet according to a second data packet standard to obtain N image data packets, where N is an integer greater than 0, and each image data packet carries packet identification information identifying the image data packet as a first packet, a middle packet, or a last packet; and transmitting the obtained N image data packets to the electronic device.

[0006] The IoT device may be any device capable of capturing images and transmitting image data to an electronic device. Specifically, in this embodiment, the IoT device may be, for example, a desk lamp with a camera, or a camera itself.

[0007] The electronic devices include mobile phones, tablets and other electronic devices with powerful processing capabilities.

[0008] The photo-taking request is generated, for example, by the electronic device in response to a user's operation in a word-looking scenario or a homework submission scenario.

[0009] For details about the word search scenario and homework submission scenario, please refer to the detailed description below and will not be repeated here.

[0010] Understandably, since word search requires accurate recognition of the content selected by the user, and submission of the assignment requires ensuring that the content of the submitted assignment is clearly visible, the resolution carried in the photo request is relatively high, such as 1080P.

[0011] The target application is, for example, the educational application mentioned below.

[0012] The first data packet standard indicates how large a data header (hereinafter referred to as the first data) the generated image data packet needs to include, what specific content the data header carries, and where the data body (hereinafter referred to as the first data body) starts.

[0013] Among them, the second data packet standard indicates the size of each image data packet, as well as how large a data header each image data packet needs to include (the second data header mentioned below), what specific content the data header carries, the identification field, and where the data body (the second data body mentioned below) starts.

[0014] Therefore, by setting an identification field in each image data packet that can record the packet identification information used to identify whether the image data packet is the first packet, the middle packet, or the last packet, the electronic device that receives these image data packets can quickly and accurately determine whether the image data packet is the first packet, the middle packet, or the last packet based on the packet identification information recorded in the identification field without parsing the image data packets, that is, without knowing the specific content carried in the second data header and the second data body. In this way, when the received image data packet is identified as the last packet, the data packetization operation can be performed.

[0015] According to the first aspect, the first data packet standard indicates that the image data packet includes a first data header and a first data body; the first data body is used to store image data; the first data header includes an image data identification field, and the image data identification information recorded in the image data identification field is used to identify whether the image data stored in the first data body is normal.

[0016] The image data is the actual data captured by the camera, which is stored in the first data body in the form of binary code, such as Figure 13 In the data body.

[0017] According to the first aspect, or any implementation of the first aspect above, the first data header further includes an extension field.

[0018] For example, in some implementations, the extended field may be further divided, for example, a portion of the extended field may be used to record a unique identifier of the image data to achieve accurate uplink transparent transmission.

[0019] Exemplarily, the identifier for uniquely identifying the image data may be generated by the electronic device and carried in the photo-taking request.

[0020] According to the first aspect, or any implementation of the first aspect above, the size of the first data header is 128 bytes, the identification field is located at 0 to 3 bytes, and the extension field is located at 4 to 127 bytes. For the structure of the first data header, please refer to Figure 14 As shown, no further details are given here.

[0021] According to the first aspect, or any implementation method of the first aspect above, the second data packet standard indicates that the image data packet includes a second data header, a packet identification field and a second data body; the second data header occupies 12 bytes, and the packet identification field occupies 1 byte; the packet identification field is the content packet identification information after the binary data corresponding to the 0th frame and the 1st frame are combined; the second data body is used to store image data.

[0022] The second data body is, for example, a real-time transport protocol RTP data header. The specific structure of the RTP data header can be found below and will not be described in detail here.

[0023] The size of each image data packet is, for example, 1394 bytes. The positions of the second data header, the packet identification field, and the second data body, as well as the bytes they occupy, can be found below and will not be described in detail here.

[0024] For example, in some implementations, it may be agreed that the 0th frame is "1" and when the 1st frame is "0", the corresponding packet identification information "10" indicates that the current image data packet is the first packet.

[0025] For example, in some implementations, it may be agreed that when the 0th frame is "0" and the 1st frame is "1", the corresponding packet identification information "01" indicates that the current image data packet is the tail packet.

[0026] For example, in some implementations, it may be agreed that the 0th frame is "0" and when the 1st frame is "0", the corresponding packet identification information "00" indicates that the current image data packet is an intermediate packet.

[0027] The specific structure of the first package, middle package and last package can be found below and will not be repeated here.

[0028] According to the first aspect, or any implementation of the first aspect above, when the packet identification information indicates that the image data packet is the first packet, the second data body is also used to store the first data header. Thus, the first data header is only carried in the second data body of the first packet, and the data in the first data body is directly stored in the remaining image data packets, thereby reducing the number of image data packets. Furthermore, by carrying the first data header in the second data body of the first packet, the electronic device can determine whether the entire image data is normal based on the image data identification information in the first data header in the second data body during packet assembly, thereby avoiding invalid packet assembly operations.

[0029] According to the first aspect, or any implementation of the first aspect above, after receiving a photo request sent by an electronic device, the method is also used to: detect whether the camera is abnormal; when the camera is abnormal, send a first abnormal feedback packet to the electronic device, the first abnormal feedback packet carries the abnormal reason indicating the abnormality of the camera; when the camera is normal, detect whether the camera supports resolution; when the camera supports resolution, execute the step of calling the camera to shoot an image according to the resolution; when the camera does not support resolution, send a second abnormal feedback packet to the electronic device, the second abnormal feedback packet carries the abnormal reason indicating the abnormal resolution. In this way, by detecting the camera and the resolution before shooting, it is possible to ensure that the captured image data is normal data, and avoid the transmission of invalid data from occupying network resources.

[0030] According to the first aspect, or any implementation of the first aspect above, the resolution is 1080P. In this way, in word lookup scenarios and homework submission scenarios, the content in the image can be accurately recognized based on the high-resolution image data, thereby ensuring the accuracy of the processing results.

[0031] According to the first aspect, or any implementation of the first aspect above, the Internet of Things device is a desk lamp.

[0032] According to the first aspect, or any implementation of the first aspect above, the electronic device includes a mobile phone and a tablet computer.

[0033] In a second aspect, the present application provides an image data transmission method. The method is applied to an electronic device, wherein a target application in the electronic device is bound to an Internet of Things device, and the Internet of Things device is provided with a camera, which is used to collect image data downward. The image data transmission method includes: registering a virtual camera corresponding to the camera in the system, and sending a photo request to the Internet of Things device by calling the virtual camera, wherein the photo request carries the resolution of the image data captured by the camera; receiving an image data packet sent by the Internet of Things device, wherein the image data packet is obtained by the Internet of Things device by subpacketizing the image data packet according to a second data packet standard, the image data packet is obtained by the Internet of Things device by processing the image data according to a first data packet standard, and the image data is obtained by the camera at a resolution. The image data packet can be divided into N image data packets, where N is an integer greater than 0, and each image data packet carries packet identification information that identifies the image data packet as a first packet, an intermediate packet, or a last packet; when receiving an image data packet carrying the packet identification information of the last packet, the N received image data packets are grouped to restore the image data captured by the camera at the resolution, and the image data is displayed in the target application.

[0034] Therefore, the electronic device can quickly and accurately determine whether the image data packet is the first packet, the middle packet, or the last packet based on the packet identification information recorded in the identification field without parsing the data body of the received image data packet, that is, without knowing the specific content carried in the second data header and the second data body. In this way, when the received image data packet is identified as the last packet, the data packetization operation can be performed.

[0035] According to the second aspect, the first data packet standard indicates that the image data packet includes a first data header and a first data body; the first data body is used to store image data; the first data header includes an image data identification field, and the image data identification field is used to identify whether the image data stored in the first data body is normal.

[0036] According to the second aspect, or any implementation of the second aspect above, the first data header further includes an extension field.

[0037] According to the second aspect, or any implementation of the second aspect above, the size of the first data header is 128 bytes, the identification field is located at 0 to 3 bytes, and the extension field is located at 4 to 127 bytes.

[0038] According to the second aspect, or any implementation method of the above second aspect, the second data packet standard indicates that the image data packet includes a second data header, a packet identification field and a second data body; the second data header occupies 12 bytes, and the packet identification field occupies 1 byte; the packet identification field is the content packet identification information after the binary data corresponding to the 0th frame and the 1st frame are combined; the second data body is used to store image data.

[0039] According to the second aspect, or any implementation of the second aspect above, when the packet identification information indicates that the image data packet is the first packet, the second data body is also used to store the first data header.

[0040] According to the second aspect, or any implementation of the second aspect above, after receiving the image data packet sent by the Internet of Things device, the method also includes: extracting packet identification information from the bytes occupied by the packet identification field according to the second data packet standard; when the packet identification information indicates that the image data packet is the first packet, adding the first packet to the image data packet cache queue; when the packet identification information indicates that the image data packet is the middle packet, adding the middle packet to the image data packet cache queue; when the packet identification information indicates that the image data packet is the last packet, adding the last packet to the image data packet cache queue, and executing the step of grouping the N received image data packets.

[0041] According to the second aspect, or any implementation method of the above second aspect, the received N image data packets are grouped, including: parsing the second data header of each image data packet according to the second data packet standard; extracting the data in the second data body of each image data packet in sequence according to the serial number and timestamp recorded in the second data header for combination.

[0042] According to the second aspect, or any implementation of the second aspect above, before parsing the second data header of each image data packet, the method also includes: determining the byte where the image data identification field in the first packet is located according to the second data packet standard and the first data packet standard; extracting the image data identification information from the bytes occupied by the image data identification field; when the image data identification information indicates that the image data is normal, executing the step of parsing the second data header of each image data packet; when the image data identification information indicates that the image data is abnormal, clearing the N image data packets cached in the image data packet cache queue.

[0043] According to the second aspect, or any implementation of the second aspect above, the method also includes: receiving a first abnormal feedback package sent by the Internet of Things device, and responding according to the abnormal cause in the first abnormal feedback package, the first abnormal feedback package carries the abnormal cause indicating the camera abnormality; receiving a second abnormal feedback package sent by the Internet of Things device, and responding according to the abnormal cause in the second abnormal feedback package, the second abnormal feedback package carries the abnormal cause indicating the resolution abnormality.

[0044] According to the second aspect, or any implementation of the second aspect above, the resolution is 1080P.

[0045] According to the second aspect, or any implementation of the second aspect above, the Internet of Things device is a desk lamp.

[0046] According to the second aspect, or any implementation of the second aspect above, the electronic device includes a mobile phone and a tablet computer.

[0047] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0048] In a third aspect, the present application provides an Internet of Things (IoT) device. The IoT device includes a memory and a processor, the memory and the processor being coupled; the memory storing program instructions, which, when executed by the processor, cause the IoT device to execute the method of the first aspect or any possible implementation of the first aspect.

[0049] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0050] In a fourth aspect, the present application provides an electronic device. The electronic device includes: a memory and a processor, the memory and the processor being coupled; the memory storing program instructions, which, when executed by the processor, cause the electronic device to execute the instructions of the method in the second aspect or any possible implementation of the second aspect.

[0051] The fourth aspect and any implementation of the fourth aspect correspond to the second aspect and any implementation of the second aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the above-mentioned second aspect and any implementation of the second aspect, and will not be repeated here.

[0052] In a fifth aspect, the present application provides a computer-readable medium for storing a computer program, which includes instructions for executing the method in the first aspect or any possible implementation of the first aspect, or instructions for executing the method in the second aspect or any possible implementation of the second aspect.

[0053] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, or the second aspect or any possible implementation of the second aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or the technical effects corresponding to the second aspect or any possible implementation of the second aspect, and are not further described here.

[0054] In a sixth aspect, the present application provides a computer program product, comprising a computer program, which includes instructions for executing the method in the first aspect or any possible implementation of the first aspect, or instructions for executing the method in the second aspect or any possible implementation of the second aspect.

[0055] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, or the second aspect or any possible implementation of the second aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or the technical effects corresponding to the second aspect or any possible implementation of the second aspect, and are not further described here.

[0056] In a seventh aspect, the present application provides a chip comprising a processing circuit and a transceiver pin. The transceiver pin and the processing circuit communicate with each other via an internal connection path, and the processing circuit executes the method of the first aspect or any possible implementation of the first aspect, or executes the method of the second aspect or any possible implementation of the second aspect, to control the receive pin to receive a signal and to control the transmit pin to send a signal.

[0057] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, or the second aspect or any possible implementation of the second aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, or the technical effects corresponding to the second aspect or any possible implementation of the second aspect, and are not further described here. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1a-Figure 1b An application scenario is shown as an example;

[0059] Figure 2a is a schematic diagram of the hardware structure of an electronic device shown as an example;

[0060] Figure 2b is a schematic diagram of the software structure of an electronic device shown as an example;

[0061] Figure 3a Schematic diagram of the hardware structure of an IoT device shown as an example;

[0062] Figure 3b Schematic diagram of the software structure of an IoT device shown as an example;

[0063] Figure 4a A schematic diagram of module interaction provided in an embodiment of the present application;

[0064] Figure 4b A schematic diagram of module interaction provided in an embodiment of the present application;

[0065] Figure 5a-5b An application scenario is shown as an example;

[0066] Figure 6a A schematic diagram of module interaction provided in an embodiment of the present application;

[0067] Figure 6b A schematic diagram of module interaction provided in an embodiment of the present application;

[0068] Figure 7a-7b An application scenario is shown as an example;

[0069] Figure 8 A schematic diagram of module interaction provided in an embodiment of the present application;

[0070] Figure 9a-9c An application scenario is shown as an example;

[0071] Figure 10 A schematic diagram of module interaction provided in an embodiment of the present application

[0072] Figure 11a-Figure 11b An application scenario is shown as an example;

[0073] Figure 12 This is a schematic diagram illustrating an exemplary embodiment of image data transmission between a tablet and a desk lamp;

[0074] Figure 13 is a schematic diagram of the data structure of the entire image data packet;

[0075] Figure 14 is a schematic diagram showing the structure of a data header in an exemplary image data packet;

[0076] Figure 15 is a schematic diagram of the data structure of an exemplary image data packet;

[0077] Figure 16 Schematic diagram of the structure of the RTP data header in an exemplary image data packet;

[0078] Figure 17 Schematic diagram of the structure of the identification field in the image data packet shown as an example;

[0079] Figure 18 This is a schematic diagram showing a structure in which the first packet of an image data packet is an abnormal packet;

[0080] Figure 19 This is a schematic structural diagram of an exemplary image data packet in which the first packet is a normal packet;

[0081] Figure 20 Schematic diagram of the structure of an intermediate packet of an exemplary image data packet;

[0082] Figure 21 Schematic diagram of the structure of the tail packet of an exemplary image data packet. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0084] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0085] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0086] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0087] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0088] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0089] With the development of the internet, online education has become increasingly popular, and users (such as students) are increasingly demanding it. In some scenarios, when encountering an unfamiliar word, students can look up the word online to obtain relevant explanations; in other scenarios, online reading of textbook content facilitates students' learning and pronunciation; and in still other scenarios, students need to submit their homework online. Therefore, how to meet users' online education needs based on smart devices is a challenge that needs to be addressed.

[0090] Currently, for online education, users typically use smart learning machines that combine both camera and display functions. These devices require cameras or mirrors in specialized locations to capture images of books, making them less versatile and user-friendly. Furthermore, devices with both camera and display capabilities require robust hardware and system support, resulting in high device costs. Consequently, the challenge remains to provide users with a superior online education experience based on smart devices, while enhancing their versatility and usability and reducing the cost of online education.

[0091] A collaborative work system provided by an embodiment of the present application can be applied to online education scenarios. The system working system includes an electronic device and a desk lamp that establish a communication connection. The desk lamp is provided with a camera that can be used to shoot books downward. The electronic device calls the camera of the desk lamp to capture images, and combines the platform's online education resources to meet the user's online education needs. Among them, the electronic device can be a tablet computer or a mobile phone. In addition to online education scenarios, electronic devices and desk lamps can also serve users based on their respective basic functions (i.e., communication functions and lighting functions). In this way, the system working system can create a better online education experience for users based on two smart devices commonly used by users, with strong versatility and ease of use. In addition, since tablet computers or mobile phones have long been essential products for every household, and the cost of a desk lamp device with only a shooting function will be lower than that of a device with both shooting and display functions, the user's online education cost is greatly reduced.

[0092] The following uses a tablet as an example to explain the technical solution provided by this application.

[0093] Figure 1aAn example application scenario is shown. Figure 1a As shown, the collaborative work system includes a tablet 100 and a desk lamp 200 that establish a communication connection. Among them, the desk lamp 200 includes a camera 201 for capturing images downward, for example, it can be used to shoot the text content or picture content in a book downward. An education APP (Application) is installed in the tablet 100. The education APP can call the camera 201 of the desk lamp 200 to capture images, and provide users with various online education functions based on the images captured by the camera 201 of the desk lamp 200, such as online word search, online reading, online homework submission, etc.

[0094] Although the tablet also has a front camera and a rear camera, no matter which camera is used to take pictures of the book, the user needs to hold the tablet and point the camera at the book, which not only makes the shooting unstable, but also affects the user's finger reading and point reading operations, and cannot provide users with a good online education experience. Figure 1a As shown, both the tablet 100 and the desk lamp 200 can be placed in fixed positions. The tablet 100 uses the camera 201 of the desk lamp 200 to shoot a book. The stable shooting image makes the content recognition success rate high, and the user can also flexibly perform finger reading and point reading operations in the book. Therefore, the linkage between the tablet and the desk lamp can provide users with a better online education experience.

[0095] like Figure 1b As shown, the tablet 100 and the lamp 200 can perform near-field communication or far-field communication. Near-field communication can be accomplished through devices such as routers to exchange information between devices, while far-field communication can be accomplished through cloud servers to exchange information between devices. For example, the tablet 100 and the lamp 200 can achieve near-field communication based on Wi-Fi (wireless fidelity) network protocols.

[0096] like Figure 2a The figure shows the structure diagram of the electronic device 100. Optionally, the electronic device 100 may be a terminal, which may also be called a terminal device. The terminal may be a cellular phone or a tablet computer, etc., which is not limited in this application. It should be noted that the structure diagram of the electronic device 100 may be applicable to Figure 1a-1b It should be understood that Figure 2a The illustrated electronic device 100 is merely one example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the figures, may combine two or more components, or may have a different configuration of components. Figure 2a The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0097] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, an air pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0098] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0099] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0100] The processor 110 may further include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0101] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.

[0102] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0103] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0104] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0105] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0106] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc.

[0107] The wireless communication module 160 can provide wireless communication solutions applied to the electronic device 100, including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc.

[0108] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 , and antenna 2 is coupled to wireless communication module 160 , so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0109] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0110] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0111] In the embodiment of the present application, the display screen 194 can display a shooting preview interface and a photo image interface, etc. It should be noted that, in the embodiment of the present application, the shooting preview interface refers to an interface where a user can view images captured in real time by the desk lamp camera through the display screen 194 .

[0112] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function.

[0113] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, for example, enabling the electronic device 100 to implement the collaborative working method in the embodiment of the present application. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0114] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0115] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0116] Speaker 170A, also known as a "horn," is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music or make hands-free calls through speaker 170A. In some embodiments, electronic device 100 can be provided with multiple speakers 170A.

[0117] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0118] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0119] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0120] The pressure sensor is used to sense pressure signals and convert the pressure signals into electrical signals. In some embodiments, the pressure sensor can be provided on the display screen 194. The electronic device 100 can also calculate the touch position based on the detection signal of the pressure sensor.

[0121] A touch sensor, also known as a "touch panel," can be mounted on display screen 194. The touch sensor and display screen 194 together form a touch screen, also known as a "touch screen." The touch sensor detects touch operations applied to or near the touch sensor. The touch sensor can communicate the detected touch operations to the application processor to determine the type of touch event.

[0122] The buttons 190 include a power button (also known as a power button), a volume button, etc. The buttons 190 can be mechanical buttons or touch buttons. The electronic device 100 can receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0123] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or touch vibration feedback. For example, touch operations on different applications (such as taking photos, playing audio, etc.) can correspond to different vibration feedback effects.

[0124] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0125] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0126] Figure 2b It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0127] The layered architecture of electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into four layers: application layer, application framework layer, system layer, HAL layer (Hardware Abstract Layer), and kernel layer.

[0128] The application layer can include a series of application packages.

[0129] like Figure 2b As shown, the application package may include call, video, Bluetooth, camera, WLAN, education applications, device manager applications, etc. The application package may also include calendar, map, navigation, music, short message applications, etc.

[0130] Among them, educational applications can be used to provide users with online educational functions, such as online word recognition, online reading, online homework submission, etc.

[0131] In some examples, the device manager application can be used to bind IoT devices such as a desk lamp. In some examples, the education application can be used to bind IoT devices such as a desk lamp.

[0132] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0133] like Figure 2b As shown, the application framework layer may include camera services, authentication services, hardware virtualization services, device management services, transmission management services, etc.

[0134] The camera service may be used to call a camera (including a front camera and / or a rear camera) in response to a request from an application.

[0135] In an embodiment of the present application, the camera service can be used to call the virtual camera on the electronic device side, that is, to call the camera in the IOT device, in response to a request from an application.

[0136] Authentication services are used to provide security permission management capabilities.

[0137] Hardware virtualization services can be used to establish a logical channel between the electronic device side (i.e., the central device side) and the IoT device side, providing the ability to virtualize the camera.

[0138] Device management services can be used for IoT device discovery and management, providing far-field (i.e., cloud-based) IoT device information and near-field (i.e., nearby connectable) IoT device information for applications such as educational applications.

[0139] Transmission management services can be used to establish physical transmission channels and provide data transmission capabilities.

[0140] In addition, it can also include window managers, content providers, view systems, phone managers, resource managers, notification managers, etc.

[0141] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0142] The content provider is used to store and obtain data and make the data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures. The phone manager is used to provide communication functions for the electronic device 100. For example, the management of call status (including connecting, hanging up, etc.). The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction.

[0143] The system library and runtime layer (ie, the system layer) includes the system library and the Android runtime.

[0144] The Android Runtime consists of the core library and the virtual machine. The Android Runtime is responsible for scheduling and management of the Android system.

[0145] The core library consists of two parts: one containing the Java language's callable functions and the other the Android core library. The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0146] In the embodiment of the present application, Android Runtime also includes a virtual camera adaptation layer that provides the ability to register a virtual camera.

[0147] The system library in the system layer can include multiple functional modules, such as multimedia platform, graphics and image processing library, codec, etc.

[0148] The multimedia platform can be used for multimedia management, supporting playback and recording of a variety of common audio and video formats, as well as static image files. The multimedia platform can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0149] The graphics and image processing library can be used to implement graphics drawing, image rendering, compositing, and layer processing.

[0150] Codecs can be used to implement encoding and decoding operations on audio data and video data.

[0151] The HAL layer is the interface layer between the operating system kernel and the hardware circuit. The HAL layer includes but is not limited to: audio HAL, sensor HAL, modem HAL, camera HAL, and virtual camera HAL.

[0152] The audio HAL is used to process audio streams, for example, by performing noise reduction and directional enhancement. The camera HAL is used to process image streams corresponding to the camera on the electronic device side, and the virtual camera HAL is used to process image streams corresponding to the virtual camera registered on the electronic device side, that is, to process image streams captured by the camera on the IoT device side.

[0153] The kernel layer is the layer between hardware and software. It includes at least the display driver, camera area, audio driver, network driver (such as Wi-Fi driver), CPU driver, USB driver, storage driver, and printer driver. Hardware includes at least the processor, display, and Wi-Fi module.

[0154] It is understandable that Figure 2bThe layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0155] like Figure 3a The schematic diagram of the hardware structure of the Internet of Things device 200 is shown. It should be noted that the schematic diagram of the structure of the Internet of Things device 200 can be applied to Figure 1a-1b It should be understood that Figure 3a The illustrated IoT device 200 is merely one example of an electronic device, and the IoT device 200 may have more or fewer components than shown, may combine two or more components, or may have a different configuration of components. Figure 3a The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0156] The IoT device 200 may include: a processor 210, a camera 201, a wireless communication module 202, a memory 203, an audio module 204, a USB interface 205, a charging management module 206, a power management module 207, a battery 208, a lighting device 209, a button 211, etc.

[0157] The processor 210 may include one or more processing units, for example, the processor 210 may include a GPU, an ISP, a controller, a memory, a video codec, etc. The different processing units may be independent devices or integrated into one or more processors.

[0158] The controller can be the nerve center and command center of the IoT device 200. The controller can generate operation control signals based on instruction opcodes and timing signals to complete the control of instruction fetching and execution.

[0159] The camera 201 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. Taking a desk lamp as an example, the camera 201 can be set on the desk lamp stand to capture images downward.

[0160] The IoT device 200 can implement a shooting function through an ISP, a camera 201, a video codec, a GPU, and the like.

[0161] The ISP processes data fed back by camera 201. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 201.

[0162] The wireless communication module 202 can provide wireless communication solutions including WLAN (such as Wi-Fi network) and Bluetooth (BT) for the IoT device 200. In some embodiments, the antenna of the IoT device 200 is coupled to the wireless communication module 202, so that the IoT device 200 can communicate with the network and other devices through wireless communication technology.

[0163] The memory 203 can be used to store computer-executable program code, which includes instructions. The processor 210 executes the instructions stored in the memory 203 to execute various functional applications and data processing of the IoT device 200, for example, enabling the IoT device 200 to implement the collaborative working method in the embodiment of the present application.

[0164] The IoT device 200 can implement audio functions, such as music playback, through the audio module 204, the speaker 212, etc.

[0165] The USB interface 205 is an interface that complies with USB standards, and may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 205 can be used to connect a charger to charge the IoT device 200, and can also be used to transfer data between the IoT device 200 and peripheral devices.

[0166] The charging management module 206 is configured to receive charging input from a charger. While charging the battery 208 , the charging management module 206 can also provide power to the IoT device 200 through the power management module 207 .

[0167] The power management module 207 is used to connect the battery 208, the charging management module 206, and the processor 210. The power management module 207 receives input from the battery 208 and / or the charging management module 206 to provide power to the processor 210, the memory 203, the camera 201, the wireless communication module 202, and the lighting device 209.

[0168] The buttons 211 include a power button (or power button) and the like.

[0169] The software system of the Internet of Things device 200 may adopt a layered architecture or other architectures, etc. In the embodiment of the present application, the layered architecture is used as an example to illustrate the software structure of the Internet of Things device 200.

[0170] Figure 3b This is a software structure block diagram of the Internet of Things device 200 according to an embodiment of the present application.

[0171] The layered architecture of IoT device 200 divides the software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the system of IoT device 200 is divided into three layers: the application layer, the application framework layer, the system layer, and the kernel layer, from top to bottom.

[0172] like Figure 3b As shown, the application layer may include device application services, which may be understood as system-level applications. The device application services are started after the system of the Internet of Things device 200 is started.

[0173] like Figure 3b As shown, the application framework layer may include device interconnection services, hardware abstraction services, resource managers, etc.

[0174] Device interconnection services can be used to establish physical transmission channels, provide data transmission capabilities, and manage the startup switches of hardware abstraction services.

[0175] The hardware abstraction service can be used to establish a logical channel between the electronic device side (i.e., the central device side) and the IoT device, provide the ability to virtualize the camera, and provide an open camera interface for the IoT device.

[0176] The resource manager can provide various resources to applications.

[0177] like Figure 3b As shown, the system layer may include a multimedia platform, a graphics and image processing library, a codec, a device adaptation module, etc.

[0178] The multimedia platform can be used for multimedia management and supports a variety of commonly used audio, video, and static image files. The multimedia platform can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0179] The graphics and image processing library can be used to implement graphics drawing, image rendering, compositing, and layer processing.

[0180] Codecs can be used to implement encoding and decoding operations on audio data and video data.

[0181] The device adaptation module can implement the interface of hardware abstraction service, provide device information and capability query, and provide functions for performing related operations on the IoT device side, such as opening the camera, taking photos, previewing, etc.

[0182] It is understandable that, in order to implement the collaborative working method in the embodiment of the present application, the electronic device 100 and the Internet of Things device 200 include hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0183] like Figure 4a The following is a schematic diagram of the interaction between modules. Figure 4a , the embodiment of the present application provides a method process for collaborative operation of a tablet and a desk lamp, specifically including:

[0184] 0. Device service initialization phase

[0185] S0.1: In response to a user operation, the device application service of the desk lamp is started and the device interconnection service is loaded.

[0186] For example, the user operation can be turning on the power of a desk lamp. In response to the user operation, the desk lamp system starts up, the device application service starts up, and the device interconnection service is loaded. The device interconnection service can be used to establish a physical transmission channel between the tablet and the desk lamp to provide data transmission capabilities.

[0187] S0.2, the device interconnection service of the desk lamp loads the hardware abstraction service.

[0188] The device interconnection service can also control the startup of the hardware abstraction service. For example, after the device interconnection service is started, the device interconnection service can load the hardware abstraction service as a plug-in. The hardware abstraction service can be used to establish a logical channel between the tablet and the desk lamp, providing virtualized camera capabilities and an open interface for the desk lamp camera.

[0189] Reference Figure 4bAs shown in the module interaction diagram, the hardware abstraction service may include at least a basic component and a camera component. In the device service initialization phase, the device interconnection service first loads the basic component and initializes the basic component. After the basic component is initialized, information can be exchanged with the device adaptation module of the desk lamp to obtain device information and virtualization capability information. Exemplarily, the device information includes but is not limited to the device name, device identification, device type, etc. Exemplarily, the virtualization capability information includes but is not limited to whether a virtualized camera is supported, whether a virtualized microphone is supported, etc.

[0190] Among them, the desk lamp has the ability to support virtualized cameras, which can be understood as the camera of the desk lamp allowing other electronic devices (such as tablets) to call it, that is, it can be understood as the camera of the desk lamp allowing it to be used as a virtual camera of other electronic devices.

[0191] After the basic component obtains the device information and capability information of the desk lamp, if the desk lamp has the ability to support virtualized cameras, the basic component loads the camera component to provide the capability of virtualized cameras. At this point, the basic component can prepare to establish a negotiation channel, waiting to establish a negotiation channel with the tablet to negotiate network connection related information (including but not limited to IP addresses and ports, etc.). Among them, when the basic component prepares to establish a negotiation channel, it creates a session service (SessionServer) and sends the session name (Session Name) of the session service to the device interconnection service, waiting for the transmission management service on the tablet side to establish a negotiation channel with the device interconnection service on the desk lamp side.

[0192] 1. Device discovery phase

[0193] S1.1, in response to a user operation, the tablet's education APP sends a device discovery instruction to the device management service.

[0194] The user operation may be an operation in which the user clicks on a function option in the education APP that requires calling a virtual camera. For example, the user operation may be an operation in which the user clicks on a reading function, a word search function, an assignment function, or a photo taking function in the education APP.

[0195] The tablet's education app receives a user action and, in response, sends a device discovery instruction to the tablet's device management service. The device discovery instruction is used to instruct a search for IoT devices that can establish a connection with the tablet. For example, the device discovery instruction may include, but is not limited to, the instruction type and the type of device to be discovered. In this embodiment, the device discovery instruction is specifically used to search for a desk lamp that can establish a connection with the tablet.

[0196] S1.2, the device management service in the tablet calls the authentication service to authenticate the education app and obtains the authentication result of the education app.

[0197] After receiving the device discovery command, the device management service can obtain the name (or identifier) ​​of the education app based on the existing Android system mechanism and authenticate the education app based on the name of the education app. The device management service can then call the authentication service to authenticate the education app and obtain the authentication result for the education app.

[0198] In this embodiment, the tablet application framework layer also provides a device management API corresponding to the device management service, and a hardware virtualization API corresponding to the hardware virtualization service. To implement the technical solution provided by this embodiment, the education app needs to register on the relevant platform (such as the platform provided by the tablet manufacturer), adapt to the framework of the device management service, hardware virtualization service, and transmission management service, and apply for permissions for the device management API and hardware virtualization API.

[0199] Exemplarily, the authentication service accesses the authentication server to authenticate the education APP through the authentication server, including but not limited to verifying whether it is registered on the relevant platform, whether it is adapted to the relevant framework, and whether relevant API permissions have been applied for.

[0200] As another example, the authentication service can authenticate the education APP based on the local whitelist.

[0201] The authentication service obtains the authentication result of the education APP (authentication success or authentication failure) and sends the authentication result to the device management service.

[0202] S1.3, the device management service in the tablet sends a device search instruction to the transmission management service when the education APP is successfully authenticated.

[0203] If the education app successfully authenticates, the device management service sends a device search instruction to the transmission management service. The device search instruction may include, but is not limited to, the instruction type, the device type to be searched, and the search method. Exemplary search methods include, but are not limited to, near-field device scanning and obtaining device information from a cloud server. In this implementation, the device type to be searched is a desk lamp.

[0204] S1.4, the transmission management service in the tablet obtains a list of near- and far-field devices according to the device search instruction, and sends the list of near- and far-field devices to the device management service.

[0205] The far-field and near-field device lists include both the far-field device list and the near-field device list. The far-field devices listed in the far-field device list are registered devices obtained from the cloud server, while the near-field devices listed in the near-field device list are devices scanned via near-field communication. Device information in both the far-field and near-field device lists includes, but is not limited to, the device name, device ID, and device type.

[0206] When the transmission management service receives a device search instruction, it performs relevant device search operations based on the device type to be searched and the search method carried in the device search instruction, such as performing a near-field device scanning operation and obtaining relevant device information in the cloud server, obtaining a far-field device list and a near-field device list, and sending the far-field device list and the near-field device list to the device management service.

[0207] S1.5, the device management service in the tablet filters devices based on the far-field and near-field device lists, and reports the filtered device information to the education app.

[0208] The device management service filters devices based on the far-field device list and the near-field device list, determines the information of desk lamps that can be linked with the tablet, and sends it to the education APP. Among them, the device management service can perform an intersection operation on the far-field device list and the near-field device list, filtering out desk lamps that only exist in the far-field device list or only exist in the near-field device list, and treating desk lamps that exist in both the far-field device list and the near-field device list as desk lamp information that can be linked with the tablet. In this way, the device management service can filter out desk lamps that are not registered in the cloud server, and can also filter out desk lamps that cannot communicate with the tablet in the near field.

[0209] In another optional embodiment, regardless of whether the tablet and the desk lamp are in the same local area network, the tablet's transmission management service can obtain a communication device list and a registered device list based on a device search instruction. The devices included in the communication device list are devices scanned via near-field communication or far-field communication, and the devices included in the registered device list are registered devices obtained from the cloud server. The near-field devices included in the near-field device list are devices scanned via near-field communication. In the communication device list and the registered device list, device information includes, but is not limited to, device name, device ID, device type, etc.

[0210] The transmission management service in the tablet sends the communication device list and the registered device list to the device management service. The device management service filters the devices based on the communication device list and the registered device list, and reports the filtered device information to the education APP. Among them, the device management service can perform an intersection operation on the communication device list and the registered device list, filtering out desk lamps that only exist in the communication device list or only exist in the registered device list, and using desk lamps that exist in both the communication device list and the registered device list as desk lamp information that can be linked with the tablet. In this way, the device management service can filter out desk lamps that are not registered in the cloud server, and can also filter out desk lamps that cannot perform near-field communication with the tablet.

[0211] 2. Virtual camera enabling stage

[0212] S2.1, the tablet's education APP determines the desk lamp to be linked.

[0213] The device management service can filter out one or more desk lamps that can be linked to the tablet. If there is only one desk lamp, the education app will default it to the lamp to be linked. If there are multiple desk lamps, the education app will display a list of lamps to be linked to the user for selection. In response to the user's selection, the education app will select the lamp selected by the user as the lamp to be linked.

[0214] It should be pointed out that the step of the education APP determining the desk lamp to be linked can also be divided into the device discovery stage, which is not limited in this embodiment.

[0215] In step S2.2, the tablet's educational app performs device verification and device connection on the desk lamp, and obtains the ability of the desk lamp to support a virtualized camera.

[0216] S2.3, the tablet's education app sends a virtual camera enablement request to the hardware virtualization service.

[0217] After the tablet's educational app detects that the desk lamp supports virtualized cameras, it sends a virtual camera enablement request to the hardware virtualization service. This request instructs the hardware virtualization service to register a virtual camera with the virtual camera HAL. The request may include, but is not limited to, the request type, device name, device identifier, device type, and virtual camera identifier.

[0218] S2.4, the hardware virtualization service of the tablet registers the virtual camera with the virtual camera HAL.

[0219] After receiving the virtual camera enable request, the hardware virtualization service registers the corresponding virtual camera with the virtual camera HAL according to the virtual camera enable request.

[0220] S2.5, after the virtual camera registration is completed, the tablet's virtual camera HAL sends a virtual camera enabling success indication to the education APP.

[0221] The following combination Figure 4b The module interaction diagram shown in the figure provides a detailed explanation of the process of virtual camera enabling stage. Figure 4b The process of the virtual camera enabling phase mainly includes the device verification sub-phase (S301-S309), the device connection sub-phase (S310-S321), the device service capability request sub-phase (S322-S325) and the virtual camera enabling sub-phase (S326-S331).

[0222] Reference Figure 4b The process of the virtual camera enabling stage includes the following steps:

[0223] S301: The education APP in the tablet sends a virtual camera enable instruction to the hardware virtualization API.

[0224] The virtual camera enabling instruction is used to instruct to enable the virtual camera, and the virtual camera enabling instruction may include but is not limited to an instruction type, a device name, a device identifier, and a device type.

[0225] S302 : After receiving the virtual camera enable instruction, the hardware virtualization API in the tablet sends a device verification instruction to the interface scheduling module of the hardware virtualization service.

[0226] The device verification instruction is used to instruct to verify the device information carried in the virtual camera enable instruction. The device verification instruction may include but is not limited to the instruction type, device name, device identifier, and device type.

[0227] S303: The interface scheduling module of the hardware virtualization service in the tablet sends an APP authentication instruction to the authority management module of the hardware virtualization service.

[0228] After receiving the device verification instruction, the hardware virtualization service's interface scheduling module first sends an app authentication instruction to the hardware virtualization service's permission management module to authenticate the app that initiated the virtual camera enable instruction. The app authentication instruction may include, but is not limited to, the name of the app.

[0229] S304: The rights management module of the hardware virtualization service in the tablet performs APP authentication on the education APP.

[0230] For example, the rights management module can access the authentication server to authenticate the education app through the authentication server, including but not limited to verifying whether it is registered on the relevant platform, whether it is adapted to the relevant framework, and whether it has applied for relevant API permissions. The rights management module can access the authentication server through the authentication service, which is not limited in this embodiment.

[0231] S305 , when the rights management module of the hardware virtualization service in the tablet successfully authenticates the education APP, it sends an authentication success indication to the interface scheduling module.

[0232] After the permission management module obtains the authentication result of the education APP, if the education APP authentication is successful, it sends an authentication success indication to the interface scheduling module; if the education APP authentication fails, it sends an authentication failure indication to the hardware virtualization API, so that the hardware virtualization API returns an indication information that the APP has no permission to the education APP based on the authentication failure indication.

[0233] S306 , when the interface scheduling module of the hardware virtualization service in the tablet determines that the education APP authentication is successful, it sends a device verification instruction to the device management module.

[0234] After receiving the successful authentication indication and confirming that the education app has been successfully authenticated, the hardware virtualization service's interface scheduling module sends a device verification instruction to the device management module. The device verification instruction is used to verify the status of the linked device, specifically, in this embodiment, the status of the linked desk lamp. Exemplarily, the device verification instruction may include, but is not limited to, the instruction type, device name, device identifier, and device type.

[0235] S307 , the device management module of the hardware virtualization service in the tablet sends a device information query instruction to the device introduction module of the device management service.

[0236] The device profile module of the device management service stores the information of the currently online devices.

[0237] After receiving the device verification instruction, the device management module of the hardware virtualization service sends a device information query instruction to the device introduction module of the device management service. The device information query instruction may include but is not limited to the device name, device identifier, and device type.

[0238] S308 , the device introduction module of the device management service in the tablet returns the device information to the device management module of the hardware virtualization service.

[0239] If the device profile module of the device management service finds the corresponding device according to the device information query instruction, it returns the device information to the device management module of the hardware virtualization service. The returned device information may include but is not limited to the device name, device identifier, device type and online status.

[0240] If the device profile module of the device management service fails to find the corresponding device according to the device information query instruction, it returns a null value to the device management module of the hardware virtualization service to indicate that the corresponding device was not found. At this time, the device management module of the hardware virtualization service can send a device verification failure indication to the hardware virtualization API, so that the hardware virtualization API returns an indication of device verification failure to the education app based on the device verification failure indication.

[0241] S309 , after receiving the device information, the device management module of the hardware virtualization service in the tablet sends a device verification success indication to the hardware virtualization API.

[0242] The device management module of the hardware virtualization service receives the device information returned by the device introduction module of the device management service, and then sends a device verification success indication to the hardware virtualization API to indicate that the table lamp to be linked has been successfully verified.

[0243] S310: The hardware virtualization API in the tablet sends a device connection request to the device management module of the hardware virtualization service.

[0244] After confirming that the desk lamp to be linked has been successfully verified, the hardware virtualization API sends a device connection request to the device management module of the hardware virtualization service. The device connection request is used to instruct the establishment of a network connection with the device to be linked. In this embodiment, it specifically instructs the establishment of a network connection with the desk lamp to be linked. The device connection request may include, but is not limited to, the request type, device name, device identifier, and device type.

[0245] S311 , after receiving the device connection request, the device management module of the hardware virtualization service performs a negotiation channel establishment preparation operation and sends a negotiation channel opening request to the transmission management service.

[0246] After receiving the device connection request, the device management module of the hardware virtualization service prepares a negotiation channel. When preparing the negotiation channel, the device management module creates a Session Server and sends the Session Name of the session service to the transmission management service. After preparing the negotiation channel, a negotiation channel opening request is sent to the transmission management service. The negotiation channel opening request is used to indicate the establishment of a negotiation channel. The negotiation channel opening request may include, but is not limited to, the peer device identifier (i.e., the lamp identifier) ​​and the Session Name.

[0247] In this embodiment, the negotiation channel opening request is initiated by the tablet side, that is, the tablet needs to establish a connection with the desk lamp. In this case, the desk lamp can be understood as the server, and the tablet can be understood as the client that needs to access the server.

[0248] S312: The transmission management service in the tablet establishes a negotiation channel with the device interconnection service in the desk lamp.

[0249] After receiving the negotiation channel opening request, the transmission management service interacts with the device interconnection service in the desk lamp based on the Session Name to establish a negotiation channel. Establishing the negotiation channel can specifically involve creating a session and determining a session identifier.

[0250] S313, the device interconnection service in the desk lamp sends an indication of successful negotiation channel establishment to the camera component in the hardware abstraction service.

[0251] After the negotiation channel is established, the device interconnection service in the desk lamp sends a successful negotiation channel establishment indication to the camera component in the hardware abstraction service, indicating that the negotiation channel is established and that there is a device that needs to be connected. The successful negotiation channel establishment indication may include, but is not limited to, device information (i.e., tablet device information) and a session identifier.

[0252] S314 , the transmission management service in the tablet sends an indication of successful establishment of a negotiation channel to the device management module of the hardware virtualization service.

[0253] After the negotiation channel is established, the transport management service in the tablet sends a successful negotiation channel establishment indication to the device management module of the hardware virtualization service, indicating that the negotiation channel is established and that there is a device that needs to be connected. The successful negotiation channel establishment indication may include, but is not limited to, device information (i.e., desk lamp device information) and a session identifier.

[0254] This embodiment does not limit the execution order of S313 and S314.

[0255] S315 , the device management module of the hardware virtualization service in the tablet sends a device information negotiation request to the camera component of the hardware abstraction service in the desk lamp based on the negotiation channel.

[0256] The device negotiation request may include but is not limited to device information (such as device name, device identification, device type, etc.) and a control channel connection request.

[0257] S316 , after receiving the device negotiation request, the camera component of the hardware abstraction service in the desk lamp prepares a control channel and returns device negotiation information to the device management module of the hardware virtualization service in the tablet.

[0258] After the camera component of the hardware abstraction service in the desk lamp receives the device negotiation request, it parses the device negotiation request to obtain the device information of the other end and records it. It then determines an IP address and port to be monitored based on the control channel connection request, adds the IP address and port to the device negotiation information, and returns it to the device management module of the hardware virtualization service in the tablet.

[0259] It should be noted that the device negotiation request and the device negotiation information are transmitted based on the established negotiation channel.

[0260] S317 , after receiving the returned device negotiation information, the device management module of the hardware virtualization service in the tablet closes the negotiation channel.

[0261] Closing the negotiation channel may specifically be closing a session. When the device management module of the hardware virtualization service in the tablet receives the returned device negotiation information, the session ends and the device management module of the hardware virtualization service may close the corresponding session according to the session identifier.

[0262] S318 , the device management module of the hardware virtualization service in the tablet sends a control channel opening request to the transmission management service.

[0263] A control channel open request is used to establish a network communication connection with the lamp. The control channel open request may include, but is not limited to, the communication protocol, source IP address, source port number, destination IP address, and destination port number. The destination IP address and destination port number are the IP address and port number monitored by the camera component of the hardware abstraction service in the lamp.

[0264] S319, the transmission management service in the tablet connects to the control channel of the camera component of the hardware abstraction service in the desk lamp, and sends a control channel successful connection indication to the device management module of the hardware virtualization service in the tablet.

[0265] The tablet's transmission management service receives the control channel open request and, based on the information in the control channel open request, establishes a control channel connection with the lamp, effectively establishing a network communication connection between the tablet and the lamp. Furthermore, the tablet's hardware virtualization service's device management module and the lamp's hardware abstraction service's camera component can communicate over this control channel.

[0266] After the control channel is successfully established, the transport management service in the tablet sends a control channel successful connection indication to the device management module of the hardware virtualization service in the tablet. The control channel successful connection indication may include but is not limited to a connection success flag and control channel related information.

[0267] S321 , the device management module of the hardware virtualization service in the tablet sends a device connection success indication to the hardware virtualization API.

[0268] The device connection success indication may include but is not limited to a connection success identifier and connected device information.

[0269] S322: The hardware virtualization API in the tablet sends a device capability request to the device management module of the hardware virtualization service.

[0270] After receiving the successful device connection indication, the hardware virtualization API sends a device capability request to the device management module of the hardware virtualization service. This device capability request can be used to request virtualization capability information for the peer device (i.e., the desk lamp). For example, virtualized device capability information includes, but is not limited to, support for virtualized cameras and microphones.

[0271] S323, the device management module of the hardware virtualization service in the tablet sends a device capability request to the camera component of the hardware abstraction service in the desk lamp in the control channel.

[0272] S324 , the camera component of the hardware abstraction service in the desk lamp returns device capability information to the device management module of the hardware virtualization service in the tablet in the control channel.

[0273] In this embodiment, the returned device capability information of the desk lamp may at least include the capability of supporting a virtualized camera and the camera identifier of the desk lamp.

[0274] S325 , the device management module of the hardware virtualization service in the tablet sends the device capability information to the hardware virtualization API.

[0275] The device management module of the hardware virtualization service in the tablet sends the received device capability information to the hardware virtualization API, so that the hardware virtualization API knows whether the desk lamp has the capability to support the virtualized camera.

[0276] S326: The hardware virtualization API in the tablet sends a virtual camera enablement request to the device management module of the hardware virtualization service.

[0277] The hardware virtualization API in the tablet learns that the desk lamp supports virtualized cameras and sends a virtual camera enablement request to the device management module of the hardware virtualization service. The virtual camera enablement request may include, but is not limited to, the request type and the camera ID of the desk lamp.

[0278] S327 , the device management module of the hardware virtualization service in the tablet registers the virtual camera in the virtual camera HAL.

[0279] After receiving the virtual camera enable request, the device management module of the hardware virtualization service sends a virtual camera registration request to the virtual camera HAL. The virtual camera registration request may include, but is not limited to, the request type and the camera identifier of the desk lamp. After receiving the virtual camera registration request, the virtual camera HAL registers a virtual camera driver for the desk lamp's camera in the virtual camera HAL, assigns a camera ID (i.e., a virtual camera ID) to the desk lamp's camera, and registers the camera ID with the system. Thus, a mapping relationship between the desk lamp's camera and the virtual camera is established in the virtual camera HAL.

[0280] S328, the device management module of the hardware virtualization service in the tablet sends a service status update indication to the camera component of the hardware abstraction service in the desk lamp.

[0281] The service status update indication is used to instruct the camera component of the hardware abstract service in the desk lamp to update its virtualized service status. The virtualized service status may include an occupied state and an unoccupied state, or may include a registered state and an unregistered state. Exemplarily, the service status update indication may include but is not limited to the device information of the peer device (i.e., the desk lamp), the hardware identifier (such as the desk lamp camera identifier), and the virtualized service status corresponding to the hardware identifier.

[0282] S329, the camera component of the hardware abstract service in the desk lamp updates the service status according to the service status update instruction.

[0283] When the service status update indication indicates that the virtualized service status corresponding to the desk lamp camera is occupied (or registered), the camera component updates the virtualized service status corresponding to its desk lamp camera to occupied (or registered).

[0284] S330: The device management module of the hardware virtualization service in the tablet sends a virtual camera enabling success indication to the hardware virtualization API.

[0285] Among them, the virtual camera enabling success indication may include but is not limited to an enabling success identifier (or a virtualization success identifier), the camera identifier of the desk lamp, and the camera ID corresponding to the virtual camera (or the camera ID corresponding to the desk lamp camera).

[0286] This embodiment does not limit the execution order of S328 and S330.

[0287] S331: The hardware virtualization API in the tablet sends a virtual camera enabling success indication to the education app.

[0288] 3. Virtual camera preview access stage

[0289] S3.1, the hardware virtualization API in the tablet sends a virtual camera access instruction to the camera service.

[0290] A virtual camera access instruction refers to an instruction for invoking a virtual camera. The virtual camera access instruction may include, but is not limited to, an instruction type, a virtual camera ID, and camera configuration parameters, including, but not limited to, camera resolution and acquisition frame rate.

[0291] S3.2, the camera service in the tablet sends an image preview request to the virtual camera HAL according to the virtual camera access instruction.

[0292] After receiving the virtual camera access instruction, the camera service generates a corresponding image preview request based on the virtual camera ID and sends it to the virtual camera HAL. The image preview request is used to request a preview of the image data stream. For example, the image preview request may include, but is not limited to, a request identifier, a virtual camera ID, and camera configuration parameters.

[0293] S3.3, the virtual camera HAL in the tablet sends an image preview request to the hardware virtualization service.

[0294] After receiving the image preview request, the virtual camera HAL determines the matching virtualized hardware identifier based on the virtual camera ID carried in the image request. In this embodiment, the virtual camera HAL determines the linked desk lamp camera based on the virtual camera ID and the mapping relationship between the virtual camera ID and the desk lamp camera, and generates a corresponding image request based on the determined virtualized hardware identifier and sends it to the hardware virtualization service. Exemplarily, the image preview request may include but is not limited to the request identifier, device information (i.e., desk lamp information), virtualized hardware identifier (i.e., desk lamp camera identifier), and camera configuration parameters.

[0295] S3.4, the hardware virtualization service in the tablet sends an image preview request to the transmission management service.

[0296] The hardware virtualization service sends an image preview request to the transmission management service. The image preview request may include, but is not limited to, a request identifier, device information (i.e., desk lamp information), a virtualized hardware identifier (i.e., desk lamp camera identifier), and camera configuration parameters.

[0297] When the tablet's hardware virtualization service sends an image preview request to the transmission management service, if it finds that a data channel with the desk lamp is not established, it generates a data channel establishment request and sends it to the transmission management service. This data channel establishment request is used to instruct data transmission between the desk lamp and the tablet. The data channel establishment request may include, but is not limited to, a session identifier, connection information, and data encoding and decoding mode.

[0298] The tablet's transmission management service receives the data channel establishment request and, based on the information in the data channel establishment request, establishes a data channel connection with the lamp. This establishes a data channel between the tablet and the lamp. The tablet's transmission management service and the lamp's device interconnection service can then use this data channel to transmit various data, including but not limited to image data.

[0299] After the data channel is successfully established, the tablet's transport management service sends a successful data channel connection indication to the tablet's hardware virtualization service, and the lamp's device interconnection service sends a successful data channel connection indication to the camera component in the hardware abstraction service. The successful data channel connection indication may include, but is not limited to, a successful connection flag and data channel-related information.

[0300] S3.5: The transfer management service in the tablet transmits an image preview request to the device interconnection service in the desk lamp.

[0301] The transmission management service in the tablet determines the corresponding control channel according to the device information carried in the image preview request, and transmits the image preview request to the device interconnection service of the desk lamp through the control channel.

[0302] S3.6: The device interconnection service in the desk lamp sends an image preview request to the camera driver.

[0303] After receiving the image preview request, the device interconnection service in the desk lamp determines the corresponding hardware driver (in this embodiment, the camera driver) according to the virtualized hardware identifier, and sends the corresponding image preview request to the camera driver.

[0304] In S3.7, the camera driver in the desk lamp drives the camera to capture images and transmits the preview image data to the hardware virtualization service of the tablet through the data channel.

[0305] The camera driver turns on the camera and drives it to capture an image using the camera configuration parameters carried in the image preview request. This generates a preview image data stream and sends it to the device interconnection service via the hardware abstraction service. The device interconnection service then continuously transmits the preview image data stream to the tablet's hardware virtualization service via the data channel. The details of the preview image data stream's packetization, assembly, and encoding / decoding are not detailed here.

[0306] S3.8, the hardware virtualization service in the tablet sends the preview image data to the virtual camera HAL.

[0307] The hardware virtualization service continuously receives the preview image data stream and sends the preview image data stream to the virtual camera HAL.

[0308] S3.9, the virtual camera HAL in the tablet sends the preview image data to the camera service.

[0309] At this time, the virtual camera HAL continues to obtain the preview image data collected by the desk lamp camera and continues to send the preview image data to the camera service.

[0310] S3.10, the camera service in the tablet sends the preview image data to the education app.

[0311] S3.11, the educational app on the tablet displays a preview image.

[0312] After the education app receives the preview image data stream through the camera service, it can display the preview image in the corresponding interface.

[0313] 4. Virtual camera photo taking stage

[0314] S4.1, in response to the received user operation, the education APP in the tablet sends a photo-taking request to the hardware virtualization service.

[0315] For example, a user operation can be clicking a photo option. In response to the received user operation, the education app in the tablet sends a photo request to the hardware virtualization service. The photo request may include, but is not limited to, the captured image sequence number, device information (i.e., desk lamp information), virtualized hardware identifier (i.e., desk lamp camera identifier), and camera configuration parameters. The camera configuration parameters include, but are not limited to, image resolution.

[0316] The photo request may also carry a task identifier to ensure orderly management of multiple photo tasks.

[0317] S4.2, the hardware virtualization service in the tablet sends a photo-taking request to the transmission management service.

[0318] S4.3, the transmission management service in the tablet transmits the photo request to the device interconnection service of the desk lamp.

[0319] The transmission management service in the tablet determines the corresponding control channel according to the device information carried in the image preview request, and transmits the image preview request to the device interconnection service of the desk lamp through the control channel.

[0320] S4.4: The device interconnection service in the desk lamp sends a photo request to the camera driver.

[0321] After receiving the image preview request, the device interconnection service in the desk lamp determines the corresponding hardware driver (in this embodiment, the camera driver) according to the virtualized hardware identifier, and sends the corresponding photo request to the camera driver.

[0322] In S4.5, the camera driver in the desk lamp drives the camera to capture images and transmits the captured image data to the hardware virtualization service of the tablet through a data channel.

[0323] The camera driver drives the camera to capture an image using the camera configuration parameters carried in the photo request, obtains the captured image data, and sends the captured image data to the device interconnection service through the hardware abstraction service. The device interconnection service then continuously transmits the captured image data to the tablet's hardware virtualization service through the data channel. The image data packetization, grouping, encoding, and decoding processes are not detailed here.

[0324] S4.6, the hardware virtualization service in the tablet sends the captured image data to the education app.

[0325] S4.7, the tablet's educational app displays the captured image.

[0326] After the educational APP receives the captured image through the hardware virtualization service, it can display the captured image in the corresponding interface.

[0327] In this embodiment, the virtual camera preview access phase is implemented based on Android's native camera framework, while the virtual camera capture phase is implemented based on a proprietary virtualized camera framework. This shortens the processing path involved in the virtual camera capture phase and reduces capture latency. Furthermore, because image preview is still implemented based on Android's native camera framework, educational apps require fewer modifications to adapt to the technical solutions provided by this embodiment.

[0328] It should be noted that the stage divisions in the above process are merely illustrative and are not intended to limit this embodiment of the present application. Furthermore, after the virtual camera preview access stage is executed and the preview image is displayed on the tablet, the real-time preview image display process and the virtual camera photo capture process can be executed simultaneously. Any details not fully explained in the above process can be referred to existing technologies and will not be elaborated upon here.

[0329] Figure 4a The communication architecture of the collaborative work system is given in the paper. This communication architecture is used to complete the management of the virtual camera (i.e., the desk lamp camera) by the tablet, the control command interaction between the tablet and the desk lamp, and the return and processing of image data.

[0330] It should be noted that instructions and requests transmitted across devices (i.e., between the tablet and the desk lamp) need to be encapsulated based on communication protocols and parameter sequences, which will not be described in detail in this embodiment. The hardware virtualization service in the tablet can also manage the lifecycle of preview image streams and captured images by dynamically allocating and destroying memory.

[0331] In addition, it should be pointed out that before executing the collaborative working method provided in this embodiment, the education APP needs to be bound to the desk lamp and the desk lamp needs to be registered in the cloud server.

[0332] An embodiment of the present application provides a framework solution for an Android system device to use the camera of an external device to take photos. This solution can not only be applied in educational scenarios, but also is applicable to other devices equipped with cameras. These devices can share their camera capabilities with Android system devices such as mobile phones and tablets to achieve interconnection and interoperability between Android system devices and these devices.

[0333] Figure 5a-5b An example application scenario is shown. Figure 5a As shown in (1), the tablet displays an interface 401, which displays multiple application icons. The user clicks on the education application icon 4011. In response to the received user operation, the tablet opens the education application and displays the education application interface. Figure 5a As shown in (2). Figure 5a As shown in (2), the tablet displays an educational application interface 402, on which various functional options of the educational application are displayed, including but not limited to a word lookup function, a point reading function, an assignment function, a photo taking function, etc. In order to use the photo taking function, the user clicks on the photo taking function option 4021. In response to the user operation, the tablet executes the processes of the device discovery phase, the virtual camera service enabling phase, and the virtual camera preview access phase.

[0334] Among them, in the device discovery phase, if the number of desk lamps that can be linked with the tablet is filtered by the tablet's device management service is one, the tablet automatically executes the virtual camera service enabling phase and the virtual camera preview access phase process, and displays, for example Figure 5b The interface shown in (1). In the device discovery phase, if the number of desk lamps that can be linked to the tablet is filtered by the tablet's device management service is multiple, the tablet displays a desk lamp selection interface. Exemplarily, a list of desk lamps to be linked is displayed on the desk lamp selection interface, and the user can perform a selection operation. In response to the user's selection operation, the educational application determines a desk lamp to be linked, and continues to execute the virtual camera service enabling phase and the virtual camera preview access phase process to display, for example Figure 5b The interface shown in (1).

[0335] like Figure 5b As shown in (1), the interface 403 displays an image preview window 4031 and a photo option 4032. The image preview window 4031 displays a preview image captured by the lamp camera in real time. At this time, if the user clicks the photo option 4032, in response to the user operation, the tablet executes the process of the virtual camera photo stage, and displays, for example Figure 5b The interface shown in (2) is shown in the figure. Figure 5b(2) The interface 404 shown displays the image captured by the lamp camera in the image preview window 4041. At this time, if the user clicks the confirmation option 4041, the tablet responds to the user operation, saves the captured image, and continues to display, for example Figure 5b If the user clicks the cancel option 4042, the tablet responds to the user operation and can display, for example, Figure 5b The preview interface shown in (1) is shown in the figure.

[0336] It should be pointed out that if Figure 5b (2) The interface shown is merely an illustrative example. The image captured by the desk lamp camera may not be displayed in the image preview window 4041, but may be displayed in other areas of the interface. The image preview window 4041 continues to display the preview image captured in real time by the desk lamp camera. This application does not limit this.

[0337] The collaborative working method provided in this embodiment mainly describes a low-cost technical solution for implementing online education functions based on a combination of a tablet device and a desk lamp device. The following describes the technical solution provided in this embodiment in combination with several different functions involved in online education.

[0338] Scene 1

[0339] Reference Figure 1a The following application scenario illustrates the technical solution using the word lookup function as an example. When students encounter an unfamiliar word, they can point their finger at the word's location. The lamp's camera then captures the image, which the tablet then uses to identify the word. After the online word lookup is complete, the tablet provides feedback on the meaning of the word, such as displaying and announcing the word's meaning on the screen.

[0340] like Figure 6a The following is a schematic diagram of the interaction between modules. Figure 6a , the embodiment of the present application provides a method process for collaborative operation of a tablet and a desk lamp, specifically including:

[0341] S501, in response to the user clicking on the word search function, the tablet and the desk lamp execute the processes of the device discovery phase, the virtual camera enabling phase and the virtual camera preview access phase, and the tablet displays the preview interface.

[0342] Among them, the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0343] It's important to note that during the virtual camera preview access phase, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service, along with camera configuration parameters. These parameters may include, but are not limited to, image resolution and frame rate. The desk lamp camera configures itself according to the received configuration parameters and captures preview image data at the corresponding resolution and frame rate.

[0344] In this scenario, the tablet needs to accurately recognize the preview image to determine the text content pointed to by the user. Therefore, this scenario requires high image quality for the preview image. For example, the image resolution can be set to 1080P. This ensures a high success rate for word lookup.

[0345] S502: The education APP in the tablet performs finger recognition on the preview image.

[0346] For example, the educational APP may perform finger recognition on each preview image received, or may periodically perform finger recognition on the latest preview image received, which is not limited in this embodiment.

[0347] For example, an image recognition algorithm may be integrated into an education APP to implement image recognition operations, and the education APP may also call an image recognition service to perform image recognition operations, which is not limited in this embodiment.

[0348] Regarding the image recognition algorithm, reference may be made to existing technologies, and this embodiment will not be described in detail here.

[0349] S503 , in response to the user pointing to a word with a finger, the education APP in the tablet recognizes the user's finger in the preview image.

[0350] When a user points his finger at a word in a book, the desk lamp camera can capture a preview image of the word pointed at by the finger, and then the education app in the tablet can recognize the user's finger in the preview image.

[0351] Among them, when the educational APP performs finger recognition on the preview image, if a finger is recognized, the position information of the finger in the preview image, such as coordinate information, can be obtained.

[0352] S504 , the education APP in the tablet determines a ROI (region of interest) image according to the position of the finger in the preview image.

[0353] After the educational app recognizes the user's finger in the preview image, it can determine the ROI image based on the finger's position in the preview image. Specifically, the educational app can determine the ROI information based on the finger's coordinate information in the preview image. The ROI information includes, but is not limited to, the center point coordinates and the region range (such as width and height information). Furthermore, the educational app can crop the ROI image from the preview image based on the ROI information.

[0354] S505: The education APP in the tablet accurately recognizes the ROI image and determines the new words to be explained.

[0355] For example, an image recognition algorithm may be integrated into the education APP to accurately identify the ROI image. The education APP may also call an image recognition service to accurately identify the ROI image to determine the new words to be explained. This embodiment does not limit this.

[0356] Regarding the image recognition algorithm, reference may be made to existing technologies, and this embodiment will not be described in detail here.

[0357] S506, the education APP in the tablet searches for the new word to be explained and displays the meaning of the new word to be explained.

[0358] After determining the new word to be explained, the education app can perform an online word search or a word search in a database to obtain the definition of the new word to be explained. Furthermore, the education app can display the definition of the new word to be explained for the user to review. Furthermore, the education app can also read the displayed definition of the new word aloud, which is not limited in this embodiment.

[0359] Similarly, users can also use a pointing tool (or indicator tool) such as a reading pen to perform a pointing operation, which is not limited in this embodiment. Accordingly, the education app performs a pointing tool recognition on the preview image to determine whether the user intends to look up a word, and determines the ROI image based on the position information of the pointing tool in the preview image.

[0360] Similarly, users can use a finger or a pointing tool like a reading pen to point to an image in a book. Accordingly, the education app determines the ROI image based on the position of the finger or reading pen, performs image content recognition on the ROI image, displays the corresponding interpretation of the image, and can also read the displayed interpretation aloud. This scenario is not further described in this embodiment.

[0361] Figure 1a 、 Figure 7a-7b An example application scenario is shown. Figure 7aAs shown in (1), the tablet displays an education APP interface 701, which displays various function options of the education application, including but not limited to a word search function, a point reading function, a homework function, a photo taking function, etc. The user clicks on the word search function option 7011. In response to the user operation, the tablet executes the device discovery phase, the virtual camera service enabling phase, and the virtual camera preview access phase.

[0362] Among them, in the device discovery phase, if the number of desk lamps that can be linked with the tablet is filtered by the tablet's device management service is one, the tablet automatically executes the virtual camera service enabling phase and the virtual camera preview access phase process, and displays, for example Figure 7a The interface shown in (2). In the device discovery phase, if the number of desk lamps that can be linked to the tablet is filtered by the tablet's device management service is multiple, the tablet displays a desk lamp selection interface. Exemplarily, a list of desk lamps to be linked is displayed on the desk lamp selection interface, and the user can perform a selection operation. In response to the user's selection operation, the educational application determines a desk lamp to be linked, and continues to execute the virtual camera service enabling phase and the virtual camera preview access phase process to display, for example Figure 7a The interface shown in (2).

[0363] like Figure 7a As shown in (2), the interface 702 displays an image preview window 7021 and a diagram 7022 for the word search function operation. In the image preview window 7021, a preview image captured in real time by the lamp camera is displayed. The user can refer to the diagram 7022 for the word search function operation to perform a word pointing or image pointing operation to trigger the word search function. The education APP performs word pointing or image pointing recognition based on the preview image. Continue to refer to Figure 7a In (2), when the user points to a word in the book, the desk lamp camera captures a preview image of the finger pointing to the word, and the preview image of the finger pointing to the word is displayed in the image preview window 7021. Then, the education APP can recognize the user's finger in the preview image and determine the position information of the finger in the preview image, such as coordinate information. The education APP determines the ROI image based on the position of the finger in the preview image, and accurately recognizes the ROI image to determine the new word to be explained. After the education APP finds the meaning of the new word, the corresponding new word meaning is displayed on the interface, which can be referred to Figure 7b shown.

[0364] However, in order to ensure the success rate of word lookup in the above process, the desk lamp camera needs to continuously return a high-resolution (e.g., 1080p) preview image stream, which places high bandwidth requirements on 4-8Mbps. This also increases the hardware chip requirements of the desk lamp, which in turn increases its cost.

[0365] In order to realize the solution of the tablet and the desk lamp cooperating to realize the word search function and reduce the hardware cost of the desk lamp, this embodiment also provides a technical solution. Among them, since the finger recognition or word pointing tool recognition and position recognition in the image do not require high image resolution, the desk lamp camera is used to continuously return a low-resolution (for example, 480p) preview image stream during the virtual camera preview access stage. If the educational APP recognizes the user's word pointing or picture pointing operation based on the preview image, the desk lamp is triggered to take a high-resolution (for example, 1080p) image, so that the educational APP can accurately identify and determine the new words or pictures to be explained based on this high-resolution image. In this way, the use of the desk lamp camera to continuously return a low-resolution preview image stream does not require high bandwidth, only 0.5-1Mbps bandwidth is required, and more bandwidth is only required when transmitting this high-resolution image. Therefore, this technical solution not only reduces the requirements of the desk lamp on the hardware chip and reduces the cost of the desk lamp, but also ensures the success rate of word search.

[0366] like Figure 6b The following is a schematic diagram of the interaction between modules. Figure 6b , the embodiment of the present application provides a method process for collaborative operation of a tablet and a desk lamp, specifically including:

[0367] S601, in response to the user clicking on the word search function, the tablet and the desk lamp execute the processes of the device discovery phase, the virtual camera enabling phase and the virtual camera preview access phase, and the tablet displays the preview interface.

[0368] Among them, the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0369] It's important to note that during the virtual camera preview access phase, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service, along with camera configuration parameters. These parameters may include, but are not limited to, image resolution and frame rate. The desk lamp camera configures itself according to the received configuration parameters and captures preview image data at the corresponding resolution and frame rate.

[0370] In this scenario, to reduce the bandwidth occupied by the preview image stream, the desk lamp camera can capture preview images at a low resolution during the virtual camera preview access phase. For example, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service with first configuration parameters, where the first configuration parameters include a first image resolution (e.g., 480P). The desk lamp camera then configures itself according to the received configuration parameters and captures preview image data at the first image resolution and the corresponding image acquisition frame rate.

[0371] S602: The education APP in the tablet performs finger recognition on the preview image.

[0372] S603 , in response to the user pointing to a word with a finger, the education APP in the tablet recognizes the user's finger in the preview image.

[0373] S604: The education APP in the tablet determines ROI information according to the position of the finger in the preview image, and generates a photo request according to the ROI information.

[0374] The ROI information refers to information used to determine the ROI, and may include but is not limited to center point coordinates and region range (such as width and height information).

[0375] S605: The tablet transmits the photo-taking request to the desk lamp.

[0376] The photo request may include, but is not limited to, the virtual camera ID corresponding to the desk lamp camera, second configuration parameters of the desk lamp camera, and ROI information. The second configuration parameters may include, but are not limited to, a second image resolution, which is higher than the first image resolution, such as setting the second image resolution to 1080P. This allows the educational app to accurately identify the new word or image to be explained based on this high-resolution image.

[0377] S606 , the camera of the desk lamp is configured according to the second configuration parameters carried in the photo request, captures an image according to the second image resolution, and sends the captured image to the hardware abstraction service.

[0378] S607: The hardware abstraction service determines the ROI image according to the ROI information.

[0379] The hardware abstraction service can crop the ROI image from the captured image based on the ROI information.

[0380] In an optional embodiment, the photo request includes, but is not limited to, the second configuration parameters, but does not include ROI information. The tablet then transmits the photo request to the desk lamp. The lamp's camera is configured according to the second configuration parameters in the photo request, captures an image at the second image resolution, and returns the captured image to the education app on the tablet. The education app can then determine the ROI image based on the ROI information, for example, by cropping the ROI image from the captured image based on the ROI information.

[0381] S608, the desk lamp transmits the ROI image to the education APP in the tablet.

[0382] Compared with the desk lamp directly returning the high-resolution captured image to the education APP in the tablet, returning the cropped ROI image to the education APP in the tablet can also reduce the amount of data transmission and reduce bandwidth usage.

[0383] S609: The education APP in the tablet accurately recognizes the ROI image and determines the new words to be explained.

[0384] S610, the education APP on the tablet performs a word lookup operation on the new word to be explained, and displays the definition of the new word to be explained.

[0385] For any details not explained in detail in this process, please refer to the previous article and I will not repeat them here.

[0386] Similarly, users can also use pointing tools such as a reading pen to perform word pointing operations, which is not limited in this embodiment. Accordingly, the education app recognizes the pointing tool on the preview image to determine whether the user intends to look up a word, and determines the ROI information based on the position information of the pointing tool in the preview image.

[0387] Similarly, users can use a finger or a pointing tool like a reading pen to point to an image in a book. Accordingly, the education app determines the ROI information based on the position of the finger or reading pen, and then uses the ROI information to determine the ROI image in the captured image. It then performs image content recognition on the ROI, displays the corresponding interpretation of the image, and can also read the displayed image interpretation aloud. This scenario is not further described in this embodiment.

[0388] For the application scenarios of this process, please refer to Figure 1a 、 Figure 7a-7b The application scenario shown. Figure 7a In (2), when the user points to a word in the book, the desk lamp camera captures a preview image of the finger pointing to the word, and the preview image of the finger pointing to the word is displayed in the image preview window 7021. Then, the education APP can recognize the user's finger in the preview image, determine the ROI information, and generate a photo request based on the ROI information and the image high resolution, triggering the desk lamp camera to capture the image. The desk lamp camera captures the image based on the image high resolution, and the desk lamp side crops the high resolution captured image based on the ROI information to obtain the ROI image, and returns the ROI image to the education APP in the tablet. The education APP accurately recognizes the ROI image and determines the new word to be explained. After the education APP finds the meaning of the new word, the corresponding new word meaning is displayed on the interface, which can be referred to Figure 7b shown.

[0389] Scene 2

[0390] Reference Figure 1a The following application scenario illustrates the technical solution using the homework function as an example. When students need to submit their homework online, they can click "Photo" in the education app, use the desk lamp camera to capture an image of the homework, and then upload the image to the database through the education app.

[0391] like Figure 8 The following is a schematic diagram of the interaction between modules. Figure 8 , the embodiment of the present application provides a method process for collaborative operation of a tablet and a desk lamp, specifically including:

[0392] S801, in response to the user clicking on the homework function, the education APP in the tablet displays a homework submission list.

[0393] It should be noted that the homework submission list refers to a list of multiple homework submission options. One homework submission option corresponds to one homework. Figure 9a The middle interface is shown as 704. The homework options may be divided by subject or by time, which is not limited in this embodiment.

[0394] If the education app's homework function only requires submitting homework images for one course, the app won't display the homework submission list. In this case, in response to the user clicking the homework function, the tablet and desk lamp execute the device discovery phase, virtual camera enablement phase, and virtual camera preview access phase, with the tablet displaying the preview interface.

[0395] S802, in response to the user clicking the submit job option, the tablet and the desk lamp execute the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase, and the tablet displays the preview interface.

[0396] Among them, the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0397] It's important to note that during the virtual camera preview access phase, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service, along with camera configuration parameters. These parameters may include, but are not limited to, image resolution and frame rate. The desk lamp camera configures itself according to the received configuration parameters and captures preview image data at the corresponding resolution and frame rate.

[0398] In this scenario, to reduce the bandwidth occupied by the preview image stream, the desk lamp camera can capture preview images at a low resolution during the virtual camera preview access phase. For example, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service with first configuration parameters, where the first configuration parameters include a first image resolution (e.g., 480P). The desk lamp camera then configures itself according to the received configuration parameters and captures preview image data at the first image resolution and the corresponding image acquisition frame rate.

[0399] S803: In response to the user clicking the photo option, the education APP in the tablet generates a photo request.

[0400] When the user places homework or books within the collection area of ​​the desk lamp camera, the user can click the photo option to trigger the desk lamp camera to capture the homework image.

[0401] The photo request may include, but is not limited to, the virtual camera ID corresponding to the desk lamp camera and second configuration parameters of the desk lamp camera. The second configuration parameters may include, but are not limited to, a second image resolution, which is higher than the first image resolution, such as setting the second image resolution to 1080P. This allows the education app to upload high-resolution images of homework.

[0402] S804: The tablet transmits the photo-taking request to the desk lamp.

[0403] S805: The camera of the desk lamp is configured according to the second configuration parameters carried in the photo taking request, and takes an image according to the second image resolution.

[0404] S806, the desk lamp transmits the captured image to the education APP on the tablet.

[0405] S807, the educational APP on the tablet displays the captured image.

[0406] The tablet's education app receives the image of the homework captured by the desk lamp's camera and displays it. If the user is satisfied with the captured image, they can click Submit to upload it to the database. If they are not satisfied with the captured image, they can click Take Photo again to trigger the desk lamp's camera to retake the image.

[0407] S808 , in response to the user clicking the submit option, the education APP in the tablet uploads the captured image to the database.

[0408] For any details not explained in detail in this process, please refer to the previous article and I will not repeat them here.

[0409] It should be noted that the homework images mentioned above are merely illustrative examples. Users can click the photo option to trigger the desk lamp's camera to capture other images. After the desk lamp sends the captured images to the education app on the tablet, the education app can then upload the received images to the corresponding database.

[0410] Figure 1a 、 9a - Figure 9c An example application scenario is shown. Figure 9aAs shown in (1), the tablet displays an education APP interface 701, and various function options of the education application are displayed on the interface 701, including but not limited to a word search function, a point reading function, a homework function, a photo taking function, etc. The user clicks on the word search function option 7012, and in response to the user operation, the education APP in the tablet displays a homework submission list, which can be referred to Figure 9a As shown in (2). Figure 9a In the job submission list interface 704 shown in (2), multiple job submission options are displayed (such as Submit Job 1, Submit Job 2, Submit Job 3, Submit Job 4, etc.), and different job submission options correspond to different jobs. For example, if the user needs to upload a job image for Submit Job 4, the user clicks Submit Job 4 option 7042. In response to the user operation, the tablet can display the following Figure 9b The job submission interface 705 shown in (1) is shown in FIG.

[0411] Continuing to refer to (1) in 9b, in the homework submission interface 705, an image preview window 7041, a photo option 7051 and a submission option 7052 are displayed. Among them, the image preview window 7041 displays the preview image captured in real time by the desk lamp camera. When the user places homework or books, etc. in the acquisition area of ​​the desk lamp camera, the user can click the photo option 7051 to trigger the desk lamp camera to capture the homework image. In response to the user operation, the education APP generates a photo request and sends it to the desk lamp side to call the desk lamp camera to capture the homework image. Among them, the desk lamp camera captures the homework image with high resolution according to the image carried in the photo request, and returns the captured homework image to the education APP in the tablet to be displayed through the education APP. Please refer to the following example. Figure 9b Interface 706 shown in (2).

[0412] Continuing to refer to (2) in 9b, the interface 706 displays an image preview window 7041, a photo option 7051, a submit option 7052, and a job image 7061 captured by the desk lamp camera. Among them, a close option 7062 is also displayed on the job image 7061. If the user is not satisfied with the job image 7061, the user can click the close option 7062, and the job image 7061 will no longer be displayed on the interface. At this time, the user can click the photo option 7051 to trigger the desk lamp camera to re-record the job image. If the user is satisfied with the job image 7061, the user can click the submit option 7052. In response to the user operation, the tablet can display the following Figure 9cThe to-be-confirmed interface 701 is shown. In the to-be-confirmed interface 701, a homework image 7061 to be submitted and a confirmation window 7071 are displayed. If the user clicks the confirmation option 7072 in the confirmation window 7071, the education APP responds to the user operation and uploads the homework image 7061 to the database. If the user clicks the cancel option 7071 in the confirmation window 7071, the education APP responds to the user operation and may display the following information: Figure 9b The interface shown in (1) is waiting for the user to click the photo option 7051 to trigger the desk lamp camera to retake the work image.

[0413] It should be noted that, referring to step (2) in step 9b, if the user is satisfied with the homework image 7061, the user can click the submit option 7052. In response to the user operation, the education APP may no longer display the following information: Figure 9c The interface shown directly uploads the operation image 7061 to the database. This embodiment does not limit this.

[0414] Scene 3

[0415] Reference Figure 1a The following application scenario diagram illustrates the technical solution using the touch-to-read feature (also known as finger-reading) as an example. When a student wants to read aloud content from a book through an educational app, the desk lamp camera can capture the book image in real time. The educational app can then load the corresponding book content based on the book image and determine the content to be read aloud based on the student's finger position or page turning.

[0416] like Figure 10 The following is a schematic diagram of the interaction between modules. Figure 10 , the embodiment of the present application provides a method process for collaborative operation of a tablet and a desk lamp, specifically including:

[0417] S901, in response to the user clicking the point-to-read function, the tablet and the desk lamp execute the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase, and the tablet displays the preview interface.

[0418] Among them, the processes of the device discovery phase, the virtual camera enabling phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0419] It's important to note that during the virtual camera preview access phase, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service, along with camera configuration parameters. These parameters may include, but are not limited to, image resolution and frame rate. The desk lamp camera configures itself according to the received configuration parameters and captures preview image data at the corresponding resolution and frame rate.

[0420] In this scenario, to reduce the bandwidth occupied by the preview image stream, the desk lamp camera can capture preview images at a low resolution during the virtual camera preview access phase. For example, the tablet's hardware virtualization API sends a virtual camera access instruction to the camera service with first configuration parameters, where the first configuration parameters include a first image resolution (e.g., 480P). The desk lamp camera then configures itself according to the received configuration parameters and captures preview image data at the first image resolution and the corresponding image acquisition frame rate.

[0421] S902: The education APP in the tablet recognizes the preview image and determines the title of the book.

[0422] For example, the education APP may perform book information recognition for each preview image received, or may periodically perform book information recognition for the latest preview image received, which is not limited in this embodiment.

[0423] For example, an image recognition algorithm may be integrated into an education APP to implement image recognition operations, and the education APP may also call an image recognition service to perform image recognition operations, which is not limited in this embodiment.

[0424] Regarding the image recognition algorithm, reference may be made to existing technologies, and this embodiment will not be described in detail here.

[0425] S903, the education APP in the tablet searches the database according to the book title and loads the book content corresponding to the book title.

[0426] If the tablet education app searches the database based on the book name and determines different versions of the book, it can display the corresponding book list for the user to choose. Then, in response to the user's selection of a certain version of the book, the education app loads the content corresponding to that version of the book.

[0427] S904 , in response to the user turning pages or finger-pointing reading operation, the education APP in the tablet recognizes the preview image, determines the paragraph to be read aloud, and reads the corresponding paragraph aloud.

[0428] After loading the book content, the educational app can identify the pointing tool of the finger or reading pen command for each preview image received, or can periodically identify the pointing tool of the finger or reading pen command for the most recently received preview image, which is not limited in this embodiment. In response to the user's point reading operation, the educational app can identify the book page number and the user's click location information, such as coordinate information, based on the preview image. Then, based on the book page number and the user's click location information, it can determine the paragraph to be read aloud in the loaded book content and read the corresponding paragraph aloud.

[0429] After loading the book content, the education app can also perform page turning recognition based on the preview image stream. In response to the user's page turning operation, the education app can determine the paragraph to be read aloud in the loaded book content based on the recognized book page number and read the corresponding paragraph aloud.

[0430] For any details not explained in detail in this process, please refer to the previous article and I will not repeat them here.

[0431] Figure 1a 、 Figure 11a-Figure 11b An example application scenario is shown. Figure 11a As shown in (1), the tablet displays an education APP interface 701, which displays various function options of the education application, including but not limited to a word lookup function, a point reading function, a homework function, a photo taking function, etc. The user clicks on the point reading function option 7013. In response to the user operation, the tablet executes the device discovery phase, the virtual camera service enabling phase, and the virtual camera preview access phase.

[0432] Among them, in the device discovery phase, if the number of desk lamps that can be linked with the tablet is filtered by the tablet's device management service is one, the tablet automatically executes the virtual camera service enabling phase and the virtual camera preview access phase process, and displays, for example Figure 11a The interface shown in (2). In the device discovery phase, if the number of desk lamps that can be linked to the tablet is filtered by the tablet's device management service is multiple, the tablet displays a desk lamp selection interface. Exemplarily, a list of desk lamps to be linked is displayed on the desk lamp selection interface, and the user can perform a selection operation. In response to the user's selection operation, the educational application determines a desk lamp to be linked, and continues to execute the virtual camera service enabling phase and the virtual camera preview access phase process to display, for example Figure 11a The interface shown in (2).

[0433] like Figure 11a As shown in (2), an image preview window 7081 is displayed in the interface 708. The image preview window 7081 displays a preview image captured in real time by the desk lamp camera. The education APP recognizes the preview image to determine the book name. After recognizing the book name, the education APP searches the database based on the book name. If the corresponding book is retrieved, the book content loading operation is executed, which can be referred to Figure 11b As shown in (1).

[0434] Continue to refer to Figure 11bAs shown in (1), the interface 709 displays an image preview window 7081, the recognized book name 7091, and a book content loading progress indicator 7092. After the book content is loaded, the education APP can recognize the user's point-to-read operation or page-turning operation based on the preview image. Take the user performing the page-turning operation as an example, refer to Figure 11b In the interface 710 shown in (2), the image preview window 7081 can display the user's page turning action. In response to the user's operation, the education app recognizes the book page number based on the preview image. Furthermore, the education app can determine the paragraph to be read aloud in the loaded book content based on the recognized book page number and read the corresponding paragraph aloud.

[0435] In the collaborative work method provided in the embodiment of this application, a professional online education experience can be combined using a home tablet and a desk lamp with a camera. The desk lamp's camera is used in conjunction with the tablet's education app to complete scenarios that require photography, such as student finger lookup, homework submission, and book reading.

[0436] When the electronic device 100 (taking a tablet as an example) and the IOT device 200 (taking a desk lamp as an example) based on the above-mentioned hardware and software structures cooperate with each other and are applied in the learning scenarios mentioned above, the data transmitted by the desk lamp to the tablet will vary depending on the business. For example, in the point reading scenario mentioned above, the desk lamp transmits preview stream data captured by the camera to the tablet, that is, a continuously transmitted video data stream; in the word lookup scenario and homework submission scenario mentioned above, the desk lamp transmits image data captured by the camera to the tablet.

[0437] Specifically in practical applications, when the desk lamp transmits preview stream data and image data to the tablet, it can be based on the same data packaging principle, with a set size as the unit, to encapsulate the currently generated preview stream data into a preview stream data packet, and encapsulate a part of the entire image data into an image data packet, and then transmit it to the tablet side based on the Real-time Transport Protocol (RTP). For preview stream data, this method does not involve the packaging process, so the preview stream data packet can be directly processed and displayed, that is, there is no distinction between the first packet, the last packet, and the middle packet. For image data, the final displayed photo must be determined according to the positional relationship between the image data packets, and the packaging operation is triggered according to the last packet. However, in the current image data transmission method, the data body of the image data packet must be parsed to determine whether the image data packet is the first packet.

[0438] In addition, even if the data body of the image data packet is parsed, it cannot be determined whether the image data packet is the tail packet, that is, the packet grouping cannot be triggered quickly and accurately.

[0439] In view of this, the present application provides an image data transmission method, which aims to quickly and accurately determine whether the image data packet is the first packet, the middle packet, or the last packet without parsing the data body of the received image data packet, so that when the received image data packet is identified as the last packet, the data packetization operation can be performed.

[0440] See also Figure 12 , which illustrates the specific process of image data transmission between a tablet and a desk lamp:

[0441] S1001: The tablet generates a photo request (including parameter information that the photo needs to meet) in response to a user's operation.

[0442] It is understandable that from the above description, the transmission of image data is only involved in the word search scenario and the homework submission scenario. Therefore, the tablet responds to the user's operation behavior, specifically when the user triggers the operation of the word search function or the homework submission function.

[0443] For example, in some implementations, the operation performed by the user is, for example, clicking on a control corresponding to a point-to-read function / scene displayed in the current interface of the tablet, or clicking on a control corresponding to a submitting job function / scene.

[0444] For example, in other implementations, the user performs an operation such as controlling the tablet to respond through voice commands, thereby selecting a word lookup function or submitting an assignment function.

[0445] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0446] Furthermore, as described above, the image data resolution requirements for word lookup and assignment submission are higher than those for preview stream data in point reading scenarios. Therefore, when generating a photo request in response to a user's action, it must at least include the parameters that the captured photo must meet, such as resolution.

[0447] S1002: The tablet sends a photo-taking request to the desk lamp.

[0448] After generating the photo request required above, the tablet can send the photo request to the desk lamp through a control channel negotiated with the desk lamp, for example.

[0449] S1003, the desk lamp detects whether the camera is abnormal.

[0450] In actual applications, the camera of the desk lamp may not be able to take pictures due to certain factors. Therefore, in order to minimize unnecessary processing, such as parsing of photo requests and judgment of parameter information, it is possible to first detect whether the camera of the desk lamp is abnormal in the current scene, that is, whether it can be used normally.

[0451] Correspondingly, if the camera is abnormal, the desk lamp can generate an abnormal feedback package. For the sake of convenience, the abnormal feedback package fed back when the camera is abnormal is called abnormal feedback package 1, and the abnormal feedback package fed back when the parameter is abnormal is called abnormal feedback package 2, and step S1004 is executed.

[0452] On the contrary, when the camera is normally available, the parameter information carried in the photo request is extracted, and step S1006 is executed.

[0453] S1004: The desk lamp sends an abnormality feedback packet 1 to the tablet.

[0454] Exemplarily, in some implementations, the abnormality feedback package 1 may be transmitted to the tablet via an image data channel negotiated with the tablet.

[0455] S1005: Respond according to the abnormal cause in abnormal feedback package 1.

[0456] For example, in some implementations, to facilitate the tablet's understanding of the cause of the exception in Exception Feedback Packet 1, an error code identifying the exception may be included in the data header of Exception Feedback Packet 1. This allows the tablet to quickly determine the cause of the exception based on the data header without having to parse the data itself, and to respond accordingly. For example, the tablet may display a prompt in the user interface to inform the user that a desk lamp's camera has experienced an abnormality, allowing the user to check the lamp's camera according to the prompt.

[0457] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0458] S1006: The desk lamp extracts parameter information from the photo request.

[0459] Specifically, when the camera of the desk lamp is normally available, the desk lamp can extract parameter information in the photo request sent by the tablet, such as resolution information of the photo to be taken.

[0460] S1007, the desk lamp detects whether parameter information is abnormal.

[0461] Specifically, in this embodiment, the detection of parameter information by the desk lamp is, for example, to determine whether the resolution information is supported by the camera.

[0462] Accordingly, if it is not supported, it is determined that the parameter is abnormal and step S1008 needs to be executed; if it is supported, it is determined that the parameter information is normal and step S1010 can be executed.

[0463] S1008: The desk lamp sends an exception feedback packet 2 to the tablet.

[0464] Similarly, the abnormal feedback package 2 can be transmitted to the tablet through the image data channel negotiated with the tablet.

[0465] In addition, it should be noted that in actual applications, the desk lamp side can also generate other abnormal feedback packets based on other abnormal reasons, such as network transmission abnormalities, and feed them back to the tablet side. The above are only two specific abnormal scenarios and are not intended to be a specific limitation of this embodiment.

[0466] S1009: Respond according to the abnormal cause in abnormal feedback package 2.

[0467] For example, the error code carried in Exception Feedback Packet 2 can also be stored in the data header of Exception Feedback Packet 2. This allows the tablet to quickly determine the cause of the exception based on the data header without having to parse the data body, and then respond accordingly. For example, the tablet can regenerate a photo request based on the resolution supported by the desk lamp and send the regenerated photo request to the desk lamp.

[0468] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0469] S1010: The desk lamp adjusts the shooting parameters of the camera according to the parameter information, and uses the adjusted camera to shoot to obtain an image data packet.

[0470] It should be noted that the image data packet in this embodiment specifically includes the image data (data body) of a whole photo that needs to be displayed on the tablet side, and the data header allocated to the image data.

[0471] In order to better understand the generation process of the above-mentioned image data packet, the following Figure 13 Provide explanation.

[0472] See also Figure 13 For example, using the adjusted camera to capture a photo with a size of 10,000 bytes to be displayed on the tablet, after obtaining the image data, based on the image data packet generation standard of this embodiment, the image data is used as a data body, and a 128-byte data header is added before the data body to generate the image data packet corresponding to the entire photo.

[0473] As you can understand, bytes start at 0, so the total size of the image data packet is 10,128 bytes: a 128-byte header and a 10,000-byte body. Bytes 0 to 127 correspond to the header, and bytes 128 to 10,127 correspond to the 10,000-byte image data.

[0474] Specifically in this embodiment, the data header in the image data packet includes two parts. One part is used to record whether the image data is normal, which is called the data normal identification field in this embodiment, and the other part is reserved for the expansion of subsequent services, which is called the extension field in this embodiment. In order to better understand the data header of this structure, let's combine Figure 14 Provide explanation.

[0475] See also Figure 14 In the 128 bytes of the data header, bytes 0 to 3 are used to record whether the image data is normal, that is, the data normality identification field, and bytes 4 to 127 are reserved for the expansion of subsequent services, that is, the extension field.

[0476] For example, in some implementations, it can be agreed that the user uses "0" to indicate that the data is normal, and uses "1" to indicate that the data is abnormal. That is, if the content in byte 0 is "0", it indicates that the image data in the data body is normal data (actually the binary data of the photo that needs to be returned to the tablet), that is, the corresponding photo can be restored after processing by the tablet. If the content in byte 0 is "1", it indicates that the image data in the data body is abnormal data, for example, it may be an abnormal character string.

[0477] For example, in some implementations, it may be agreed that different exception character strings correspond to different exception causes, which facilitates locating the exception later and resolving the exception.

[0478] In addition, it should be noted that in some implementations, the extended field can be further divided according to business needs. For example, for one-to-many scenarios, that is, a desk lamp needs to transmit image data to multiple tablets and mobile devices, it can be agreed that the photo request carries the unique identifier of the photo to be taken, and when the desk lamp generates the above-mentioned image data packet, it can add an identifier that identifies the uniqueness of the photo in the 4 to 32 bytes of the extended field.

[0479] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0480] S1011, the desk lamp divides the data in the image data packet into N small packets according to the set size to obtain N small packets of image data (each small packet of image data includes a data header; an identification field that records whether the small packet of image data is the first packet, the middle packet, or the last packet; and a data body that stores the image data).

[0481] It is understandable that the size of the image data packet can be dynamically set according to the bandwidth resources of the transmission channel, the current network quality, etc., or can be set to a fixed value. The specific implementation method is not limited in this embodiment.

[0482] For the sake of convenience, this embodiment takes the size of 1394 bytes as an example, and the image data packet is 10128 bytes as mentioned above. Figure 15 The process of dividing an image data packet into multiple image data packets is described.

[0483] See also Figure 15 Taking the divided image data packets as an example, which are subsequently transmitted using the RTP protocol, the data structure of the divided image data packets in this embodiment includes three parts: an RTP data header, an identification field, and a data body.

[0484] Continue to see Figure 15 For any image data packet, bytes 0 to 11 are used as the data header, byte 12 is used as the identification field, and bytes 13 to 1393 are used as the data body. That is, bytes 13 to 1393 of each image data packet are used to store the data in the image data packet. Therefore, each image data packet only stores 1381 bytes of data from the image data packet.

[0485] In addition, it should be noted that, in the image data packet as the first packet, the data body also includes the 128-byte data header in the image data packet, so in the image data packet of the first packet, such as Figure 15 The image data actually stored in the image data packet 1 is 1253 bytes of content, and the subsequent image data packets do not need to include the 128-byte data header in the image data packet, so it is 1381 bytes. Based on this, for an image data packet of 10128 bytes, it can be split into 8 image data packets with the above structure. The image data packet 8 as the tail packet is less than 1394 bytes in size, specifically a 12-byte RTP data header + a 1-byte identification field + 589 bytes of image data, that is, the size of image data packet 8 is 601 bytes.

[0486] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0487] See also Figure 16 , exemplarily shows the specific format of the RTP data header of each image data packet. Figure 16 As shown, byte 0 specifically stores the current version number (V), byte 1 specifically stores the data type (T) of the image data, bytes 2 and 3 specifically store the sequence number (SEQNUM), bytes 4 to 7 specifically store the timestamp (RTPTIME), and bytes 8 to 11 specifically store the synchronization source identifier (SSRC).

[0488] It should be noted that the data types mentioned above may specifically be Portable Network Graphics (PGN) type, Portable Network Graphics (JPG) type, etc., which are not listed here one by one and are not limited in this embodiment.

[0489] In addition, regarding the serial number mentioned above, in actual application, the serial number will be increased by 1 every time the desk lamp sends an image data packet.

[0490] Furthermore, the timestamp mentioned above specifically reflects the time when the first byte of the image data in the data body of the image data packet was captured. This allows the image data packets to be sequentially assembled based on the timestamp to restore the entire photo image data packet, allowing the tablet to process the photo image data and display it on the user interface.

[0491] In addition, the synchronization source identifier mentioned above is specifically used to identify the uniqueness of each image data packet, thereby avoiding repeated sending of the same image data packet, which would cause redundancy in the cache queue on the tablet side.

[0492] See also Figure 17 , which exemplarily shows the specific format of the identification field of each image data packet. Figure 17 As shown, the 12-byte 0 and 1 frames specifically store packet identification information that identifies whether the current image data packet is the first packet, the middle packet, or the last packet. The remaining 2 to 7 frames are reserved frames for subsequent expansion according to business needs.

[0493] That is, in the image data packet provided in this embodiment, which includes an RTP data header, an identification field, and a data body structure, the identification field is specifically used by the tablet for packet assembly processing.

[0494] Regarding the packet identification information mentioned above, in some implementations, it can be agreed to use "10" (the position of frame 0 is "1", and the position of frame 1 is "0") to indicate that the current image data packet is the first packet, that is, it includes the first frame of image data, and when assembling the packet, it is necessary to start from the first frame of image data in the image data packet.

[0495] For example, it can also be agreed that "01" (the position of frame 0 is "0", and the position of frame 1 is "1") is used to indicate that the current image data packet is the last packet, that is, it includes the last frame of the image data. When assembling the packets, the image data packet is the last packet of the photos that need to be obtained this time, and the assembly operation is triggered after receiving the packet.

[0496] For example, it can also be agreed that "00" (the position of frame 0 is "0", and the position of frame 1 is "0") is used to indicate that the current image data packet is an intermediate packet.

[0497] In addition, it should be noted that, considering actual applications, when image data of multiple photos need to be transmitted, the data of the previous and next photos may be located in the same image data packet. For example, the last frame of the previous photo and the first frame of the next photo are located in the same image data packet. The user can agree that "11" (the position of frame 0 is "1", and the position of frame 1 is "1") indicates that the current image data packet includes the last frame of the previous photo and the first frame of the next photo.

[0498] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0499] In addition, it should be noted that based on the data structure of the image data packet given in this embodiment, the abnormal feedback package 1 sent to the tablet when an abnormality occurs in the above-mentioned camera, and the abnormal feedback package 2 sent to the tablet when the camera does not support the resolution carried in the photo request, can also be sub-packetized based on the above-mentioned sub-packetization principle.

[0500] It is understandable that since the abnormal feedback packets (abnormal feedback packet 1, abnormal feedback packet 2) are usually small, after subpacketization processing using the above subpacketization principle, all data can be stored in a small packet. In this case, the above-mentioned packet identification information is "11".

[0501] S1012, the desk lamp sends N image data packets.

[0502] Specifically, after the data in the image data packet is divided into packets according to the above data structure to obtain N image data packets that need to be sent, the desk lamp can send these N image data packets through the image data channel negotiated with the tablet.

[0503] It is understandable that if it is determined during the negotiation phase that the transmitted data needs to be encrypted, each image data packet needs to be encrypted according to the negotiated encryption method, and then the encrypted image data packet is sent to the tablet.

[0504] S1013: The tablet performs subsequent processing according to the content of the identification field in the currently received image data packet.

[0505] As you can understand, if the received image data packet is encrypted, the tablet first decrypts it according to the decryption method negotiated with the lamp to obtain the plaintext image data packet. The tablet then determines whether the packet identification information is the agreed-upon "10" by checking whether the 0th frame in the 12-byte identification field is "1" and the 1st frame is "0." This determines whether the currently received image data packet contains the first frame, i.e., the first packet.

[0506] Correspondingly, if so, the image data packet is cached in the image data packet cache queue, and then the above-mentioned processing is performed on the received image data packet. If the packet identifier is the agreed "00", it is determined to be an intermediate packet and is cached in the image data packet cache queue. Then the above-mentioned processing is performed on the received image data packet until the packet identification information of the received image data packet is the agreed "01" or "11", and it is cached in the image data packet cache queue, and the reception of image data packets is stopped, and the packet assembly operation is started.

[0507] For ease of understanding, the following Figures 18 to 21 Describes the operations involved in package assembly.

[0508] Still taking the above example where the image data is 10,000 bytes as an example, when performing the packet grouping operation, the tablet will take out the image data packet 1 from the image data packet cache queue, such as Figure 18 As shown, in the 12-byte identification field, the recorded packet identification information is "10", indicating that the image data packet 1 is the first packet, and the data body of the packet includes Figure 13 The data header of the entire image data packet is shown, and the first byte of the data header records the identification of whether the image data is normal. Therefore, the data body of the image data packet 1 can be parsed to extract the content recorded in byte 13, and then determine whether the image data is normal.

[0509] For example, when the flag recorded in byte 13 is "1" (e.g. Figure 18As shown), according to the above description, "1" indicates that the image data is abnormal. In this case, there is no need to parse and process other image data packets, such as image data packet 2 to image data packet 8 mentioned above, and the image data packet cache queue can be directly cleared.

[0510] For example, in some implementations, after clearing the image data packet cache queue, the tablet may prompt the user on the user interface that the image data captured this time is abnormal and cannot be displayed, thereby enabling the user to trigger a photo request again to obtain image data.

[0511] For example, in other implementations, after clearing the image data packet cache queue, the tablet can also simulate user operations and automatically generate a photo request, so that the desk lamp can re-execute the above process according to the new photo request.

[0512] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0513] For example, when the identifier recorded in byte 13 is "0" (e.g. Figure 19 As shown), according to the above description, "0" indicates that the image data is normal. In this case, other image data packets cached in the image data packet cache queue can be parsed and processed.

[0514] For example, if the content of frame 0 in the 12 bytes of the currently parsed image data packet is "0", the content of frame 1 is "0", that is, the packet identification information is "00" (such as Figure 20 As shown), this indicates that the image data packet is an intermediate packet. By parsing the intermediate packet, the image data of the intermediate packet can be obtained.

[0515] For example, if the content of frame 0 in the 12 bytes of the currently parsed image data packet is "0", and the content of frame 1 is "1", that is, the packet identification information is "01" (e.g. Figure 21 As shown), this indicates that the image data packet is the tail packet. After parsing the image data packet, the image data of the tail packet can be obtained.

[0516] After parsing the first, middle, and last packets, the positional relationship between the image data can be determined based on the 2- to 3-byte sequence number and 4- to 7-byte timestamp in the RTP header of each image data packet, thereby restoring the image data of the photo captured by the camera. The tablet can then process the image data according to business needs and display the processed photo on the user interface.

[0517] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0518] Therefore, the image data transmission method provided by this embodiment divides the image data packet into packets according to the above-mentioned data structure before transmission, so that the tablet that receives the image data packet can quickly and accurately determine whether the current image data packet is the first packet, the middle packet, or the last packet based on the packet identification information of the identification field in each image data packet.

[0519] That is to say, based on the image data transmission method provided in this embodiment, there is no need to parse the data body of the received image data packet. The first packet, middle packet and last packet of each photo that needs to be restored can be quickly and accurately determined, thereby quickly and accurately realizing package assembly.

[0520] In addition, it can be understood that the above is an explanation of the image data transmission method provided by this application from the perspective of the two devices, the tablet and the desk lamp. In the specific implementation, the implementation of the above process needs to involve educational applications, device management services, camera services, transmission management services, hardware virtualization services, virtual camera HAL on the tablet side, and device interconnection services, hardware abstraction services, and cameras on the desk lamp side. For the specific interactions between these modules, please refer to the description of the word search scenario and homework submission scenario above, which will not be repeated here.

[0521] In addition, it can also be understood that, in order to implement the above functions, the electronic device includes hardware and / or software modules that perform the corresponding functions. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0522] In addition, it should be noted that the methods provided in the above embodiments implemented by electronic devices in actual application scenarios can also be performed by a chip system included in the electronic device, wherein the chip system may include a processor. The chip system can be coupled to a memory so that when the chip system is running, it calls the computer program stored in the memory to implement the steps performed by the above electronic device. The processor in the chip system can be an application processor or a processor other than an application processor.

[0523] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the methods in the above-mentioned embodiments.

[0524] In addition, an embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned related steps to implement the methods in the above-mentioned embodiments.

[0525] In addition, an embodiment of the present application also provides a chip (which may also be a component or module), which may include one or more processing circuits and one or more transceiver pins; wherein the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment to control the receiving pin to receive the signal, so as to control the transmitting pin to send the signal.

[0526] In addition, it can be seen from the above description that the electronic device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0527] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for transmitting image data, characterized in that: Applied to an Internet of Things device, the Internet of Things device is provided with a camera, the camera is used to collect image data downward, the Internet of Things device is bound to a target application in an electronic device, and the image data transmission method includes: receiving a photographing request sent by an electronic device, wherein the photographing request carries an indication of a resolution of image data captured by the camera; Calling the camera to capture an image at the resolution to obtain image data; processing the image data into an image data packet according to a first data packet standard; The image data packet is packetized according to a second data packet standard to obtain N image data packets, where N is an integer greater than 0. The second data packet standard indicates that the image data packet includes a second data header, a packet identification field, and a second data body. The second data header carries information for determining a positional relationship between image data carried in the image data packet and image data carried in other image data packets. The packet identification field records packet identification information for identifying the image data packet as a first packet, an intermediate packet, or a last packet. The contents of the second data header and the second data body need to be parsed to obtain, while the content recorded in the identification field can be obtained without parsing. The obtained N image data packets are sent to the electronic device.

2. The method according to claim 1, characterized in that The first data packet standard indicates that the image data packet includes a first data header and a first data body; The first data body is used to store the image data; The first data header includes an image data identification field, and the image data identification information recorded in the image data identification field is used to identify whether the image data stored in the first data body is normal.

3. The method according to claim 2, characterized in that The first data header also includes an extension field.

4. The method according to claim 3, characterized in that The size of the first data header is 128 bytes, the identification field is located at 0 to 3 bytes, and the extension field is located at 4 to 127 bytes.

5. The method according to claim 4, characterized in that The second data header occupies 12 bytes, and the packet identification field occupies 1 byte; The packet identification information is the combination of the binary data corresponding to the 0th frame and the 1st frame in the packet identification field; The second data volume is used to store the image data.

6. The method according to claim 5, characterized in that When the packet identification information indicates that the image data packet is the first packet, the second data body is also used to store the first data header.

7. The method according to any one of claims 1 to 6, characterized in that After receiving the photo-taking request sent by the electronic device, the method is further configured to: Detecting whether the camera is abnormal; When the camera is abnormal, a first abnormality feedback packet is sent to the electronic device, where the first abnormality feedback packet carries an abnormality reason indicating the abnormality of the camera; When the camera is normal, detecting whether the camera supports the resolution; When the camera supports the resolution, executing the step of calling the camera to capture an image according to the resolution; When the camera does not support the resolution, a second abnormality feedback packet is sent to the electronic device, where the second abnormality feedback packet carries an abnormality reason indicating that the resolution is abnormal.

8. The method according to any one of claims 1 to 6, characterized in that The resolution is 1080P.

9. The method according to any one of claims 1 to 6, characterized in that The IoT device is a desk lamp.

10. The method according to any one of claims 1 to 6, characterized in that The electronic devices include mobile phones and tablet computers.

11. A method for transmitting image data, characterized in that: Applied to an electronic device, wherein a target application in the electronic device is bound to an Internet of Things device, and the Internet of Things device is provided with a camera, the camera is used to collect image data downward, and the image data transmission method includes: Registering a virtual camera corresponding to the camera in the system, and sending a photo request to the IoT device by calling the virtual camera, wherein the photo request carries an indication of the resolution of the image data captured by the camera; Receive an image data packet sent by the IoT device, where the image data packet is obtained by the IoT device by subpacketizing an image data packet according to a second data packet standard, the image data packet is obtained by the IoT device by processing image data according to a first data packet standard, the image data being captured by the camera at the resolution, the image data packet being divided into N image data packets, where N is an integer greater than 0, the second data packet standard indicating that the image data packet includes a second data header, a packet identification field, and a second data body, the second data header carrying information for determining a positional relationship between image data carried in the image data packet and image data carried in other image data packets, the packet identification field recording packet identification information for identifying the image data packet as a first packet, an intermediate packet, or a last packet, the contents of the second data header and the second data body need to be parsed to be obtained, and the content recorded in the identification field can be obtained without parsing; When an image data packet carrying packet identification information as a tail packet is received, the N received image data packets are grouped to restore the image data captured by the camera at the resolution, and the image data is displayed in the target application.

12. The method according to claim 11, characterized in that The first data packet standard indicates that the image data packet includes a first data header and a first data body; The first data body is used to store the image data; The first data header includes an image data identification field, and the image data identification field is used to identify whether the image data stored in the first data body is normal.

13. The method according to claim 12, characterized in that The first data header also includes an extension field.

14. The method according to claim 13, wherein: The size of the first data header is 128 bytes, the identification field is located at 0 to 3 bytes, and the extension field is located at 4 to 127 bytes.

15. The method according to claim 14, characterized in that The second data header occupies 12 bytes, and the packet identification field occupies 1 byte; The packet identification information is the combination of the binary data corresponding to the 0th frame and the 1st frame in the packet identification field; The second data volume is used to store the image data.

16. The method according to claim 15, characterized in that When the packet identification information indicates that the image data packet is the first packet, the second data body is also used to store the first data header.

17. The method according to claim 16, characterized in that After receiving the image data packet sent by the IoT device, the method further includes: extracting the packet identification information from bytes occupied by the packet identification field according to the second data packet standard; When the packet identification information indicates that the image data packet is a first packet, adding the first packet to an image data packet cache queue; When the packet identification information indicates that the image data packet is an intermediate packet, adding the intermediate packet to an image data packet cache queue; When the packet identification information indicates that the image data packet is a tail packet, the tail packet is added to the image data packet cache queue, and the step of grouping the received N image data packets is performed.

18. The method according to claim 17, characterized in that The step of grouping the received N image data packets comprises: parsing the second data header of each of the image data packets according to the second data packet standard; According to the serial number and timestamp recorded in the second data header, the data in the second data body of each image data packet is extracted in sequence and combined.

19. The method according to claim 18, characterized in that Before parsing the second data header of each of the image data packets, the method further includes: Determining the byte where the image data identification field in the first packet is located according to the second data packet standard and the first data packet standard; Extracting image data identification information from the bytes occupied by the image data identification field; When the image data identification information indicates that the image data is normal, performing the step of parsing the second data header of each of the image data packets; When the image data identification information indicates that the image data is abnormal, the N image data packets cached in the image data packet cache queue are cleared.

20. The method according to any one of claims 11 to 19, characterized in that The method further comprises: receiving a first abnormality feedback packet sent by the IoT device, and responding according to an abnormality cause in the first abnormality feedback packet, where the first abnormality feedback packet carries an abnormality cause indicating that the camera is abnormal; Receive a second abnormality feedback packet sent by the Internet of Things device, and respond according to the abnormality cause in the second abnormality feedback packet, where the second abnormality feedback packet carries the abnormality cause indicating a resolution abnormality.

21. The method according to any one of claims 11 to 19, characterized in that The resolution is 1080P.

22. The method according to any one of claims 11 to 19, characterized in that The IoT device is a desk lamp.

23. The method according to any one of claims 11 to 19, characterized in that The electronic devices include mobile phones and tablet computers.

24. An Internet of Things device, characterized in that: The Internet of Things device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the Internet of Things device executes the image data transmission method according to any one of claims 1 to 10.

25. An electronic device, characterized in that: The electronic device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the image data transmission method according to any one of claims 1 to 23.

26. A collaborative work system, characterized in that: Includes the Internet of Things device as described in claim 24 and the electronic device as described in claim 25.

Images