Image data transmission method, device and collaborative working system

Abstract

The present application provides an image data transmission method, device, and collaborative working system. When an abnormality occurs in the channel for transmitting picture data packets between an electronic device and an Internet of Things device, the Internet of Things device deletes all local image data packets that have not been transmitted to the tablet and sends an image data interruption feedback packet to the electronic device that can indicate the abnormality in this transmission. When the electronic device determines that the received data packet is an image data interruption feedback packet, it can directly reconstruct the transmission channel and end the transmission of this image data packet in advance, thereby effectively reducing the occupation of device resources and network resources and improving the transmission performance of image data.

Description

Technical Field

[0001] This application relates to the field of communication technologies, and in particular, to an image data transmission method, device, and collaborative working system. Background Art

[0002] With the development of communication technologies, data can be transmitted between any two electronic devices according to service requirements, such as image data transmission. Considering that image data is usually large, to accelerate the transmission speed and avoid network congestion, the image data of a large-capacity photo, such as a photo with a size of 100K - 20M, is usually decomposed into multiple small image data packets, and then the obtained multiple small image data packets are placed in a transmission queue. Subsequently, the small image data packets are sequentially taken out from the transmission queue for transmission.

[0003] Since data placement and data transmission are asynchronous, during the image data transmission process, if the speed of placing small image data packets into the transmission queue is greater than the speed of taking out small image data packets from the transmission queue and sending them to the peer device, the small image data packets cached in the transmission queue will exceed the set threshold. In this case, the small image data packets that need to be continuously placed into the transmission queue will be discarded. Although this method can reduce the blocking phenomenon during transmission and prevent the device from freezing. However, since the small image data packets in the transmission queue still wait for transmission, and the peer device will still perform packet assembly operations after receiving the last small image data packet, this will not only cause the peer device to be unable to restore the image data, but also waste device resources and network resources. Summary of the Invention

[0004] To solve the above technical problems, this application provides an image data transmission method, device, and collaborative working system, aiming to enable the Internet of Things device to directly empty the small image data packets in the transmission queue when an abnormality occurs in the channel for transmitting small picture data between the electronic device and the Internet of Things device, and enable the electronic device to directly reconstruct the transmission channel, ending the transmission of the small image data packets in this round in advance, reducing the occupation of device resources and network resources, and improving the transmission performance of image data.

[0005] In a first aspect, the present application provides an image data transmission method. This method is applied to an Internet of Things device, on which a camera is provided. The camera is used to collect image data downward. The Internet of Things device is bound to a target application in an electronic device, and includes: a first thread packets the image data obtained by the camera shooting according to the resolution carried in the photographing request, obtaining N small image data packets, where N is an integer greater than 0; after obtaining the N small image data packets, start a second thread, which is used to read small image data packets from the head of the queue to be transmitted; the first thread sequentially adds the obtained N small image data packets to the end of the queue to be transmitted; during the process that the first thread adds small image data packets to the queue to be transmitted, the second thread performs the following operations: when reading a small image data packet from the queue to be transmitted each time, obtain the number of small image data packets in the queue to be transmitted; when the number of small image data packets in the queue to be transmitted is less than a set threshold, take out a small image data packet from the head of the queue to be transmitted and send it to the electronic device; otherwise, notify the first thread to stop adding small image data packets to the queue to be transmitted, and wait for a preset duration; after waiting for the preset number of preset durations, if the number of small image data packets in the queue to be transmitted is still not less than the set threshold, the second thread empties the queue to be transmitted, and generates an image data interruption feedback packet, which carries an identification that it is an image data interruption feedback packet and identification information of non-small image data packets; send the generated image data interruption feedback packet to the electronic device.

[0006] Among them, the Internet of Things device can be any device capable of taking pictures and transmitting image data to an electronic device. Specifically in this embodiment, the Internet of Things device can be, for example, a table lamp with a camera, or directly a camera, etc.

[0007] Among them, the electronic device is, for example, an electronic device with powerful processing capabilities such as a mobile phone or a tablet.

[0008] Among them, the photographing request is generated, for example, when the electronic device responds to the user's operation behaviors in the word query scenario and the homework submission scenario.

[0009] For the word query scenario and the homework submission scenario, refer to the specific descriptions below, and details are not elaborated here.

[0010] It can be understood that since accurate identification of the content selected by the user is required for word query, and the content of the submitted homework needs to be clearly visible, the resolution carried in the photographing request is relatively high, such as 1080P.

[0011] Among them, the target application is, for example, the education application mentioned below.

[0012] Thus, when an abnormality occurs in the channel for transmitting small packets of picture data between an electronic device and an Internet of Things (IoT) device, the IoT device deletes all local small packets of image data that have not been transmitted to the tablet, and sends an image data interruption feedback packet to the electronic device that can indicate the abnormality of the current transmission. When the electronic device determines that the received data packet is an image data interruption feedback packet, it can directly reconstruct the transmission channel and end the transmission of the small packets of image data in advance, thereby effectively reducing the occupation of device resources and network resources and improving the transmission performance of image data.

[0013] In addition, specifically in practical applications, the two devices for image data transmission can both be the above-mentioned electronic devices, such as tablets, mobile phones, etc., or both can be IoT devices, such as the above-mentioned table lamps, cameras, etc. This application does not limit this.

[0014] For the sake of convenience of description, this application takes the device sending image data as an IoT device and the device receiving image data as an electronic device as an example.

[0015] According to the first aspect, when the second thread clears the transmission queue, the method further includes: the second thread notifies the first thread to delete all small packets of image data that have not been written into the transmission queue. In this way, the first thread will not continue to write small packets of image data into the transmission queue when the second thread clears the transmission queue (the data of the cached small packets of image data is less than the set threshold), and the second thread will not perform the operation of taking out small packets of image data from the transmission queue, that is, transmitting to the electronic device, thereby reducing the transmission pressure of the image data channel and enabling the image data interruption feedback packet generated by the second thread to be transmitted to the electronic device side as much as possible.

[0016] According to the first aspect, or any one of the above implementations of the first aspect, the preset number of times is 10 times, and the preset duration for each wait is 50 ms.

[0017] It can be understood that in practical applications, a total waiting duration and a preset number of times can also be set. The preset duration for each wait can be the same or different.

[0018] It can be understood that the total waiting duration needs to be less than the timeout duration set by the electronic device. In order to improve the user experience, the image data channel can be automatically re-established without the user's perception, and the small packets of image data can be received within a short time. The set total waiting duration can be one-tenth, one-twentieth, etc. of the timeout duration set by the electronic device.

[0019] For example, when the timeout duration set by the electronic device, that is, the duration for waiting to receive small packets of image data, is 10 s, the set total waiting duration can be one-twentieth of the timeout duration, that is, 500 ms as mentioned above.

[0020] It is understandable that the setting for 100% can be determined according to the size of the image data captured at the specified resolution and the average transmission duration.

[0021] According to the first aspect, or any implementation manner of the above first aspect, the first thread splits the image data obtained by the camera shooting according to the resolution carried in the shooting request, including: the first thread processes the image data into image data packets according to the first data packet standard; the first thread splits the image data packets according to the second data packet standard; wherein, each small image data packet carries packet identification information indicating that the small image data packet is the first packet, or the middle packet, or the last packet.

[0022] Among them, the first data packet standard indicates how large a data header (the first data as described below) the generated image data packet needs to include, what specific content is carried in the data header, and from which position the data body (the first data body as described below) starts.

[0023] Among them, the second data packet standard indicates the size of each small image data packet, how large a data header (the second data header as described below) each small image data packet needs to include, what specific content is carried in the data header, the identification field, and from which position the data body (the second data body as described below) starts.

[0024] Thus, by setting an identification field in each small image data packet that can record the packet identification information used to identify whether the small image data packet is the first packet, the middle packet, or the last packet, the electronic device receiving these small image data packets can quickly and accurately determine whether the small image data packet is the first packet, the middle packet, or the last packet according to the packet identification information recorded in the identification field without parsing the small image data packet, that is, without knowing the specific content carried in the second data header and the second data body. In this way, when it is recognized that the received small image data packet is the last packet, the data packet assembly operation can be executed.

[0025] According to the first aspect, or any implementation manner of the above first aspect, the first data packet standard indicates that the image data packet includes a first data header and a first data body; the first data body is used to store the image data; the first data header includes an image data identification field, and the image data identification information recorded in the image data identification field is used to identify whether the image data stored in the first data body is normal.

[0026] Among them, the image data is the actual data captured by the camera, and is specifically stored in the first data body in the form of binary code, such as Figure 13 in the data body.

[0027] According to the first aspect, or any implementation of the above first aspect, the first data header further includes an extended field.

[0028] Exemplarily, in some implementations, the extended field can be further divided. For example, a part of it is used to record the unique identifier of the image data to achieve accurate uplink transparent transmission.

[0029] Exemplarily, regarding the identifier for uniquely identifying the image data, it can be generated by the electronic device and carried in the capture request.

[0030] According to the first aspect, or any implementation of the above first aspect, the size of the first data header is 128 bytes. The identification field is located at bytes 0 to 3, and the extended field is located at bytes 4 to 127. Regarding the structure of the first data header, reference can be made to Figure 14 as shown, which will not be elaborated here.

[0031] According to the first aspect, or any implementation of the above first aspect, the second data packet standard indicates that the image data packetlet includes a second data header, a packet identification field, and a second data body; the second data header occupies 12 bytes, and the packet identification field occupies 1 byte; the content of the binary data corresponding to the 0th frame and the 1st frame in the packet identification field is the packet identification information; the second data body is used to store the image data.

[0032] Among them, the second data body is, for example, a Real-Time Transport Protocol (RTP) data header. Regarding the specific structure of the RTP data header, reference can be made to the following text, which will not be elaborated here.

[0033] Among them, the size of each image data packetlet is, for example, 1394 bytes. Regarding the positions of the second data header, the packet identification field, and the second data body, as well as the bytes occupied, reference can be made to the following text, which will not be elaborated here.

[0034] Exemplarily, in some implementations, it can be agreed that when the 0th frame is "1" and the 1st frame is "0", the corresponding packet identification information "10" indicates that the current image data packetlet is the first packet.

[0035] Exemplarily, in some implementations, it can be agreed that when the 0th frame is "0" and the 1st frame is "1", the corresponding packet identification information "01" indicates that the current image data packetlet is the last packet.

[0036] Exemplarily, in some implementations, it can be agreed that when the 0th frame is "0" and the 1st frame is "0", the corresponding packet identification information "00" indicates that the current image data packetlet is an intermediate packet.

[0037] Regarding the specific structures of the first packet, the intermediate packet, and the last packet, reference can be made to the following text, which will not be elaborated here.

[0038] According to the first aspect, or any implementation of the above first aspect, when the packet identification information indicates that the small packet of image data is the first packet, the second data body is further used to store the first data header. In this way, only the first data header is carried in the second data body of the first packet, and the data in the first data body is directly stored in other small packets of image data, thereby reducing the number of small packets of image data. Moreover, by carrying the first data header in the second data body of the first packet, it is possible for the electronic device to determine whether the entire image data is normal according to the image data identification information of the first data header in the second data body during packet assembly, avoiding invalid packet assembly operations.

[0039] According to the first aspect, or any implementation of the above first aspect, the second thread generates an image data interruption feedback packet, including: the second thread generates an image data interruption feedback packet including a third data header and an interruption identification field according to the third data packet standard; wherein, the third data header occupies 12 bytes, and the interruption identification field is filled with the agreed interruption identification information.

[0040] Among them, the interruption identification information is, for example, 0xE0 mentioned below.

[0041] Among them, the third data header, for example, is an RTP data header, and its specific structure can be as Figure 22 shown.

[0042] In the second aspect, the present application provides a method for transmitting image data. The method is used in an electronic device, in which a target application is bound to an Internet of Things device, and a camera is provided on the Internet of Things device, and the camera is used to collect image data downward, including: registering a virtual camera corresponding to the camera in the system, and sending a photo request to the Internet of Things device by calling the virtual camera, wherein the photo request carries the resolution of the image data captured by the camera; receiving a data packet sent by the Internet of Things device, and determining the attributes of the data packet according to the identification information carried in the data packet; when the data packet is an image data packet, adding the image data packet to a cache queue, the image data packet is obtained by the Internet of Things device by sub-packetizing the image data packet according to the second data packet standard, the image data packet is obtained by the Internet of Things device by processing the image data according to the first data packet standard, the image data packet is obtained by capturing the image data by the camera according to the resolution, and the image data packet can be The image data packets are divided into N packets, where N is an integer greater than 0, and each image data packet carries packet identification information for identifying the image data packet as a first packet, a middle packet, or a tail packet; when the packet grouping condition is met, the N image data packets in the cache queue are grouped to restore the image data captured by the camera according to the resolution, and the image data is displayed in the target application; when the data packet is an image data interruption feedback packet, the reconstruction operation of the image data channel is triggered. The image data interruption feedback packet is generated by the Internet of Things device according to the second data packet standard or the third data packet standard. The image data interruption feedback packet carries identification information for identifying it as an image data interruption feedback packet, not an image data packet. The image data channel is a channel for the Internet of Things device to transmit image data packets or image data interruption feedback packets to the electronic device.

[0043] According to the second aspect, triggering the reconstruction operation of the image data channel includes: calling a virtual camera to send a photo request to the IoT device, the photo request carries the resolution of the image data captured by the camera; wherein, each time a photo request is generated, the electronic device and the IoT device need to negotiate to determine an image data channel. Since each time a photo request is triggered, the electronic device and the Internet device must negotiate to determine the image data channel corresponding to the image data captured for this photo request, when it is determined that the received data packet is an image data interruption feedback packet, the virtual camera is directly called to resend the photo request to the IoT device, so that the image data channel can be reconstructed.

[0044] According to a second aspect, or any implementation manner of the above second aspect, the method further includes: when receiving an image data packet with the carried identification information being the end packet, determining that the packet assembly condition is satisfied. In this way, without parsing the data body of the received image data packet, that is, without knowing the specific content carried in the second data header and the second data body, the electronic device can quickly and accurately determine whether the image data packet is the first packet, the middle packet, or the end packet according to the packet identification information recorded in the identification field. Thus, when it is recognized that the received image data packet is the end packet, the data packet assembly operation can be performed.

[0045] According to a second aspect, or any implementation manner of the above second aspect, the first data packet standard indicates that the image data packet includes a first data header and a first data body; the first data body is used to store image data; the first data header includes an image data identification field, and the image data identification field is used to identify whether the image data stored in the first data body is normal.

[0046] According to a second aspect, or any implementation manner of the above second aspect, the first data header further includes an extension field.

[0047] According to a second aspect, or any implementation manner of the above second aspect, the size of the first data header is 128 bytes, the identification field is located at bytes 0 to 3, and the extension field is located at bytes 4 to 127.

[0048] According to a second aspect, or any implementation manner of the above second aspect, the second data packet standard indicates that the image data packet includes a second data header, a packet identification field, and a second data body; the second data header occupies 12 bytes, and the packet identification field occupies 1 byte; the content obtained by combining the binary data corresponding to the 0th frame and the 1st frame in the packet identification field is the packet identification information; the second data body is used to store image data.

[0049] According to a second aspect, or any implementation manner of the above second aspect, when the packet identification information indicates that the image data packet is the first packet, the second data body is further used to store the first data header.

[0050] According to a second aspect, or any implementation manner of the above second aspect, when the image data interruption feedback packet is generated according to the second data packet standard, the image data interruption feedback packet includes a third data header and an interruption identification field; wherein, the third data header occupies 12 bytes, and the interruption identification field is filled with the agreed interruption identification information.

[0051] The second aspect and any implementation manner of the second aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the second aspect and any implementation manner of the second aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect as described above, and details are not repeated here.

[0052] In a third aspect, the present application provides an Internet of Things device. The Internet of Things device includes: a memory and a processor, the memory and the processor being coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the Internet of Things device is caused to execute the instructions of the method in the first aspect or any possible implementation manner of the first aspect.

[0053] The third aspect and any implementation manner of the third aspect respectively correspond to the first aspect and any implementation manner of the first aspect. For the technical effects corresponding to the third aspect and any implementation manner of the third aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect as described above, and details are not repeated here.

[0054] In a fourth aspect, the present application provides an electronic device. The electronic device includes: a memory and a processor, the memory and the processor being coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device is caused to execute the instructions of the method in the second aspect or any possible implementation manner of the second aspect.

[0055] The fourth aspect and any implementation manner of the fourth aspect respectively correspond to the second aspect and any implementation manner of the second aspect. For the technical effects corresponding to the fourth aspect and any implementation manner of the fourth aspect, reference may be made to the technical effects corresponding to the second aspect and any implementation manner of the second aspect as described above, and details are not repeated here.

[0056] In a fifth aspect, the present application provides a computer-readable medium for storing a computer program, the computer program including instructions for executing the method in the first aspect or any possible implementation manner of the first aspect, or instructions for executing the method in the second aspect or any possible implementation manner of the second aspect.

[0057] The fifth aspect and any implementation manner of the fifth aspect respectively correspond to the first aspect and any implementation manner of the first aspect, or the second aspect or any possible implementation manner of the second aspect. For the technical effects corresponding to the fifth aspect and any implementation manner of the fifth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect, or the technical effects corresponding to the second aspect or any possible implementation manner of the second aspect as described above, and details are not repeated here.

[0058] Sixth aspect, the present application provides a computer program product, including a computer program, the computer program includes instructions for executing the method in the first aspect or any possible implementation manner of the first aspect, or instructions for executing the method in the second aspect or any possible implementation manner of the second aspect.

[0059] The sixth aspect and any implementation manner of the sixth aspect respectively correspond to the first aspect and any implementation manner of the first aspect, or the second aspect or any possible implementation manner of the second aspect. For the technical effects corresponding to the sixth aspect and any implementation manner of the sixth aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect, or the second aspect or any possible implementation manner of the second aspect, which will not be elaborated herein.

[0060] Seventh aspect, the present application provides a chip, the chip includes a processing circuit and transceiver pins. Among them, the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the method in the first aspect or any possible implementation manner of the first aspect, or executes the method in the second aspect or any possible implementation manner of the second aspect to control the receiving pin to receive a signal and control the sending pin to send a signal.

[0061] The seventh aspect and any implementation manner of the seventh aspect respectively correspond to the first aspect and any implementation manner of the first aspect, or the second aspect or any possible implementation manner of the second aspect. For the technical effects corresponding to the seventh aspect and any implementation manner of the seventh aspect, reference may be made to the technical effects corresponding to the first aspect and any implementation manner of the first aspect, or the second aspect or any possible implementation manner of the second aspect, which will not be elaborated herein. Description of the Drawings

[0062] Figures 1a - 1b An exemplary application scenario is shown;

[0063] Figure 2a A schematic diagram of the hardware structure of an exemplary electronic device is shown;

[0064] Figure 2b A schematic diagram of the software structure of an exemplary electronic device is shown;

[0065] Figure 3a A schematic diagram of the hardware structure of an exemplary Internet of Things device is shown;

[0066] Figure 3b A schematic diagram of the software structure of an exemplary Internet of Things device is shown;

[0067] Figure 4aA schematic diagram of module interaction provided by an embodiment of the present application;

[0068] Figure 4b A schematic diagram of module interaction provided by an embodiment of the present application;

[0069] Figures 5a - 5b An exemplary application scenario;

[0070] Figure 6a A schematic diagram of module interaction provided by an embodiment of the present application;

[0071] Figure 6b A schematic diagram of module interaction provided by an embodiment of the present application;

[0072] Figures 7a - 7b An exemplary application scenario;

[0073] Figure 8 A schematic diagram of module interaction provided by an embodiment of the present application;

[0074] Figures 9a - 9c An exemplary application scenario;

[0075] Figure 10 A schematic diagram of module interaction provided by an embodiment of the present application

[0076] Figures 11a - 11b An exemplary application scenario;

[0077] Figure 12 A schematic diagram showing the image data transmission between a tablet and a table lamp by way of example;

[0078] Figure 13 A schematic diagram of the data structure of the entire image data packet shown by way of example;

[0079] Figure 14 A schematic diagram of the structure of the data header in the image data packet shown by way of example;

[0080] Figure 15 A schematic diagram of the data structure of the small image data packet shown by way of example;

[0081] Figure 16 A schematic diagram of the structure of the RTP data header in the small image data packet shown by way of example;

[0082] Figure 17 A schematic diagram of the structure of the identification field in the small image data packet shown by way of example;

[0083] Figure 18 A schematic diagram of the structure where the first packet of the small image data packet is an abnormal packet shown by way of example;

[0084] Figure 19 Schematic diagram of the structure of the first packet of the small packet of image data shown exemplarily as a normal packet;

[0085] Figure 20 Schematic diagram of the structure of the middle packet of the small packet of image data shown exemplarily;

[0086] Figure 21 Schematic diagram of the structure of the last packet of the small packet of image data shown exemplarily;

[0087] Figure 22 Schematic diagram of image data transmission involving image data retransmission and interruption shown exemplarily;

[0088] Figure 23 Schematic diagram of the structure of the image data interruption feedback packet shown exemplarily. Detailed implementation manners

[0089] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, rather than all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without making creative efforts shall fall within the protection scope of the present application.

[0090] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0091] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of the objects. For example, the first target object and the second target object are used to distinguish different target objects, rather than to describe a specific order of the target objects.

[0092] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplary" or "for example" aims to present relevant concepts in a specific manner.

[0093] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" refers to two or more. For example, a plurality of processing units refers to two or more processing units; a plurality of systems refers to two or more systems.

[0094] In the description of the embodiments of the present application, unless otherwise specified, "a plurality of" means two or more. For example, a plurality of processing units means two or more processing units; a plurality of systems means two or more systems.

[0095] With the development of the Internet, online education has become increasingly popular among more and more people, and the demand of users (such as students) for online education is also growing. In some application scenarios, when encountering new words, students can look up words online to obtain relevant explanations; in some application scenarios, online reading of book content is more convenient for students to learn knowledge and pronunciation; in some application scenarios, students need to submit their homework online. Therefore, how to meet the online education needs of users based on intelligent devices is a problem that needs to be solved.

[0096] Currently, for the online education scenario, users usually use intelligent learning machines with both shooting and display functions. Such devices require cameras or mirrors at special positions to achieve shooting of books, and their versatility and ease of use are not strong. In addition, devices with both shooting and display functions require strong hardware and system support, and the device cost is relatively high. Furthermore, how to provide users with a better online education experience based on intelligent devices, improve versatility and ease of use, and reduce the cost of online education is a problem that needs to be solved.

[0097] A collaborative working system provided by an embodiment of the present application can be applied to the online education scenario. The system includes an electronic device and a table lamp that establish a communication connection. The table lamp is provided with a camera that can be used to shoot the book downward. The electronic device calls the camera of the table lamp to collect images, and combined with the online education resources on the platform, it can meet the online education needs of users. Among them, the electronic device can be a tablet computer or a mobile phone, etc. In addition to the online education scenario, the electronic device and the table lamp can also serve users based on their respective basic functions (i.e., communication function and lighting function). In this way, the system can create a better online education experience for users based on two commonly used intelligent devices by users, with strong versatility and ease of use. In addition, since tablet computers or mobile phones, etc. are already essential products in every household, and the device cost of a table lamp with only a shooting function is lower than that of a device with both shooting and display functions, the online education cost of users is greatly reduced.

[0098] The following takes the electronic device as a tablet as an example to explain the technical solution provided by the present application.

[0099] Figure 1a An exemplary application scenario is shown. Such as Figure 1aAs shown, the collaborative work system includes a tablet 100 and a desk lamp 200 that establish a communication connection. Among them, the desk lamp 200 includes a camera 201 for capturing images downward, for example, it can be a downward shot of text content or picture content in a book. An educational APP (Application) is installed in the tablet 100, and the educational APP can call the camera 201 of the desk lamp 200 to capture images, and provide users with various online education functions based on the images captured by the camera 201 of the desk lamp 200, such as online word search, online reading, online homework submission, etc.

[0100] Although the tablet also has a front camera and a rear camera, no matter which camera is used to take pictures of the book, the user needs to hold the tablet and aim the camera at the book, which not only makes the shooting unstable, but also affects the user's finger reading and point reading operations, and cannot provide users with a good online education experience. Figure 1a As shown, the tablet 100 and the desk lamp 200 can be placed in a fixed position, and the tablet 100 uses the camera 201 of the desk lamp 200 to shoot the book. The stable shooting picture makes the content recognition success rate high, and the user can also flexibly perform finger reading and point reading operations in the book. Therefore, the linkage between the tablet and the desk lamp can provide users with a better online education experience.

[0101] like Figure 1b As shown, the tablet 100 and the lamp 200 can perform near field communication or far field communication. Among them, near field communication can complete the information exchange between devices through devices such as routers, and far field communication can complete the information exchange between devices through cloud servers. Exemplarily, the tablet 100 and the lamp 200 can realize near field communication based on Wi-Fi (wireless fidelity) network protocols and the like.

[0102] like Figure 2a FIG. 1 is a schematic diagram of the structure of the electronic device 100. Optionally, the electronic device 100 may be a terminal, which may also be referred to as a terminal device. The terminal may be a cellular phone or a tablet computer, etc., which is not limited in this application. It should be noted that the schematic diagram of the structure of the electronic device 100 may be applicable to Figures 1a - 1b It should be understood that Figure 2a The electronic device 100 shown is only one example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 2a The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0103] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a temperature sensor, a motion sensor, a barometric pressure sensor, a magnetic sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0104] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0105] Among them, the controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0106] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0107] The USB interface 130 is an interface that complies with the USB standard specifications. Specifically, it can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. This interface can also be used to connect other electronic devices, such as AR devices, etc.

[0108] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input of the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device through the power management module 141.

[0109] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives the input from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160, etc.

[0110] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0111] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: the antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0112] The mobile communication module 150 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0113] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc.

[0114] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technologies.

[0115] The electronic device 100 implements the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change the display information.

[0116] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0117] In the embodiments of the present application, the display screen 194 may display a shooting preview interface, a photo-taking image interface, etc. It should be noted that in the embodiments of the present application, the shooting preview interface refers to an interface through which the user can view the images captured in real time by the desk lamp camera through the display screen 194.

[0118] The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to implement the storage capacity expansion of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function.

[0119] The internal memory 121 can be used to store computer-executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121, such as enabling the electronic device 100 to implement the collaborative working method in the embodiments of the present application. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.). In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0120] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc., such as music playback, recording, etc.

[0121] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0122] The speaker 170A, also called a "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or hands-free calls through the speaker 170A. In some embodiments, the electronic device 100 can be provided with multiple speakers 170A.

[0123] The receiver 170B, also called a "handset", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be received by bringing the receiver 170B close to the human ear.

[0124] The microphone 170C, also known as a "microphone" or "transmitter", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak close to the microphone 170C with their mouth to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and implement functions such as directional recording.

[0125] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0126] The pressure sensor is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, the pressure sensor can be provided on the display screen 194. The electronic device 100 can also calculate the touch position according to the detection signal of the pressure sensor.

[0127] The touch sensor, also known as a "touch panel". The touch sensor can be provided on the display screen 194, and together with the display screen 194, it forms a touch screen, also known as a "touch screen". The touch sensor is used to detect touch operations acting on it or nearby. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event.

[0128] The keys 190 include a power-on key (or power key), volume keys, etc. The keys 190 can be mechanical keys or touch keys. The electronic device 100 can receive key inputs and generate key signal inputs related to the user settings and function controls of the electronic device 100.

[0129] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts or touch vibration feedback. For example, touch operations on different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects.

[0130] The indicator 192 can be an indicator light, which can be used to indicate the charging state, power change, or can also be used to indicate messages, missed calls, notifications, etc.

[0131] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of this application, taking the Android system with a layered architecture as an example, the software structure of the electronic device 100 will be exemplarily described.

[0132] Figure 2b It is a block diagram of the software structure of the electronic device 100 in the embodiments of this application.

[0133] The layered architecture of the electronic device 100 divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom are the application layer, the application framework layer, the system layer, the HAL layer (Hardware Abstract Layer), and the kernel layer.

[0134] The application layer may include a series of application packages.

[0135] As Figure 2b shown, the application packages may include call, video, Bluetooth, camera, WLAN, education applications, device management applications, etc. The application packages may also include applications such as calendar, map, navigation, music, short message, etc.

[0136] Among them, the education application can be used to provide online education functions for users, such as online vocabulary recognition, online reading aloud, online submission of homework, etc.

[0137] In some examples, the device management application can be used to bind IOT (Internet of Things) devices such as table lamps. In some examples, the education application can implement the binding of IOT (Internet of Things) devices such as table lamps.

[0138] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0139] As Figure 2b shown, the application framework layer may include camera service, authentication service, hardware virtualization service, device management service, transmission management service, etc.

[0140] Among them, the camera service can be used to call the camera (including the front camera and / or the rear camera) in response to the request of the application.

[0141] In the embodiments of the present application, the camera service can be used to call the virtual camera on the electronic device side in response to a request from an application, that is, to call the camera in the IOT device.

[0142] The authentication service is used to provide security permission management capabilities.

[0143] The hardware virtualization service can be used to establish a logical channel between the electronic device side (i.e., the central device side) and the IOT device side, and provide the ability to virtualize the camera.

[0144] The device management service can be used for IOT device discovery and management, and provide far-field (i.e., cloud) IOT device information and near-field (i.e., nearby connectable) IOT device information for application programs such as education applications.

[0145] The transmission management service can be used to establish a physical transmission channel and provide data transmission capabilities.

[0146] In addition, it may also include a window manager, a content provider, a view system, a telephone manager, a resource manager, a notification manager, etc.

[0147] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0148] The content provider is used to store and obtain data, and make this data accessible to application programs. The data may include videos, images, audio, dialed and answered calls, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build application programs. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures. The telephone manager is used to provide the communication functions of the electronic device 100. For example, the management of call states (including connection, disconnection, etc.). The resource manager provides various resources for application programs, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables application programs to display notification information in the status bar, can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction.

[0149] The system library and the runtime layer (i.e., the system layer) include the system library and the Android Runtime.

[0150] The Android Runtime includes the core library and the virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system.

[0151] The core library consists of two parts: one is the functional functions that the Java language needs to call, and the other is the core library of Android. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0152] In the embodiment of the present application, the Android Runtime also includes a virtual camera adaptation layer, which provides the ability to register virtual cameras.

[0153] The system libraries in the system layer can include multiple functional modules. For example: multimedia platform, graphics and image processing library, codec, etc.

[0154] The multimedia platform can be used for multimedia management, supporting the playback and recording of various common audio and video formats, as well as static image files, etc. The multimedia platform can support a variety of audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0155] The graphics and image processing library can be used to implement graphics drawing, image rendering, synthesis, and layer processing, etc.

[0156] The codec can be used to implement the encoding and decoding operations of audio data and video data.

[0157] The HAL layer is the interface layer between the operating system kernel and the hardware circuit. The HAL layer includes but is not limited to: audio HAL, sensor HAL, modem HAL, camera HAL, virtual camera HAL.

[0158] Among them, the audio HAL is used to process audio streams. For example, it processes audio streams such as noise reduction and directional enhancement. The camera HAL is used to process the image stream corresponding to the camera on the electronic device side, and the virtual camera HAL is used to process the image stream corresponding to the virtual camera registered on the electronic device side, that is, to process the image stream collected by the camera on the IOT device side.

[0159] The kernel layer is the layer between hardware and software. The kernel layer at least includes display drivers, camera areas, audio drivers, network drivers (such as Wi-Fi drivers), CPU drivers, USB drivers, storage drivers, print drivers, etc. Among them, the hardware at least includes a processor, a display screen, a Wi-Fi module, etc.

[0160] It can be understood that Figure 2bThe layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which are not limited in the present application.

[0161] As Figure 3a shown is a schematic diagram of the hardware structure of the Internet of Things device 200. It should be noted that the schematic diagram of the structure of the Internet of Things device 200 can be applicable to Figures 1a - 1b the desk lamp in Figure 3a . It should be understood that the Internet of Things device 200 shown is only an example of an electronic device, and the Internet of Things device 200 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. Figure 3a The various components shown in

[0162] can be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0163] The processor 210 may include one or more processing units. For example, the processor 210 may include a GPU, an ISP, a controller, a memory, a video codec, etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0164] The controller may be the nerve center and command center of the Internet of Things device 200. The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching instructions and executing instructions.

[0165] The camera 201 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. Taking the desk lamp as an example, the camera 201 can be set on the desk lamp bracket for downward image acquisition.

[0166] The Internet of Things device 200 can implement the shooting function through an ISP, a camera 201, a video codec, a GPU, etc.

[0167] The ISP is used to process the data fed back by the camera 201. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also optimize the noise, brightness, and skin color of the image through algorithms. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 201.

[0168] The wireless communication module 202 can provide solutions for wireless communications including WLAN (such as Wi-Fi networks), Bluetooth (BT), etc. applied to the Internet of Things device 200. In some embodiments, the antenna of the Internet of Things device 200 is coupled to the wireless communication module 202, enabling the Internet of Things device 200 to communicate with the network and other devices through wireless communication technologies.

[0169] The memory 203 can be used to store computer-executable program code, and the executable program code includes instructions. The processor 210 executes various functional applications and data processing of the Internet of Things device 200 by running the instructions stored in the memory 203, such as enabling the Internet of Things device 200 to implement the collaborative working method in the embodiments of the present application.

[0170] The Internet of Things device 200 can implement audio functions such as music playback through an audio module 204, a speaker 212, etc.

[0171] The USB interface 205 is an interface that conforms to the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 205 can be used to connect a charger to charge the Internet of Things device 200, and can also be used to transfer data between the Internet of Things device 200 and peripheral devices.

[0172] The charging management module 206 is used to receive a charging input from a charger. While charging the battery 208, the charging management module 206 can also supply power to the Internet of Things device 200 through the power management module 207.

[0173] The power management module 207 is used to connect the battery 208, the charging management module 206, and the processor 210. The power management module 207 receives inputs from the battery 208 and / or the charging management module 206 and supplies power to the processor 210, the memory 203, the camera 201, the wireless communication module 202, and the lighting device 209, etc.

[0174] The button 211 includes a power-on button (or power key), etc.

[0175] The software system of the Internet of Things device 200 can adopt a layered architecture or other architectures, etc. In the embodiments of the present application, taking the layered architecture as an example, the software structure of the Internet of Things device 200 is exemplarily described.

[0176] Figure 3b It is a software structure block diagram of the Internet of Things device 200 in the embodiments of the present application.

[0177] The layered architecture of the Internet of Things device 200 divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system of the Internet of Things device 200 is divided into three layers, from top to bottom, namely the application layer, the application framework layer, the system layer, and the kernel layer.

[0178] As Figure 3b shown, the application layer may include device application services, and the device application services can be understood as system-level applications, and the device application services are started after the Internet of Things device 200 system is started.

[0179] As Figure 3b shown, the application framework layer may include device interconnection services, hardware abstraction services, resource managers, etc.

[0180] The device interconnection service can be used to establish a physical transmission channel, provide data transmission capabilities, and at the same time can manage the start switch of the hardware abstraction service.

[0181] The hardware abstraction service can be used to establish a logical channel between the electronic device side (i.e., the central device side) and the IOT device, provide the ability of virtualized cameras, and at the same time provide the camera open interface of the IOT device.

[0182] The resource manager can provide various resources for the application program.

[0183] As Figure 3b shown, the system layer may include a multimedia platform, a graphics and image processing library, a codec, a device adaptation module, etc.

[0184] The multimedia platform can be used for the management of multimedia, support various common audio, video, and static image files, etc. The multimedia platform can support various audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0185] The graphics and image processing library can be used to implement graphics drawing, image rendering, synthesis, and layer processing, etc.

[0186] Codecs can be used to implement encoding and decoding operations on audio data and video data.

[0187] The device adaptation module can implement the interface of the hardware abstract service, can provide device information and capability query, and can also provide functions for performing related operations on the IOT device side, such as opening the camera, taking pictures, previewing, etc.

[0188] It is understandable that the electronic device 100 and the Internet of Things device 200 include hardware and / or software modules corresponding to the execution of each function in order to implement the collaborative working method in the embodiment of the present application. In combination with the algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0189] like Figure 4a The following is a schematic diagram of the interaction between the modules. Figure 4a , the embodiment of the present application provides a method flow for the collaborative working of a tablet and a desk lamp, specifically including:

[0190] 0. Device service initialization phase

[0191] S0.1, in response to the user operation, the device application service of the desk lamp is started and the device interconnection service is loaded.

[0192] For example, the user operation may be an operation of turning on the power of the desk lamp. In response to the user operation, the desk lamp system starts, the device application service starts, and the device interconnection service is loaded. The device interconnection service can be used to establish a physical transmission channel between the tablet and the desk lamp to provide data transmission capabilities.

[0193] S0.2, the device interconnection service of the desk lamp loads the hardware abstraction service.

[0194] The device interconnection service can also control the startup of the hardware abstraction service. For example, after the device interconnection service is started, the device interconnection service can load the hardware abstraction service in the form of a plug-in. Among them, the hardware abstraction service can be used to establish a logical channel between the tablet and the desk lamp, provide the ability to virtualize the camera, and also provide an open interface for the desk lamp camera.

[0195] Reference Figure 4bThe schematic diagram of module interaction shown, the hardware abstraction service can at least include a basic component and a camera component. Among them, in the device service initialization stage, the device interconnection service first loads the basic component and initializes the basic component. After the basic component is initialized, it can interact with the device adaptation module of the table lamp to obtain device information and virtualization capability information. Exemplarily, the device information includes but is not limited to device name, device identifier, device type, etc. Exemplarily, the virtualization capability information includes but is not limited to whether virtualized camera is supported, whether virtualized microphone is supported, etc.

[0196] Among them, the table lamp has the ability to support virtualized camera, which can be understood as the camera of the table lamp allows other electronic devices (such as a tablet) to call, that is, it can be understood that the camera of the table lamp allows to be used as the virtual camera of other electronic devices.

[0197] After the basic component obtains the device information and capability information of the table lamp, if the table lamp has the ability to support virtualized camera, the basic component loads the camera component to provide the ability of virtualized camera. At this time, the basic component can prepare for establishing a negotiation channel to wait for establishing a negotiation channel with the tablet to negotiate network connection related information (including but not limited to IP address and port, etc.). Among them, when the basic component prepares for establishing a negotiation channel, a session service (SessionServer) is created, and the session name of the session service is sent to the device interconnection service to wait for the transmission management service on the tablet side to establish a negotiation channel with the device interconnection service on the table lamp side.

[0198] 1. Device discovery stage

[0199] S1.1, in response to a user operation, the education APP of the tablet sends a device discovery instruction to the device management service.

[0200] Among them, the user operation can be the operation that the user clicks the function option that needs to call the virtual camera in the education APP. Exemplarily, the user operation can be the operation that the user clicks the point reading function, word search function, homework function, and photo taking function in the education APP.

[0201] The education APP of the tablet receives the user operation and, in response to this operation, sends a device discovery instruction to the device management service of the tablet. Among them, the device discovery instruction is used to indicate to find IOT devices that can establish a connection with the tablet. Exemplarily, the device discovery instruction can include but is not limited to instruction type and device type to be discovered. In this embodiment, the device discovery instruction is specifically used to find a table lamp that can establish a connection with the tablet.

[0202] S1.2, the device management service in the tablet calls the authentication service to authenticate the education APP and obtains the authentication result of the education APP.

[0203] After receiving the device discovery instruction, the device management service can obtain the name (or identifier) of the education APP based on the existing mechanism of the Android system, and perform APP authentication on the education APP according to the name of the education APP. Among them, the device management service can call the authentication service to authenticate the education APP to obtain the authentication result of the education APP.

[0204] In this embodiment, the device management API corresponding to the device management service and the hardware virtualization API corresponding to the hardware virtualization service are also provided in the tablet side application framework layer. To implement the technical solution provided in this embodiment, the education APP needs to be registered on a relevant platform (such as the platform provided by the tablet manufacturer), adapt to the frameworks of the device management service, the hardware virtualization service, and the transmission management service, and apply for the permissions of the device management API and the hardware virtualization API.

[0205] Exemplarily, the authentication service accesses the authentication server to authenticate the education APP through the authentication server, including but not limited to whether it is registered on a relevant platform, whether it has adapted to relevant frameworks, and whether it has applied for relevant API permissions.

[0206] Another exemplarily, the authentication service can authenticate the education APP according to the local white list.

[0207] After the authentication service obtains the authentication result of the education APP (authentication success or authentication failure), it sends the authentication result to the device management service.

[0208] S1.3. When the education APP is successfully authenticated, the device management service in the tablet sends a device search instruction to the transmission management service.

[0209] If the education APP is successfully authenticated, the device management service sends a device search instruction to the transmission management service. Among them, the device search instruction can include but not limited to the instruction type, the device type to be searched, and the search method. Exemplarily, the search method includes but not limited to near-field device scanning and obtaining device information from the cloud server. In this implementation, the device type to be searched is a table lamp.

[0210] S1.4. The transmission management service in the tablet obtains the near and far field device list according to the device search instruction and sends the near and far field device list to the device management service.

[0211] The far-field and near-field device lists include the far-field device list and the near-field device list. Among them, the far-field devices included in the far-field device list refer to the registered devices obtained from the cloud server, and the near-field devices included in the near-field device list refer to the devices scanned through near-field communication. In the far-field device list and the near-field device list, the device information includes but is not limited to device name, device identifier, device type, etc.

[0212] When the transmission management service receives a device search instruction, it performs relevant device search operations according to the device type to be searched and the search method carried in the device search instruction, such as performing a near-field device scanning operation and an operation of obtaining relevant device information in the cloud server, obtains the far-field device list and the near-field device list, and sends the far-field device list and the near-field device list to the device management service.

[0213] S1.5. The device management service in the tablet filters devices according to the far-field and near-field device lists, and reports the filtered device information to the education APP.

[0214] The device management service filters devices according to the far-field device list and the near-field device list, determines the table lamp information that can be linked with the tablet, and sends it to the education APP. Among them, the device management service can perform an intersection operation on the far-field device list and the near-field device list, filter out the table lamps that only exist in the far-field device list or only exist in the near-field device list, and use the table lamps that exist in both the far-field device list and the near-field device list as the table lamp information that can be linked with the tablet. In this way, the device management service can filter out the table lamps that are not registered in the cloud server and can also filter out the table lamps that cannot perform near-field communication with the tablet.

[0215] In another alternative implementation, whether the tablet and the table lamp are in the same local area network or not, the transmission management service of the tablet can obtain the communication device list and the registered device list according to the device search instruction. Among them, the devices included in the communication device list refer to the devices scanned through near-field communication or far-field communication, and the devices included in the registered device list refer to the registered devices obtained from the cloud server. The near-field devices included in the near-field device list refer to the devices scanned through near-field communication. In the communication device list and the registered device list, the device information includes but is not limited to device name, device identifier, device type, etc.

[0216] The transmission management service in the tablet sends the communication device list and the registered device list to the device management service. The device management service filters the devices based on the communication device list and the registered device list, and reports the filtered device information to the education APP. Among them, the device management service can perform an intersection operation on the communication device list and the registered device list, filter out the table lamps that only exist in the communication device list or only exist in the registered device list, and use the table lamps that exist in both the communication device list and the registered device list as the table lamp information that can be linked with the tablet. In this way, the device management service can filter out the table lamps that are not registered in the cloud server and can also filter out the table lamps that cannot perform near-field communication with the tablet.

[0217] 2. Virtual camera enabling stage

[0218] S2.1, the education APP of the tablet determines the table lamp to be linked.

[0219] The number of table lamps that can be linked with the tablet filtered by the device management service can be one or more. When the number of table lamps is one, the education APP defaults it as the table lamp to be linked; when the number of table lamps is multiple, the education APP can display a list of table lamps to be linked for the user to select from. In response to the user's selection operation, the education APP takes the table lamp selected by the user as the table lamp to be linked.

[0220] It should be noted that the step of the education APP determining the table lamp to be linked can also be classified into the device discovery stage, and this embodiment does not limit this.

[0221] S2.2, the education APP of the tablet performs device verification, device connection on the table lamp, and obtains the ability of the table lamp to support virtualized cameras.

[0222] S2.3, the education APP of the tablet sends a virtual camera enabling request to the hardware virtualization service.

[0223] After the education APP of the tablet obtains the ability of the table lamp to support virtualized cameras, it sends a virtual camera enabling request to the hardware virtualization service. Among them, the virtual camera enabling request is used to indicate registering a virtual camera in the virtual camera HAL. The virtual camera enabling request can include but is not limited to the request type, device name, device identifier, device type, and the identifier of the virtual camera.

[0224] S2.4, the hardware virtualization service of the tablet registers the virtual camera with the virtual camera HAL.

[0225] After receiving the virtual camera enabling request, the hardware virtualization service registers the corresponding virtual camera with the virtual camera HAL according to the virtual camera enabling request.

[0226] S2.5, after the virtual camera registration is completed, the virtual camera HAL of the tablet sends a virtual camera enabling success indication to the education APP.

[0227] Combine the following Figure 4b The module interaction diagram shown in the figure provides a detailed explanation of the process of virtual camera enabling stage. Figure 4b The process of the virtual camera enabling phase mainly includes a device verification sub-phase (S301-S309), a device connection sub-phase (S310-S321), a device service capability request sub-phase (S322-S325) and a virtual camera enabling sub-phase (S326-S331).

[0228] Reference Figure 4b , the process of the virtual camera enabling stage includes the following steps:

[0229] S301, the education APP in the tablet sends a virtual camera enable instruction to the hardware virtualization API.

[0230] The virtual camera enabling instruction is used to instruct to enable the virtual camera, and the virtual camera enabling instruction may include but is not limited to an instruction type, a device name, a device identifier, and a device type.

[0231] S302: After receiving the virtual camera enable instruction, the hardware virtualization API in the tablet sends a device verification instruction to the interface scheduling module of the hardware virtualization service.

[0232] The device verification instruction is used to instruct to verify the device information carried in the virtual camera enable instruction. The device verification instruction may include but is not limited to the instruction type, device name, device identifier, and device type.

[0233] S303: The interface scheduling module of the hardware virtualization service in the tablet sends an APP authentication instruction to the authority management module of the hardware virtualization service.

[0234] After receiving the device verification instruction, the interface scheduling module of the hardware virtualization service first sends an APP authentication instruction to the permission management module of the hardware virtualization service to authenticate the APP that initiated the virtual camera enable instruction. The APP authentication instruction may include but is not limited to the name of the APP.

[0235] S304, the permission management module of the hardware virtualization service in the tablet performs APP authentication on the education APP.

[0236] Exemplarily, the permission management module can access the authentication server to authenticate the education APP through the authentication server, including but not limited to authenticating whether it is registered on the relevant platform, whether it is adapted to the relevant framework, and whether it has applied for the relevant API permissions. Among them, the permission management module can access the authentication server through the authentication service, and this embodiment does not limit this.

[0237] S305, when the permission management module of the hardware virtualization service in the tablet successfully authenticates the education APP, it sends an authentication success indication to the interface scheduling module.

[0238] After the permission management module obtains the authentication result of the education APP, if the education APP is successfully authenticated, it sends an authentication success indication to the interface scheduling module. If the education APP is authenticated failed, it sends an authentication failure indication to the hardware virtualization API, so that the hardware virtualization API returns an indication message that the APP has no permission to the education APP according to the authentication failure indication.

[0239] S306, when the interface scheduling module of the hardware virtualization service in the tablet determines that the education APP is successfully authenticated, it sends a device verification instruction to the device management module.

[0240] After the interface scheduling module of the hardware virtualization service receives the authentication success indication and determines that the education APP is successfully authenticated, it sends a device verification instruction to the device management module. Among them, the device verification instruction is used to perform a status verification on the device to be linked, and specifically used to perform a status verification on the desk lamp to be linked in this embodiment. Exemplarily, the device verification instruction may include but not limited to the instruction type, device name, device identifier, and device type.

[0241] S307, the device management module of the hardware virtualization service in the tablet sends a device information query instruction to the device profile module of the device management service.

[0242] Among them, the device profile module of the device management service stores the information of the currently online devices.

[0243] After the device management module of the hardware virtualization service receives the device verification instruction, it sends a device information query instruction to the device profile module of the device management service. The device information query instruction may include but not limited to the device name, device identifier, and device type.

[0244] S308, the device profile module of the device management service in the tablet returns the device information to the device management module of the hardware virtualization service.

[0245] If the device profile module of the device management service queries the corresponding device according to the device information query instruction, it returns the device information to the device management module of the hardware virtualization service. Among them, the returned device information may include, but is not limited to, device name, device identifier, device type, and online status.

[0246] If the device profile module of the device management service fails to query the corresponding device according to the device information query instruction, it returns a null value to the device management module of the hardware virtualization service to indicate that the corresponding device has not been queried. At this time, the device management module of the hardware virtualization service may send a device verification failure indication to the hardware virtualization API, so that the hardware virtualization API returns an indication message of device verification failure to the education APP according to the device verification failure indication.

[0247] S309. After receiving the device information, the device management module of the hardware virtualization service in the tablet sends a device verification success indication to the hardware virtualization API.

[0248] If the device management module of the hardware virtualization service receives the device information returned by the device profile module of the device management service, it sends a device verification success indication to the hardware virtualization API to indicate that the verification of the desk lamp to be linked is successful.

[0249] S310. The hardware virtualization API in the tablet sends a device connection request to the device management module of the hardware virtualization service.

[0250] After confirming that the verification of the desk lamp to be linked is successful, the hardware virtualization API sends a device connection request to the device management module of the hardware virtualization service. Among them, the device connection request is used to indicate establishing a network connection with the device to be linked, and specifically used to indicate establishing a network connection with the desk lamp to be linked in this embodiment. The device connection request may include, but is not limited to, request type, device name, device identifier, and device type.

[0251] S311. After receiving the device connection request, the device management module of the hardware virtualization service performs negotiation channel establishment preparation operations and sends a negotiation channel open request to the transmission management service.

[0252] After receiving the device connection request, the device management module of the hardware virtualization service prepares a negotiation channel. When the device management module prepares the negotiation channel, it creates a Session Server and sends the Session Name of the session service to the transmission management service. After preparing the negotiation channel, it sends a negotiation channel open request to the transmission management service. Among them, the negotiation channel open request is used to indicate establishing a negotiation channel, and the negotiation channel open request may include, but is not limited to, the peer device identifier (i.e., the desk lamp identifier) and the Session Name.

[0253] In this embodiment, the negotiation channel opening request is actively initiated by the tablet side, that is, the tablet needs to establish a connection with the table lamp. At this time, the table lamp can be understood as the server, and the tablet can be understood as the client that needs to access the server.

[0254] S312. The transmission management service in the tablet establishes a negotiation channel with the device interconnection service in the table lamp.

[0255] After receiving the negotiation channel opening request, the transmission management service interacts with the device interconnection service in the table lamp according to the Session Name to establish a negotiation channel. Among them, establishing a negotiation channel can specifically be creating a session and determining the session identifier.

[0256] S313. The device interconnection service in the table lamp sends an indication of successful establishment of the negotiation channel to the camera component in the hardware abstraction service.

[0257] After the negotiation channel is established, the device interconnection service in the table lamp sends an indication of successful establishment of the negotiation channel to the camera component in the hardware abstraction service to indicate that the negotiation channel is established and there is currently a device that needs to establish a connection. Among them, the indication of successful establishment of the negotiation channel can include, but is not limited to, device information that needs to establish a connection (i.e., tablet device information) and the session identifier.

[0258] S314. The transmission management service in the tablet sends an indication of successful establishment of the negotiation channel to the device management module of the hardware virtualization service.

[0259] After the negotiation channel is established, the transmission management service in the tablet sends an indication of successful establishment of the negotiation channel to the device management module of the hardware virtualization service to indicate that the negotiation channel is established and there is currently a device that needs to establish a connection. Among them, the indication of successful establishment of the negotiation channel can include, but is not limited to, device information that needs to establish a connection (i.e., table lamp device information) and the session identifier.

[0260] This embodiment does not limit the execution order of S313 and S314.

[0261] S315. The device management module of the hardware virtualization service in the tablet sends a device information negotiation request to the camera component of the hardware abstraction service in the table lamp based on the negotiation channel.

[0262] Among them, the device negotiation request can include, but is not limited to, device information (such as device name, device identifier, device type, etc.) and a control channel connection request.

[0263] S316. After receiving the device negotiation request, the camera component of the hardware abstraction service in the table lamp prepares the control channel and returns device negotiation information to the device management module of the hardware virtualization service in the tablet.

[0264] After the camera component of the hardware abstraction service in the table lamp receives the device negotiation request, it parses the device negotiation request to obtain the device information of the peer end for recording, and determines an IP address and port to be monitored according to the control channel connection request, and adds the IP address and port to the device negotiation information and returns it to the device management module of the hardware virtualization service in the tablet.

[0265] It should be noted that the device negotiation request and the device negotiation information are transmitted based on the established negotiation channel.

[0266] S317, after the device management module of the hardware virtualization service in the tablet receives the returned device negotiation information, it closes the negotiation channel.

[0267] Among them, closing the negotiation channel can specifically be closing the session. When the device management module of the hardware virtualization service in the tablet receives the returned device negotiation information, this session ends, and the device management module of the hardware virtualization service can close the corresponding session according to the session identifier.

[0268] S318, the device management module of the hardware virtualization service in the tablet sends a control channel opening request to the transmission management service.

[0269] The control channel opening request is used to indicate establishing a network communication connection with the table lamp. Among them, the control channel opening request may include but is not limited to communication protocols, source IP, source port, destination IP, and destination port, and the destination IP and destination port are the IP and port listened by the camera component of the hardware abstraction service in the table lamp.

[0270] S319, the transmission management service in the tablet connects to the control channel of the camera component of the hardware abstraction service in the table lamp and sends a control channel successful connection indication to the device management module of the hardware virtualization service in the tablet.

[0271] The transmission management service in the tablet receives the control channel opening request, establishes a control channel connection with the table lamp according to the information carried in the control channel opening request, that is, establishes a network communication connection between the tablet and the table lamp. Furthermore, the device management module of the hardware virtualization service in the tablet and the camera component of the hardware abstraction service in the table lamp can perform network communication based on this control channel.

[0272] After the control channel is successfully established, the transmission management service in the tablet sends a control channel successful connection indication to the device management module of the hardware virtualization service in the tablet. Among them, the control channel successful connection indication may include but is not limited to a connection success identifier and control channel related information.

[0273] S321, the device management module of the hardware virtualization service in the tablet sends a device connection successful indication to the hardware virtualization API.

[0274] Among them, the device connection success indication may include, but is not limited to, a connection success identifier and device information of the connected device.

[0275] S322. The hardware virtualization API in the tablet sends a device capability request to the device management module of the hardware virtualization service.

[0276] After receiving the device connection success indication, the hardware virtualization API sends a device capability request to the device management module of the hardware virtualization service. Among them, the device capability request can be used to request and obtain the virtualization capability information of the peer device (i.e., the table lamp). Exemplarily, the virtualized device capability information includes, but is not limited to, whether it supports virtualized cameras, whether it supports virtualized microphones, etc.

[0277] S323. The device management module of the hardware virtualization service in the tablet sends a device capability request to the camera component of the hardware abstraction service in the table lamp through the control channel.

[0278] S324. The camera component of the hardware abstraction service in the table lamp returns device capability information to the device management module of the hardware virtualization service in the tablet through the control channel.

[0279] In this embodiment, the returned device capability information of the table lamp may at least include the ability to support virtualized cameras and the camera identifier of the table lamp.

[0280] S325. The device management module of the hardware virtualization service in the tablet sends the device capability information to the hardware virtualization API.

[0281] The device management module of the hardware virtualization service in the tablet sends the received device capability information to the hardware virtualization API so that the hardware virtualization API can know whether the table lamp has the ability to support virtualized cameras.

[0282] S326. The hardware virtualization API in the tablet sends a virtual camera enable request to the device management module of the hardware virtualization service.

[0283] After the hardware virtualization API in the tablet learns that the table lamp has the ability to support virtualized cameras, it sends a virtual camera enable request to the device management module of the hardware virtualization service. Among them, the virtual camera enable request may include, but is not limited to, the request type and the camera identifier of the table lamp.

[0284] S327. The device management module of the hardware virtualization service in the tablet registers a virtual camera in the virtual camera HAL.

[0285] Among them, after the device management module of the hardware virtualization service receives a virtual camera enabling request, it sends a virtual camera registration request to the virtual camera HAL. The virtual camera registration request may include, but is not limited to, the request type and the camera identifier of the table lamp. After the virtual camera HAL receives the virtual camera registration request, it registers a virtual camera driver for the camera of the table lamp in the virtual camera HAL, assigns a camera ID (i.e., the virtual camera ID) to the camera of the table lamp, and registers this camera ID in the system. Thus, a mapping relationship between the camera of the table lamp and the virtual camera is established in the virtual camera HAL.

[0286] S328, the device management module of the hardware virtualization service in the tablet sends a service status update indication to the camera component of the hardware abstraction service in the table lamp.

[0287] The service status update indication is used to indicate that the camera component of the hardware abstraction service in the table lamp updates its virtualization service status. Among them, the virtualization service status may include an occupied state, an unoccupied state, or it can be said to include a registered state, an unregistered state. Exemplarily, the service status update indication may include, but is not limited to, the device information of the peer device (i.e., the table lamp), the hardware identifier (such as the camera identifier of the table lamp), and the virtualization service status corresponding to the hardware identifier.

[0288] S329, the camera component of the hardware abstraction service in the table lamp updates the service status according to the service status update indication.

[0289] When the virtualization service status corresponding to the table lamp camera indicated in the service status update indication is the occupied state (or the registered state), the camera component updates the virtualization service status corresponding to its lamp camera to the occupied state (or the registered state).

[0290] S330, the device management module of the hardware virtualization service in the tablet sends a virtual camera enabling success indication to the hardware virtualization API.

[0291] Among them, the virtual camera enabling success indication may include, but is not limited to, an enabling success identifier (or a virtualization success identifier), the camera identifier of the table lamp, and the camera ID corresponding to the virtual camera (or the camera ID corresponding to the table lamp camera).

[0292] This embodiment does not limit the execution order of S328 and S330.

[0293] S331, the hardware virtualization API in the tablet sends a virtual camera enabling success indication to the education APP.

[0294] 3. Virtual camera preview access stage

[0295] S3.1, The hardware virtualization API in the tablet sends a virtual camera access instruction to the camera service.

[0296] The virtual camera access instruction refers to an instruction for invoking a virtual camera. Among them, the virtual camera access instruction may include, but is not limited to, instruction type, virtual camera ID, and camera configuration parameters. The configuration parameters include, but are not limited to, camera resolution and acquisition frame rate.

[0297] S3.2, The camera service in the tablet sends an image preview request to the virtual camera HAL according to the virtual camera access instruction.

[0298] After receiving the virtual camera access instruction, the camera service generates a corresponding image preview request according to the virtual camera ID and sends it to the virtual camera HAL. Among them, the image preview request is used to request the preview image data stream. Exemplarily, the image preview request may include, but is not limited to, request identifier, virtual camera ID, and camera configuration parameters, etc.

[0299] S3.3, The virtual camera HAL in the tablet sends an image preview request to the hardware virtualization service.

[0300] After receiving the image preview request, the virtual camera HAL determines the matching virtualized hardware identifier according to the virtual camera ID carried in the image request. In this embodiment, the virtual camera HAL determines the linked desk lamp camera according to the virtual camera ID and the mapping relationship between the virtual camera ID and the desk lamp camera, and generates a corresponding image request according to the determined virtualized hardware identifier and sends it to the hardware virtualization service. Exemplarily, the image preview request may include, but is not limited to, request identifier, device information (i.e., desk lamp information), virtualized hardware identifier (i.e., desk lamp camera identifier), and camera configuration parameters, etc.

[0301] S3.4, The hardware virtualization service in the tablet sends an image preview request to the transmission management service.

[0302] The hardware virtualization service sends the image preview request to the transmission management service. Among them, the image preview request may include, but is not limited to, request identifier, device information (i.e., desk lamp information), virtualized hardware identifier (i.e., desk lamp camera identifier), and camera configuration parameters, etc.

[0303] When the hardware virtualization service in the tablet sends an image preview request to the transmission management service, if it is found that no data channel has been established with the desk lamp, a data channel establishment request is generated and sent to the transmission management service. The data channel establishment request is used to indicate data transmission with the desk lamp. Among them, the data channel establishment request may include, but is not limited to, session identifier, connection information, data encoding and decoding mode, etc.

[0304] The transmission management service in the tablet receives a data channel establishment request, and establishes a data channel connection with the table lamp according to the information carried in the data channel establishment request, that is, a data channel is established between the tablet and the table lamp. Furthermore, the transmission management service in the tablet and the device interconnection service in the table lamp can transmit various data based on this data channel, including but not limited to image data.

[0305] After the data channel is successfully established, the transmission management service in the tablet sends a data channel successful connection indication to the hardware virtualization service in the tablet, and the device interconnection service in the table lamp sends a data channel successful connection indication to the camera component in the hardware abstraction service. Among them, the data channel successful connection indication may include but not limited to a connection success identifier and data channel related information.

[0306] S3.5, the transmission management service in the tablet transmits an image preview request to the device interconnection service of the table lamp.

[0307] The transmission management service in the tablet determines the corresponding control channel according to the device information carried in the image preview request, and transmits the image preview request to the device interconnection service of the table lamp in this control channel.

[0308] S3.6, the device interconnection service in the table lamp sends an image preview request to the camera driver.

[0309] After the device interconnection service in the table lamp receives the image preview request, it determines the corresponding hardware driver according to the virtualized hardware identifier (in this embodiment, it is to determine the camera driver), and sends the corresponding image preview request to the camera driver.

[0310] S3.7, the camera driver in the table lamp drives the camera to collect images, and transmits the preview image data to the hardware virtualization service of the tablet through the data channel.

[0311] The camera driver drives the camera to turn on, and drives the camera to collect images according to the camera configuration parameters carried in the image preview request, obtains a preview image data stream, and sends the preview image data stream to the device interconnection service through the hardware abstraction service, so that the device interconnection service continuously transmits the preview image data stream to the hardware virtualization service in the tablet through the data channel. Among them, the process of packet splitting, packet assembling, encoding and decoding of the preview image data stream will not be elaborated here.

[0312] S3.8, the hardware virtualization service in the tablet sends the preview image data to the virtual camera HAL.

[0313] The hardware virtualization service continuously receives the preview image data stream and sends the preview image data stream to the virtual camera HAL.

[0314] S3.9, the virtual camera HAL in the tablet sends the preview image data to the camera service.

[0315] At this time, the virtual camera HAL continuously obtains the preview image data collected by the desk lamp camera and continuously sends the preview image data to the camera service.

[0316] S3.10, the camera service in the tablet sends the preview image data to the education APP.

[0317] S3.11, the education APP in the tablet displays the preview image.

[0318] After receiving the preview image data stream through the camera service, the education APP can display the preview image in the corresponding interface.

[0319] 4. Virtual camera photographing stage

[0320] S4.1, in response to the received user operation, the education APP in the tablet sends a photographing request to the hardware virtualization service.

[0321] The user operation can be, for example, an operation of clicking the photographing option. In response to the received user operation, the education APP in the tablet sends a photographing request to the hardware virtualization service. Among them, the photographing request may include, but is not limited to, the serial number of the captured image, device information (i.e., desk lamp information), virtualized hardware identifier (i.e., desk lamp camera identifier), and camera configuration parameters, etc. The camera configuration parameters include, but are not limited to, the image resolution.

[0322] Among them, the photographing request can also carry a task identifier to ensure the orderly management of multiple photographing tasks.

[0323] S4.2, the hardware virtualization service in the tablet sends the photographing request to the transmission management service.

[0324] S4.3, the transmission management service in the tablet will transmit the photographing request to the device interconnection service of the desk lamp.

[0325] The transmission management service in the tablet determines the corresponding control channel according to the device information carried in the image preview request and transmits the image preview request to the device interconnection service of the desk lamp in this control channel.

[0326] S4.4, the device interconnection service in the desk lamp sends the photographing request to the camera driver.

[0327] After receiving the image preview request, the device interconnection service in the desk lamp determines the corresponding hardware driver according to the virtualized hardware identifier (in this embodiment, it is to determine the camera driver) and sends the corresponding photographing request to the camera driver.

[0328] S4.5, the camera driver in the desk lamp drives the camera to capture an image and transmits the captured image data to the hardware virtualization service of the tablet through the data channel.

[0329] The camera driver drives the camera to collect an image according to the camera configuration parameters carried in the photographing request, obtains the captured image data, and sends the captured image data to the device interconnection service through the hardware abstraction service, so that the device interconnection service continuously transmits the captured image data to the hardware virtualization service of the tablet in the data channel. Regarding the processes of packetizing, depacketizing, encoding, and decoding the captured image data, they will not be elaborated here.

[0330] S4.6. The hardware virtualization service in the tablet sends the captured image data to the education APP.

[0331] S4.7. The education APP in the tablet displays the captured image.

[0332] After receiving the captured image through the hardware virtualization service, the education APP can display the captured image in the corresponding interface.

[0333] In this embodiment, the virtual camera preview access stage is implemented based on the Android native camera framework, and the virtual camera photographing stage is implemented based on a proprietary virtualized camera framework, so that the processing path involved in the virtual camera photographing stage is shorter and the photographing delay is smaller. At the same time, since the image preview is still implemented based on the Android native camera framework, the education APP requires fewer modifications to adapt to the technical solution provided in this embodiment.

[0334] It should be noted that the stage division in the above process is only an exemplary expression, and the embodiments of the present application do not make any limitations in this regard. Moreover, after the preview image is displayed on the tablet during the execution of the virtual camera preview access stage process, the process of real-time displaying the preview image and the process of the virtual camera photographing stage can be executed simultaneously. For the parts not elaborated in detail in the above process, reference can be made to the existing technologies and will not be elaborated here.

[0335] Figure 4a The communication architecture of the collaborative working system is given, which is used to complete the management of the virtual camera (i.e., the table lamp camera) by the tablet, the control command interaction between the tablet and the table lamp, and the return and processing of image data, etc.

[0336] It should be pointed out that the instructions, requests, etc. for cross-device transmission (i.e., transmission between the tablet and the table lamp) need to be encapsulated based on communication protocols and parameter sequences, etc., which will not be elaborated in this embodiment. The hardware virtualization service in the tablet can also manage the life cycles of the preview image stream and the captured image by dynamically allocating memory and dynamically destroying memory.

[0337] In addition, it should be pointed out that before executing the collaborative working method provided in this embodiment, the education APP needs to be bound to the table lamp and register the table lamp in the cloud server.

[0338] The embodiments of the present application provide a framework solution for an Android system device to use the camera of an external device for taking pictures. This solution can not only be applied in educational scenarios, but also be applicable to other devices equipped with cameras. These devices can share their camera capabilities with Android system devices such as mobile phones and tablets to achieve the interconnection and interoperability between Android system devices and these devices.

[0339] Figures 5a - 5b An application scenario is exemplarily shown. As Figure 5a shown in (1) below, on the tablet display interface 401, there are multiple application icons displayed. The user clicks on the educational application icon 4011. In response to the received user operation, the tablet launches the educational application, and the tablet displays the educational application interface, which can be referred to Figure 5a as shown in (2) below. As Figure 5a shown in (2) below, the tablet displays the educational application interface 402, and various function options of the educational application are displayed on the interface 402, including but not limited to word lookup function, point reading function, homework function, photo taking function, etc. When the user uses the photo taking function, the user clicks on the photo taking function option 4021. In response to the user operation, the tablet executes the processes of the device discovery stage, the virtual camera service enabling stage, and the virtual camera preview access stage.

[0340] Among them, in the device discovery stage, if the number of table lamps that can be linked with the tablet filtered by the device management service of the tablet is one, the tablet automatically executes the processes of the virtual camera service enabling stage and the virtual camera preview access stage, and displays an interface such as Figure 5b shown in (1) below. In the device discovery stage, if the number of table lamps that can be linked with the tablet filtered by the device management service of the tablet is multiple, the tablet displays a table lamp selection interface. Exemplarily, a list of table lamps to be linked is displayed on the table lamp selection interface. The user can perform a selection operation. In response to the user's selection operation, the educational application determines one table lamp to be linked, and continues to execute the processes of the virtual camera service enabling stage and the virtual camera preview access stage to display an interface such as Figure 5b shown in (1) below.

[0341] As Figure 5b shown in (1) below, an image preview window 4031 and a photo taking option 4032 are displayed in the interface 403, and a preview image captured in real time by the table lamp camera is displayed in the image preview window 4031. At this time, if the user clicks on the photo taking option 4032, in response to the user operation, the tablet executes the process of the virtual camera photo taking stage and displays an interface such as Figure 5b shown in (2) below. As Figure 5bThe interface 404 shown in (2) displays the image captured by the desk lamp camera in the image preview window 4041. At this time, if the user clicks the confirmation option 4041, the tablet responds to the user operation, saves the captured image, and continues to display, for example, Figure 5b the preview interface shown in (1) above. If the user clicks the cancel option 4042, the tablet responds to the user operation and can display, for example, Figure 5b the preview interface shown in (1) above.

[0342] It should be noted that the interface shown in Figure 5b (2) is only an exemplary example. The image captured by the desk lamp camera may not be displayed in the image preview window 4041, but in other areas of the interface. The image preview window 4041 continues to display the preview image captured in real time by the desk lamp camera. This application does not make any limitations in this regard.

[0343] The collaborative working method provided in this embodiment mainly elaborates a low-cost technical solution for realizing the online education function based on the combination of a tablet device and a desk lamp device. The following elaborates on the technical solution provided in this embodiment in combination with several different functions involved in online education.

[0344] Scenario 1

[0345] Referring to Figure 1a the schematic diagram of the application scenario shown, this scenario takes the word lookup function as an example to elaborate on the technical solution. When a student encounters an unfamiliar new word, they can point their finger below the new word and use the desk lamp camera to take a picture. The tablet recognizes the image to determine the content of the new word, and after completing the online word lookup, feeds back the meaning of the new word to the student through the display screen, such as displaying and announcing the explanation of the new word on the interface.

[0346] As Figure 6a shown is the interaction schematic diagram of each module. Referring to Figure 6a , the method flow for the tablet and the desk lamp to work together provided in the embodiment of this application specifically includes:

[0347] S501, in response to the user's operation of clicking the word lookup function, the tablet and the desk lamp execute the processes of device discovery phase, virtual camera enabling phase, and virtual camera preview access phase, and the tablet displays the preview interface.

[0348] Among them, for the processes of the device discovery phase, virtual camera enabling phase, and virtual camera preview access phase, reference can be made to the foregoing, and details will not be elaborated here.

[0349] It should be noted that during the virtual camera preview access phase, the hardware virtualization API in the tablet sends a virtual camera access instruction to the camera service, which carries camera configuration parameters. Among them, the camera configuration parameters may include, but are not limited to, image resolution and image acquisition frame rate. The desk lamp camera is set according to the received configuration parameters and captures preview image data according to the corresponding image resolution and image acquisition frame rate.

[0350] In this scenario, the tablet needs to accurately identify the preview image to determine the text content pointed by the user. Therefore, this scenario has relatively high requirements for the image quality of the preview image. For example, the image resolution can be set to 1080P. In this way, a higher word search success rate can be ensured.

[0351] S502, the education APP in the tablet performs finger recognition on the preview image.

[0352] Exemplarily, the education APP can perform finger recognition on each frame of the received preview image, or can perform finger recognition on the latest received preview image at regular intervals. This embodiment does not limit this.

[0353] Exemplarily, an image recognition algorithm can be integrated in the education APP to implement image recognition operations, or the education APP can also call an image recognition service to perform image recognition operations. This embodiment does not limit this.

[0354] Regarding the image recognition algorithm, reference can be made to the prior art, and this embodiment will not elaborate here.

[0355] S503, in response to the user's operation of pointing at a word with a finger, the education APP in the tablet recognizes the user's finger in the preview image.

[0356] When the user points at a certain word in the book with a finger, the desk lamp camera can capture the preview image of the finger pointing at the word, and then the education APP in the tablet can recognize the user's finger in the preview image.

[0357] Among them, when the education APP performs finger recognition on the preview image, if a finger is recognized, the position information of the finger in the preview image, such as coordinate information, etc., can be obtained.

[0358] S504, the education APP in the tablet determines the ROI (region of interest) image according to the position of the finger in the preview image.

[0359] After the educational APP recognizes the user's finger in the preview image, it can determine the ROI image based on the position information of the finger in the preview image. Among them, the educational APP can determine the ROI information according to the coordinate information of the finger in the preview image, and the ROI information includes but is not limited to the center point coordinates and the area range (such as width and height information). Furthermore, the educational APP can crop the ROI image in the preview image according to the ROI information.

[0360] S505. The educational APP on the tablet accurately recognizes the ROI image to determine the new words to be explained.

[0361] Exemplarily, an image recognition algorithm can be integrated in the educational APP to accurately recognize the ROI image, or the educational APP can also call an image recognition service to accurately recognize the ROI image to determine the new words to be explained. This embodiment does not make any limitations in this regard.

[0362] Regarding the image recognition algorithm, reference can be made to the existing technology, and this embodiment will not elaborate here.

[0363] S506. The educational APP on the tablet performs a word lookup operation on the new words to be explained and displays the definitions of the new words to be explained.

[0364] After determining the new words to be explained, the educational APP can perform an online word lookup operation or a word lookup operation in the database to obtain the definitions of the new words to be explained. Furthermore, the educational APP can display the definitions of the new words to be explained for the user to view. In addition, the educational APP can read aloud the displayed word definitions. This embodiment does not make any limitations in this regard.

[0365] Similarly, the user can also use a pointing tool such as a point-reading pen (or an indicating tool) to perform a pointing operation. This embodiment does not make any limitations in this regard. Correspondingly, the educational APP recognizes the pointing tool in the preview image to determine whether the user has an intention to look up words, and determines the ROI image according to the position information of the pointing tool in the preview image.

[0366] Similarly, the user can also use a pointing tool such as a finger or a point-reading pen to point to a picture in a book. Correspondingly, the educational APP determines the ROI image according to the position information of the finger or the point-reading pen and other pointing tools, performs picture content recognition on the ROI image, displays the corresponding definitions of the picture, and can also read aloud the displayed picture definitions. This embodiment will not elaborate on this situation.

[0367] Figure 1a 、 Figures 7a - 7b Exemplarily shows an application scenario. Such as Figure 7aAs shown in (1), on the flat panel display educational APP interface 701, various function options of educational applications are displayed, including but not limited to word lookup function, point reading function, homework function, photo-taking function, etc. When the user clicks on the word lookup function option 7011, in response to the user operation, the flat panel executes the processes of device discovery phase, virtual camera service enabling phase, and virtual camera preview access phase.

[0368] Among them, in the device discovery phase, if the number of table lamps that can be linked with the flat panel filtered by the device management service of the flat panel is one, the flat panel automatically executes the processes of virtual camera service enabling phase and virtual camera preview access phase, and displays an interface such as Figure 7a shown in (2). In the device discovery phase, if the number of table lamps that can be linked with the flat panel filtered by the device management service of the flat panel is more than one, the flat panel displays a table lamp selection interface. Exemplarily, a list of table lamps to be linked is displayed on the table lamp selection interface. The user can perform a selection operation. In response to the user's selection operation, the educational application determines one table lamp to be linked, and continues to execute the processes of virtual camera service enabling phase and virtual camera preview access phase to display an interface such as Figure 7a shown in (2).

[0369] As Figure 7a shown in (2), an image preview window 7021 and a schematic diagram of word lookup function operation 7022 are displayed in the interface 702. Among them, a preview image captured in real time by the table lamp camera is displayed in the image preview window 7021. The user can perform a word-pointing or picture-pointing operation with reference to the schematic diagram of word lookup function operation 7022 to trigger the word lookup function. The educational APP performs word-pointing or picture-pointing recognition based on the preview image. Continuing to refer to Figure 7a in (2), when the user points to a word with a finger in a book, the table lamp camera captures a finger-pointing word preview image, and this finger-pointing word preview image is displayed in the image preview window 7021. Furthermore, the educational APP can recognize the user's finger in the preview image and determine the position information of the finger in the preview image, such as coordinate information, etc. The educational APP determines the ROI image based on the position of the finger in the preview image, and performs precise recognition on the ROI image to determine the new word to be explained. After the educational APP queries the definition of the new word, the corresponding new word definition is displayed on the interface, which can be referred to Figure 7b shown.

[0370] However, in the above process, in order to ensure the word lookup success rate, it is necessary to use the table lamp camera to continuously return a preview image stream with high resolution (such as 1080p), which requires a relatively high bandwidth, about 4 - 8 Mbps. In this way, the requirements for the hardware chip of the table lamp are higher, which also increases the cost of the table lamp.

[0371] To implement the solution for the tablet and the table lamp to cooperate in realizing the word lookup function and reduce the hardware cost of the table lamp, this embodiment also provides a technical solution. Among them, since the requirements for image resolution for finger recognition or finger pointing tool recognition and position recognition in the image are not high, during the virtual camera preview access stage, the table lamp camera continuously returns a preview image stream with a low resolution (such as 480p). If the education APP recognizes the user's finger pointing or finger pointing at a picture operation based on the preview image, the table lamp is triggered to take a high-resolution (such as 1080p) image, so that the education APP can perform accurate recognition based on this high-resolution image to determine the new word or picture to be explained. In this way, continuously returning a preview image stream with a low resolution by the table lamp camera has low requirements for bandwidth, only requiring a bandwidth of 0.5 - 1 Mbps, and only more bandwidth is needed when transmitting this high-resolution image. Therefore, this technical solution not only reduces the requirements of the table lamp for the hardware chip, reduces the cost of the table lamp, but also can ensure the success rate of word lookup.

[0372] As Figure 6b shown is the interaction schematic diagram of each module. Referring to Figure 6b , this application embodiment provides a method flow for the tablet and the table lamp to work together, specifically including:

[0373] S601, in response to the user's operation of clicking the word lookup function, the tablet and the table lamp execute the processes of the device discovery stage, the virtual camera enabling stage, and the virtual camera preview access stage, and the tablet displays the preview interface.

[0374] Among them, regarding the processes of the device discovery stage, the virtual camera enabling stage, and the virtual camera preview access stage, reference can be made to the previous text and will not be elaborated here.

[0375] It should be noted that during the virtual camera preview access stage, the hardware virtualization API in the tablet will carry camera configuration parameters when sending a virtual camera access instruction to the camera service. Among them, the camera configuration parameters can include but are not limited to image resolution and image acquisition frame rate. The table lamp camera is set according to the received configuration parameters and acquires preview image data according to the corresponding image resolution and image acquisition frame rate.

[0376] In this scenario, in order to reduce the bandwidth occupied by the preview image stream, during the virtual camera preview access stage, the table lamp camera can acquire preview images at a low resolution. For example, the hardware virtualization API in the tablet will carry a first configuration parameter when sending a virtual camera access instruction to the camera service, where the first configuration parameter includes a first image resolution (such as 480P). Furthermore, the table lamp camera is set according to the received configuration parameters and acquires preview image data according to the first image resolution and the corresponding image acquisition frame rate.

[0377] S602, the education APP on the tablet performs finger recognition on the preview image.

[0378] S603, in response to the user's operation of pointing at a word with a finger, the education APP on the tablet recognizes the user's finger in the preview image.

[0379] S604, the education APP on the tablet determines ROI information based on the position of the finger in the preview image, and generates a photographing request according to the ROI information.

[0380] Among them, the ROI information refers to the information used to determine the ROI, which may include but is not limited to the center point coordinates and the area range (such as width and height information).

[0381] S605, the tablet transmits the photographing request to the lamp side.

[0382] Among them, the photographing request may include but is not limited to the virtual camera ID corresponding to the lamp camera, the second configuration parameters of the lamp camera, and the ROI information. The second configuration parameters include but are not limited to the second image resolution, and the second image resolution is higher than the first image resolution. For example, the second image resolution is set to 1080P. In this way, the education APP can perform accurate recognition based on this high-resolution image to determine the unknown words or pictures to be explained.

[0383] S606, the camera of the lamp is set according to the second configuration parameters carried in the photographing request, takes a picture according to the second image resolution, and sends the taken picture to the hardware abstraction service.

[0384] S607, the hardware abstraction service determines the ROI image according to the ROI information.

[0385] The hardware abstraction service can crop out the ROI image from the taken picture according to the ROI information.

[0386] In an alternative embodiment, the photographing request includes but is not limited to the second configuration parameters, but does not include the ROI information. In this way, the tablet transmits the photographing request to the lamp side, the camera of the lamp is set according to the second configuration parameters carried in the photographing request, takes a picture according to the second image resolution, and returns the taken picture to the education APP on the tablet. Furthermore, the education APP can determine the ROI image according to the ROI information, for example, crop out the ROI image from the taken picture according to the ROI information.

[0387] S608, the lamp transmits the ROI image to the education APP on the tablet.

[0388] Compared with the lamp directly returning the high-resolution taken picture to the education APP on the tablet, returning the cropped ROI image to the education APP on the tablet can also reduce the data transmission volume and reduce the bandwidth occupation.

[0389] S609. The education APP on the tablet accurately identifies the ROI image to determine the unfamiliar words to be explained.

[0390] S610. The education APP on the tablet performs a word lookup operation on the unfamiliar words to be explained and displays the definitions of the unfamiliar words to be explained.

[0391] For the parts not explained in detail in this process, reference can be made to the previous text and will not be elaborated here.

[0392] Similarly, the user can also use a pointing tool such as a point-reading pen for pointing operations. This embodiment does not limit this. Correspondingly, the education APP identifies the pointing tool in the preview image to determine whether the user has an intention to look up words, and determines the ROI information according to the position information of the pointing tool in the preview image.

[0393] Similarly, the user can also use a pointing tool such as a finger or a point-reading pen to point to a picture in a book. Correspondingly, the education APP determines the ROI information according to the position information of the finger or the point-reading pen and other pointing tools, determines the ROI image in the captured image for the ROI information, performs picture content recognition on the ROI, displays the definition corresponding to the picture, and can also read aloud the displayed picture definition. For this situation, this embodiment will not be elaborated.

[0394] Regarding the application scenario of this process, reference can be made to Figure 1a 、 Figures 7a - 7b the application scenario shown. Refer to Figure 7a In (2) of Figure 7b as shown. When the user uses a finger to point to a word in a book, the desk lamp camera captures the finger-pointing preview image, and the finger-pointing preview image is displayed in the image preview window 7021. Furthermore, the education APP can recognize the user's finger in the preview image, determine the ROI information, and generate a photo-taking request according to the ROI information and the image high resolution, triggering the desk lamp camera to take a picture. The desk lamp camera takes a picture according to the image high resolution, and the desk lamp side crops the high-resolution captured image according to the ROI information to obtain the ROI image, and returns the ROI image to the education APP on the tablet. The education APP accurately identifies the ROI image to determine the unfamiliar words to be explained. After the education APP queries the definition of the unfamiliar word, the corresponding unfamiliar word definition is displayed on the interface, and reference can be made to

[0395] Scenario Two

[0396] Refer to Figure 1a the application scenario schematic diagram shown. In this scenario, the technical solution is elaborated taking the homework function as an example. When a student needs to submit homework online, the student can click to take a photo in the education APP, use the desk lamp camera to take a picture of the homework image, and upload the homework image to the database through the education APP.

[0397] As shown Figure 8 in the interaction schematic diagram of each module. Referring to Figure 8 , the embodiments of the present application provide a method flow for the tablet and the desk lamp to work together, which specifically includes:

[0398] S801. In response to the user's operation of clicking the homework function, the educational APP on the tablet displays a homework submission list.

[0399] It should be noted that the homework submission list refers to a list including multiple homework submission options, and one homework submission option corresponds to one homework, which can be referred to Figure 9a as shown in interface 704 in . Among them, the homework options can be divided by subject or by time, and this embodiment does not limit this.

[0400] If only one homework image needs to be submitted for the homework function of the educational APP, the educational APP will not display the homework submission list. At this time, in response to the user's operation of clicking the homework function, the tablet and the desk lamp execute the processes of the device discovery stage, the virtual camera enabling stage, and the virtual camera preview access stage, and the tablet displays a preview interface.

[0401] S802. In response to the user's click on the homework submission option, the tablet and the desk lamp execute the processes of the device discovery stage, the virtual camera enabling stage, and the virtual camera preview access stage, and the tablet displays a preview interface.

[0402] Among them, regarding the processes of the device discovery stage, the virtual camera enabling stage, and the virtual camera preview access stage, reference can be made to the foregoing, and details will not be repeated here.

[0403] It should be noted that in the virtual camera preview access stage, the virtual camera access instruction sent by the hardware virtualization API in the tablet to the camera service will carry camera configuration parameters. Among them, the camera configuration parameters can include but are not limited to image resolution and image acquisition frame rate. The desk lamp camera is set according to the received configuration parameters, and preview image data is collected according to the corresponding image resolution and image acquisition frame rate.

[0404] In this scenario, in order to reduce the bandwidth occupied by the preview image stream, in the virtual camera preview access stage, the desk lamp camera can collect preview images at a low resolution. For example, the hardware virtualization API in the tablet sends a virtual camera access instruction to the camera service, which carries a first configuration parameter, where the first configuration parameter includes a first image resolution (such as 480P). Furthermore, the desk lamp camera is set according to the received configuration parameters, and preview image data is collected according to the first image resolution and the corresponding image acquisition frame rate.

[0405] S803. In response to the user's operation of clicking the photo-taking option, the education APP in the tablet generates a photo-taking request.

[0406] When the user places homework, books, etc. within the acquisition area of the table lamp camera, the user can click the photo-taking option to trigger the table lamp camera to capture an image of the homework.

[0407] Among them, the photo-taking request may include, but is not limited to, the virtual camera ID corresponding to the table lamp camera and the second configuration parameters of the table lamp camera. The second configuration parameters include, but are not limited to, the second image resolution, and the second image resolution is higher than the first image resolution. For example, the second image resolution is set to 1080P. In this way, the education APP can upload a high-resolution image of the homework.

[0408] S804. The tablet transmits the photo-taking request to the table lamp side.

[0409] S805. The camera of the table lamp is set according to the second configuration parameters carried in the photo-taking request and captures an image according to the second image resolution.

[0410] S806. The table lamp transmits the captured image to the education APP in the tablet.

[0411] S807. The education APP in the tablet displays the captured image.

[0412] The education APP in the tablet receives the image of the homework captured by the table lamp camera and displays the image of the homework. If the user is satisfied with the captured image of the homework, the user can click the submission option to upload the image of the homework to the database. If the user is not satisfied with the captured image of the homework, the user can click the photo-taking option again to trigger the table lamp camera to capture the image of the homework again.

[0413] S808. In response to the user's operation of clicking the submission option, the education APP in the tablet uploads the captured image to the database.

[0414] For details not elaborated in this process, reference can be made to the previous text and will not be repeated here.

[0415] It should be noted that the above-mentioned image of the homework is only an exemplary illustration, and the user can click the photo-taking option to trigger the table lamp camera to capture other images. After the table lamp sends the captured image to the education APP in the tablet, the education APP can upload the received captured image to the corresponding database.

[0416] Figure 1a 、 9a - Figure 9c An application scenario is exemplarily shown. Such as Figure 9aAs shown in (1), on the flat panel display educational APP interface 701, various function options of educational applications are displayed, including but not limited to word lookup function, point reading function, homework function, photo-taking function, etc. When the user clicks on the word lookup function option 7012, in response to the user's operation, the educational APP on the flat panel displays a homework submission list, which can be referred to Figure 9a as shown in (2). As Figure 9a shown in the homework submission list interface 704 in (2), multiple homework submission options (such as submit homework 1, submit homework 2, submit homework 3, submit homework 4, etc.) are displayed, and different homework submission options correspond to different homework. Taking the example that the user needs to upload a homework image for submit homework 4, the user clicks on the submit homework 4 option 7042. In response to the user's operation, the flat panel can display Figure 9b the homework submission interface 705 as shown in (1).

[0417] Continuing to refer to (1) in 9b, in the homework submission interface 705, an image preview window 7041, a photo-taking option 7051, and a submission option 7052 are displayed. Among them, the preview image captured in real time by the desk lamp camera is displayed in the image preview window 7041. When the user places the homework or book, etc. within the acquisition area of the desk lamp camera, the user can click on the photo-taking option 7051 to trigger the desk lamp camera to take a homework image. In response to the user's operation, the educational APP generates a photo-taking request and sends it to the desk lamp side to call the desk lamp camera to take a homework image. Among them, the desk lamp camera takes the homework image at the high resolution of the image carried in the photo-taking request and returns the captured homework image to the educational APP on the flat panel for display through the educational APP, which can be referred to Figure 9b the interface 706 as shown in (2).

[0418] Continuing to refer to (2) in 9b, in the interface 706, an image preview window 7041, a photo-taking option 7051, a submission option 7052, and the homework image 7061 taken by the desk lamp camera are displayed. Among them, a close option 7062 is also displayed on the homework image 7061. If the user is not satisfied with the homework image 7061, the user can click on the close option 7062, and the homework image 7061 will no longer be displayed on the interface. At this time, the user can click on the photo-taking option 7051 to trigger the desk lamp camera to take a new homework image. If the user is satisfied with the homework image 7061, the user can click on the submission option 7052. In response to the user's operation, the flat panel can display Figure 9cThe to-be-confirmed interface 701 is shown. Among them, in the to-be-confirmed interface 701, the homework image 7061 to be submitted and the confirmation submission window 7071 are displayed. If the user clicks the confirmation option 7072 in the confirmation submission window 7071, the education APP uploads the homework image 7061 to the database in response to the user operation. If the user clicks the cancellation option 7071 in the confirmation submission window 7071, the education APP can display the interface as shown in Figure 9b in (1) of the figure to wait for the user to click the photographing option 7051 to trigger the desk lamp camera to re-take the homework image.

[0419] It should be noted that, continuing to refer to (2) in 9b, if the user is satisfied with the homework image 7061, the user can click the submission option 7052. In response to the user operation, the education APP can also stop displaying the interface as shown in Figure 9c and directly upload the homework image 7061 to the database. This embodiment does not make any limitations in this regard.

[0420] Scenario Three

[0421] Referring to Figure 1a the schematic diagram of the application scenario shown, this scenario takes the point reading function (or finger reading function) as an example to elaborate on the technical solution. When a student needs the education APP to read the content in the book aloud, the desk lamp camera can be used to collect the book image in real time. Then, the education APP can load the corresponding book content according to the book image and determine the book content to be read aloud based on the student's finger position or page-turning operation for reading aloud.

[0422] As Figure 10 shown is the interaction schematic diagram of each module. Referring to Figure 10 , the present application embodiment provides a method flow for the tablet and the desk lamp to work together, which specifically includes:

[0423] S901, in response to the user's operation of clicking the point reading function, the tablet and the desk lamp execute the processes of the device discovery stage, the virtual camera enabling stage, and the virtual camera preview access stage, and the tablet displays the preview interface.

[0424] Among them, for the processes of the device discovery stage, the virtual camera enabling stage, and the virtual camera preview access stage, reference can be made to the foregoing, and details will not be elaborated here.

[0425] It should be noted that in the virtual camera preview access stage, the hardware virtualization API in the tablet sends a virtual camera access instruction to the camera service, and the instruction will carry camera configuration parameters. Among them, the camera configuration parameters can include, but are not limited to, the image resolution and the image acquisition frame rate. The desk lamp camera is set according to the received configuration parameters and collects preview image data according to the corresponding image resolution and image acquisition frame rate.

[0426] In this scenario, in order to reduce the bandwidth occupied by the preview image stream, during the virtual camera preview access phase, the desk lamp camera can capture preview images at a low resolution. For example, the first configuration parameter is carried in the virtual camera access instruction sent by the hardware virtualization API in the tablet to the camera service. The first configuration parameter includes the first image resolution (such as 480P). Furthermore, the desk lamp camera is set according to the received configuration parameter, and captures preview image data according to the first image resolution and the corresponding image capture frame rate.

[0427] S902. The educational APP in the tablet recognizes the preview image to determine the book name.

[0428] Exemplarily, the educational APP can perform book information recognition on each received frame of the preview image, or can periodically perform book information recognition on the latest received preview image. This embodiment does not make a limitation in this regard.

[0429] Exemplarily, an image recognition algorithm can be integrated in the educational APP to implement the image recognition operation, or the educational APP can also call an image recognition service to perform the image recognition operation. This embodiment does not make a limitation in this regard.

[0430] Regarding the image recognition algorithm, reference can be made to the existing technology, and this embodiment will not elaborate here.

[0431] S903. The educational APP in the tablet retrieves the database according to the book name and loads the book content corresponding to the book name.

[0432] If the educational APP in the tablet retrieves the database according to the book name and determines books of different versions, it can display the corresponding book list for the user to select. Furthermore, in response to the user's selection operation of a certain version of the book, the educational APP loads the content corresponding to the version of the book.

[0433] S904. In response to the user's page turning or finger point reading operation, the educational APP in the tablet recognizes the preview image to determine the paragraph to be read aloud and reads the corresponding paragraph.

[0434] After the book content is loaded, the educational APP can perform recognition of the indicating tool of the finger or point reading pen instruction on each received frame of the preview image, or can periodically perform recognition of the indicating tool of the finger or point reading pen instruction on the latest received preview image. This embodiment does not make a limitation in this regard. In response to the user's point reading operation, the educational APP can recognize the book page number and the user's click position information, such as coordinate information, etc., from the preview image. Furthermore, according to the book page number and the user's click position information, the paragraph to be read aloud can be determined from the loaded book content and the corresponding paragraph is read aloud.

[0435] After the book content is loaded, the education APP can also perform page-turning recognition based on the preview image stream. In response to the user's page-turning operation, the education APP can determine the paragraph to be read aloud in the loaded book content according to the recognized book page number and read the corresponding paragraph.

[0436] For the parts not explained in detail in this process, reference can be made to the previous text and will not be elaborated here.

[0437] Figure 1a 、 Figures 11a - 11b An exemplary application scenario is shown. As Figure 11a shown in (1) below, the tablet displays the education APP interface 701, and various function options of the education application are displayed on the interface 701, including but not limited to the word lookup function, the point reading function, the homework function, the photo-taking function, etc. The user clicks on the point reading function option 7013, and in response to the user's operation, the tablet executes the processes of the device discovery stage, the virtual camera service enabling stage, and the virtual camera preview access stage.

[0438] Among them, in the device discovery stage, if the number of table lamps that can be linked with the tablet filtered by the device management service of the tablet is one, the tablet automatically executes the processes of the virtual camera service enabling stage and the virtual camera preview access stage, and displays an interface such as Figure 11a shown in (2) below. In the device discovery stage, if the number of table lamps that can be linked with the tablet filtered by the device management service of the tablet is multiple, the tablet displays a table lamp selection interface. Exemplarily, a list of table lamps to be linked is displayed on the table lamp selection interface, and the user can perform a selection operation. In response to the user's selection operation, the education application determines one table lamp to be linked and continues to execute the processes of the virtual camera service enabling stage and the virtual camera preview access stage to display an interface such as Figure 11a shown in (2) below.

[0439] As Figure 11a shown in (2) below, an image preview window 7081 is displayed in the interface 708. Among them, the preview image captured in real time by the table lamp camera is displayed in the image preview window 7081. The education APP recognizes the preview image to determine the book name. After the education APP recognizes the book name, it retrieves the database according to the book name. If the corresponding book is retrieved, the operation of loading the book content can be performed, which can be referred to Figure 11b shown in (1) below.

[0440] Continue to refer to Figure 11bAs shown in (1), an image preview window 7081, a recognized book name 7091, and a book content loading progress indicator 7092 are displayed in the interface 709. After the book content is loaded, the education APP can identify the user's page-turning operation or point-reading operation based on the preview image. Taking the user's page-turning operation as an example, referring to the interface 710 shown in (2) below, the user's page-turning action can be displayed in the image preview window 7081. In response to the user's operation, the education APP identifies the book page number based on the preview image. Furthermore, the education APP can determine the paragraph to be read aloud in the loaded book content according to the identified book page number and read the corresponding paragraph. Figure 11b As shown in (2) below, in the interface 710, the user's page-turning action can be displayed in the image preview window 7081. In response to the user's operation, the education APP identifies the book page number based on the preview image. Furthermore, the education APP can determine the paragraph to be read aloud in the loaded book content according to the identified book page number and read the corresponding paragraph.

[0441] In the collaborative working method provided by the embodiments of the present application, a professional online education experience can be combined using a household tablet and a table lamp with a camera. The camera of the table lamp device is used in cooperation with the education APP on the tablet to complete scenarios that require taking pictures, such as students' finger word lookup, homework submission, and book finger reading.

[0442] Based on the above-mentioned hardware structure and software structure, for the electronic device 100 (taking a tablet as an example) and the IOT device 200 (taking a table lamp as an example), when they are used in cooperation in the learning scenarios mentioned above, the data transmitted from the table lamp to the tablet is different according to different services. For example, in the point-reading scenario mentioned above, the table lamp transmits the preview stream data captured by the camera to the tablet, that is, a continuously transmitted video data stream; in the word lookup scenario and homework submission scenario mentioned above, the table lamp transmits the image data captured by the camera to the tablet.

[0443] Specifically in practical applications, when the table lamp transmits preview stream data and image data to the tablet, it can be based on the same data packet principle. Taking a set size as a unit, the currently generated preview stream data is encapsulated into a preview stream data packet, and a part of the entire image data is encapsulated into an image data small packet, and then transmitted to the tablet side based on the Real-time Transport Protocol (RTP). For this way of preview stream data, since there is no process of packetizing, the obtained preview stream data packet can be directly processed and displayed, that is, there is no distinction between the first packet, the last packet, and the middle packet. For image data, the finally displayed photo must be determined according to the positional relationship between the image data small packets, and the packetizing operation is triggered according to the last packet. However, in the current image data transmission method, it is necessary to parse the data body of the image data small packet to determine whether the image data small packet is the first packet.

[0444] In addition, even if the data body of the image data small packet is parsed, it cannot be determined whether the image data small packet is the last packet, that is, the packetizing cannot be triggered quickly and accurately.

[0445] In view of this, the present application provides an image data transmission method, aiming to quickly and accurately determine whether the received small packet of image data is the first packet, the middle packet, or the last packet without parsing the data body of the received small packet of image data. In this way, when it is recognized that the received small packet of image data is the last packet, the data packet assembling operation can be performed.

[0446] See Figure 12 , which exemplarily shows the specific process of image data transmission between the tablet and the table lamp:

[0447] S1001, the tablet generates a photographing request (including the parameter information that the photographed photo needs to meet) in response to the user's operation behavior.

[0448] It can be understood that, through the above description, in the word query scenario and the homework submission scenario, the transmission of image data is involved. Therefore, the tablet's response to the user's operation behavior specifically means that the user triggers the operation of the word query function or the homework submission function.

[0449] Exemplarily, in some implementation manners, the operation behavior made by the user, for example, is to click the control corresponding to the point reading function / scenario displayed on the current interface of the tablet, or click the control corresponding to the homework submission function / scenario.

[0450] Exemplarily, in some other implementation manners, the operation behavior made by the user, for example, is to control the tablet to make a response through a voice command, and then select the word query function or the homework submission function.

[0451] It should be understood that the above description is only an example listed for better understanding the technical solution of this embodiment and does not serve as the only limitation to this embodiment.

[0452] In addition, through the above description, it can be known that in the word query scenario and the homework submission scenario, the resolution requirement for image data is higher than the resolution requirement for preview stream data in the point reading scenario. Therefore, when generating a photographing request in response to the user's operation behavior, at least the parameter information that the photographed photo needs to meet, such as the resolution, needs to be carried.

[0453] S1002, the tablet sends the photographing request to the table lamp.

[0454] After generating the above-mentioned required photographing request, the tablet can send the photographing request to the table lamp through the control channel negotiated with the table lamp, for example.

[0455] S1003, the table lamp detects whether the camera is abnormal.

[0456] In practical applications, the camera of the table lamp may be unable to take pictures due to certain factors. Therefore, in order to minimize unnecessary processing, such as parsing the photo-taking request and judging parameter information, it is possible to first detect whether the camera of the table lamp is abnormal in the current scenario, that is, whether it can be used normally.

[0457] Correspondingly, if the camera is abnormal, the table lamp can generate an abnormal feedback packet. For the convenience of distinction, the abnormal feedback packet fed back when the camera is abnormal is hereinafter referred to as abnormal feedback packet 1, and the abnormal feedback packet fed back when the parameter is abnormal is referred to as abnormal feedback packet 2, and step S1004 is executed.

[0458] On the contrary, that is, when the camera is normal and available, the parameter information carried in the photo-taking request is extracted, that is, step S1006 is executed.

[0459] S1004, the table lamp sends abnormal feedback packet 1 to the tablet.

[0460] Exemplarily, in some implementation manners, abnormal feedback packet 1 can be transmitted to the tablet through an image data channel negotiated with the tablet.

[0461] S1005, make a response according to the abnormal reason in abnormal feedback packet 1.

[0462] Exemplarily, in some implementation manners, in order to facilitate the tablet to know the abnormal reason in abnormal feedback packet 1, an error code indicating the abnormality can be carried in the data header of abnormal feedback packet 1. In this way, the tablet does not need to parse the data body and can directly know the current abnormal reason according to the data header, and then make a corresponding response according to the abnormal reason. For example, the tablet can make a prompt on the user interface to inform the user that the camera of the table lamp is abnormal, so that the user can check the camera of the table lamp according to the prompt.

[0463] It should be understood that the above description is only an example listed for better understanding the technical solution of this embodiment and does not serve as the sole limitation of this embodiment.

[0464] S1006, the table lamp extracts the parameter information in the photo-taking request.

[0465] Specifically, when the camera of the table lamp is normal and available, the table lamp can extract the parameter information from the photo-taking request sent by the tablet, such as the resolution information of the photo to be taken.

[0466] S1007, the table lamp detects whether the parameter information is abnormal.

[0467] Specifically, in this embodiment, the detection of the parameter information by the table lamp is, for example, to judge whether the camera supports the resolution information.

[0468] Accordingly, if not supported, it is determined that the parameter is abnormal and step S1008 needs to be executed; if supported, it is determined that the parameter information is normal and step S1010 can be executed.

[0469] S1008, the table lamp sends an exception feedback packet 2 to the tablet.

[0470] Similarly, the exception feedback packet 2 can be transmitted to the tablet through the image data channel negotiated with the tablet.

[0471] In addition, it should be noted that in actual applications, the table lamp side can also generate other exception feedback packets according to other exception reasons, such as network transmission exceptions, etc., and feedback them to the tablet side. The two specific exception scenarios given above are only examples and do not specifically limit this embodiment.

[0472] S1009, make a response according to the exception reason in the exception feedback packet 2.

[0473] Exemplarily, the error code carried in the exception feedback packet 2 can also be stored in the data header of the exception feedback packet 2. In this way, the tablet does not need to parse the data body and can directly obtain the reason for this exception according to the data header, and then make a corresponding response according to the exception reason. For example, the tablet can regenerate a photo-taking request according to the resolution supported by the table lamp and send the regenerated photo-taking request to the table lamp.

[0474] It should be understood that the above description is only an example listed for better understanding the technical solution of this embodiment and does not serve as the only limitation of this embodiment.

[0475] S1010, the table lamp adjusts the shooting parameters of the camera according to the parameter information and uses the adjusted camera to take a picture to obtain an image data packet.

[0476] It should be noted that the image data packet mentioned in this embodiment specifically includes the image data (data body) of an entire photo that needs to be displayed on the tablet side, and the data header allocated for this image data.

[0477] For a better understanding of the generation process of the above-mentioned image data packet, the following is combined with Figure 13 for description.

[0478] See Figure 13 , taking the size of the image data of an entire photo that needs to be displayed on the tablet side obtained by using the adjusted camera as 10,000 bytes as an example. After obtaining this image data, based on the generation standard of the image data packet in this embodiment, this image data needs to be used as the data body, and a data header with a size of 128 bytes is added before this data body, so as to obtain the image data packet corresponding to the entire photo.

[0479] It is understandable that bytes start from 0. Therefore, the overall size of the image data packet is 128 bytes of data header + 10,000 bytes of data body, that is, 10,128 bytes. Among them, bytes 0 to 127 correspond to the content of the data header, and bytes 128 to 10,127 correspond to 10,000 bytes of image data.

[0480] Specifically in this embodiment, the data header in the image data packet includes two parts. One part is used to record the identifier indicating whether the image data is normal, which is called the identifier field for whether the data is normal in this embodiment. The other part is reserved for future business expansion and is called the expansion field in this embodiment. To better understand the data header of this structure, the following will be combined with Figure 14 for illustration.

[0481] See Figure 14 , among the 128 bytes of the data header, bytes 0 to 3 are used to record the identifier indicating whether the image data is normal, that is, the identifier field for whether the data is normal, and bytes 4 to 127 are reserved for future business expansion and are the expansion field.

[0482] Exemplarily, in some implementation manners, it can be agreed that the user "0" indicates that the data is normal, and "1" indicates that the data is abnormal. That is, if the content in byte 0 is "0", it means that the image data in the data body is normal data (substantially the binary data of the photo that needs to be returned to the tablet), that is, the corresponding photo can be restored through tablet processing. If the content in byte 0 is "1", it means that the image data in the data body is abnormal data, such as an abnormal string.

[0483] Exemplarily, in some implementation manners, it can be agreed that different abnormal strings correspond to different abnormal reasons, which is convenient for later positioning and resolving the abnormality.

[0484] In addition, it should be noted that in some implementation manners, the expansion field can also be further divided according to business needs. For example, for one-to-many, that is, in the scenario where a table lamp needs to transmit image data to multiple tablet and mobile phone devices, it can be agreed that the unique identifier of the photo to be taken is carried in the photographing request, and when the table lamp generates the above image data packet, it can add the identifier indicating the uniqueness of the photo in bytes 4 to 32 of the expansion field.

[0485] It should be understood that the above description is only an example listed for better understanding the technical solution of this embodiment and does not serve as the sole limitation of this embodiment.

[0486] S1011. The table lamp divides the data in the image data packet into sub - packets according to the set size, obtaining N image data sub - packets (each image data sub - packet includes a data header; an identification field recording whether the image data sub - packet is the first packet, or an intermediate packet, or the last packet; and a data body storing the image data).

[0487] Understandably, the size of the image data sub - packet can be dynamically set according to the bandwidth resources of the transmission channel, the current network quality, etc., or can be set to a fixed value. The specific implementation method is not limited in this embodiment.

[0488] For the convenience of description, in this embodiment, taking the set size of 1394 bytes and the image data packet of 10128 bytes as mentioned above as an example, combined with Figure 15 the process of dividing the image data packet into multiple image data sub - packets will be described.

[0489] Refer to Figure 15 , taking the subsequent transmission of the divided image data sub - packets still using the RTP protocol as an example, the data structure of the divided image data sub - packets in this embodiment includes three parts: an RTP data header, an identification field, and a data body.

[0490] Continue to refer to Figure 15 , for any one image data sub - packet, bytes 0 - 11 are used as the data header, byte 12 is used as the identification field, and bytes 13 - 1393 are used as the data body. That is, only the bytes 13 - 1393 of each image data sub - packet are used to store the data in the image data packet. Therefore, the data of the image data packet stored in each image data sub - packet is only 1381 bytes.

[0491] In addition, it should also be noted that in the image data sub - packet as the first packet, the data body also includes a data header with a size of 128 bytes in the image data packet. Therefore, in the image data sub - packet 1 as shown in Figure 15 , the actual image data stored is 1253 bytes of content. The subsequent image data sub - packets do not need to include the data header with a size of 128 bytes in the image data packet, so it is 1381 bytes. Based on this, for an image data packet with a size of 10128 bytes, it can be split into 8 image data sub - packets with the above - mentioned structure. The size of the image data sub - packet 8 as the last packet is less than 1394 bytes, specifically 12 - byte RTP data header + 1 - byte identification field + 589 - byte image data, that is, the size of the image data sub - packet 8 is 601 bytes.

[0492] It should be understood that the above description is only an example listed for better understanding the technical solution of this embodiment and does not serve as the only limitation of this embodiment.

[0493] See Figure 16 , which exemplarily shows the specific format of the RTP data header of each image data packet. As Figure 16 shown, the current version number (V) is specifically stored in the 0th byte, the data type (T) of the image data is specifically stored in the 1st byte, the sequence number (SEQNUM) is specifically stored in the 2nd and 3rd bytes, the timestamp (RTPTIME) is specifically stored in the 4th to 7th bytes, and the synchronization source identifier (SSRC) is specifically stored in the 8th to 11th bytes.

[0494] It should be noted that the above-mentioned data type can specifically be of the Portable Network Graphics (PGN) type, the Joint Photographic Experts Group (JPG) type, etc., and will not be listed one by one here. This embodiment does not limit this.

[0495] In addition, regarding the above-mentioned sequence number, in actual applications, each time the table lamp sends an image data packet, the sequence number will be incremented by 1.

[0496] In addition, regarding the above-mentioned timestamp, it is specifically used to reflect the acquisition time of the image data in the first byte of the data body in this image data packet. In this way, subsequent image data packets can be combined in sequence according to the timestamp, and then the image data packets of the entire photo can be restored, enabling the tablet to process the image data of the photo and then display it on the user interface.

[0497] In addition, regarding the above-mentioned synchronization source identifier, it is specifically used to identify the uniqueness of each image data packet, thereby avoiding repeated sending of the same image data packet and causing redundancy in the cache queue on the tablet side.

[0498] See Figure 17 , which exemplarily shows the specific format of the identification field of each image data packet. As Figure 17 shown, the packet identification information indicating whether the current image data packet is the first packet, the middle packet, or the last packet is specifically stored in the 0th and 1st frames of 12 bytes, and the remaining 2nd to 7th frames are reserved frames for subsequent expansion and use according to business needs.

[0499] That is to say, in the image data packet including the RTP data header, the identification field, and the data body structure given in this embodiment, the identification field is specifically used by the tablet for packet assembly processing.

[0500] Regarding the above-mentioned packet identification information, in some implementation manners, it can be agreed that "10" (the position of the 0th frame is "1", and the position of the 1st frame is "0") represents that the current small packet of image data is the first packet, that is, it includes the first frame of the image data, and when assembling packets, it is necessary to start from the first-frame image data in this small packet of image data.

[0501] Exemplarily, it can also be agreed that "01" (the position of the 0th frame is "0", and the position of the 1st frame is "1") represents that the current small packet of image data is the last packet, that is, it includes the last frame of the image data. When assembling packets, this small packet of image data is the last packet of the photo to be obtained this time, and after receiving this packet, the packet assembly operation is triggered.

[0502] Exemplarily, it can also be agreed that "00" (the position of the 0th frame is "0", and the position of the 1st frame is "0") represents that the current small packet of image data is an intermediate packet.

[0503] In addition, it should also be noted that considering that in actual applications, when transmitting image data of multiple photos, it is possible that the data of the front and back photos are in the same small packet of image data. For example, the last frame of the previous photo and the first frame of the next photo are in the same small packet of image data. It can be agreed that the user "11" ((the position of the 0th frame is "1", and the position of the 1st frame is "1") represents that the current small packet of image data includes the last frame of the previous photo and the first frame of the next photo.

[0504] It should be understood that the above description is only an example listed for better understanding the technical solution of this embodiment, and does not serve as the only limitation to this embodiment.

[0505] In addition, it should also be noted that based on the data structure of the small packet of image data given in this embodiment, the abnormal feedback packet 1 sent to the tablet when the above camera is abnormal, and the abnormal feedback packet 2 sent to the tablet when the camera does not support the resolution carried in the photographing request can also be sub-packaged based on the above sub-packaging principle.

[0506] It can be understood that since the abnormal feedback packets (abnormal feedback packet 1, abnormal feedback packet 2) are usually small, after being sub-packaged by the above sub-packaging principle, all the data can be stored in one small packet. For this situation, the above-mentioned packet identification information is "11".

[0507] S1012, the table lamp sends N small packets of image data.

[0508] Specifically, after sub-packaging the data in the image data packet according to the above data structure to obtain N small packets of image data to be sent, the table lamp can send these N small packets of image data through the image data channel negotiated with the tablet.

[0509] It is understandable that if it is determined during the negotiation phase that the data to be transmitted needs to be encrypted, each small packet of image data needs to be encrypted according to the negotiated encryption method, and then the encrypted small packet of image data is sent to the tablet.

[0510] S1013, the tablet performs subsequent processing according to the content of the identification field in the currently received small packet of image data.

[0511] It is understandable that if the received small packet of image data is in encrypted form, the tablet needs to decrypt it first according to the decryption method negotiated with the table lamp to obtain the small packet of image data in plaintext form. Then, by judging whether the content in frame 0 of the 12-byte identification field is "1" and the content of frame 1 is "0", that is, determining whether the packet identification information is the agreed "10", and further determining whether the currently received small packet of image data is a small packet of image data containing the first frame, that is, the first packet.

[0512] Accordingly, if so, the small packet of image data is cached in the small packet of image data cache queue, and then the above processing is performed on the received small packet of image data. If the packet identification is the agreed "00", it is determined as an intermediate packet, which is cached in the small packet of image data cache queue, and then the above processing is performed on the received small packet of image data until the packet identification information of the received small packet of image data is the agreed "01" or "11", which is cached in the small packet of image data cache queue, and the reception of the small packet of image data is stopped, and the packet assembly operation is started.

[0513] For the sake of easy understanding, the following Figures 18 - 21 describes the operations involved in packet assembly.

[0514] Still taking the image data of 10,000 bytes in the above example as an example, exemplarily, when performing the packet assembly operation, the tablet will take out the small packet of image data 1 from the small packet of image data cache queue. As Figure 18 shown, in the 12-byte identification field, the recorded packet identification information is "10", indicating that the small packet of image data 1 is the first packet. The data body of this packet includes the Figure 13 entire data header of the image data packet shown, and the first byte of this data header records the identifier indicating whether the image data is normal. Therefore, the data body of the small packet of image data 1 can be parsed to extract the content recorded in byte 13, and then it can be determined that the image data is normal.

[0515] Exemplarily, when the identifier recorded in byte 13 is "1" (as Figure 18As shown, according to the above description, "1" indicates that the image data is abnormal. In this case, there is no need to parse other small packets of image data, such as the small packets of image data 2 to 8 mentioned above. The cache queue of the small packets of image data can be directly cleared.

[0516] Exemplarily, in some implementation manners, after clearing the cache queue of the small packets of image data, the tablet computer can give a prompt on the user interface to prompt the user that the image data captured this time is abnormal and cannot be displayed, so that the user can trigger a photographing request again to obtain image data.

[0517] Exemplarily, in some other implementation manners, after clearing the cache queue of the small packets of image data, the tablet computer can also simulate user operations and automatically generate a photographing request so that the table lamp can re-execute the above process according to the new photographing request.

[0518] It should be understood that the above description is only an example listed for better understanding the technical solution of this embodiment and does not serve as the sole limitation of this embodiment.

[0519] Exemplarily, when the flag recorded in byte 13 is "0" (as Figure 19 shown), according to the above description, "0" indicates that the image data is normal. In this case, other small packets of image data cached in the cache queue of the small packets of image data can be parsed.

[0520] Exemplarily, if the content of frame 0 and the content of frame 1 in the 12 bytes of the currently parsed small packet of image data are both "0", that is, the packet identification information is "00" (as Figure 20 shown), this indicates that the small packet of image data is an intermediate packet. By parsing this intermediate packet, the image data of the intermediate packet can be obtained.

[0521] Exemplarily, if the content of frame 0 in the 12 bytes of the currently parsed image data packet is "0" and the content of frame 1 is "1", that is, the packet identification information is "01" (as Figure 21 shown), this indicates that the small packet of image data is an end packet. After parsing this small packet of image data, the image data of the end packet can be obtained.

[0522] After completing the parsing of the first packet, each intermediate packet, and the end packet, the positional relationship between the image data can be determined according to the sequence number recorded in bytes 2 to 3 and the timestamp recorded in bytes 4 to 7 in the RTP data header of each image data packet, and then the image data of the photo captured by the camera can be restored. In this way, the tablet computer can process the image data of the photo according to business needs and then display the processed photo on the user interface.

[0523] It should be understood that the above description is only an example listed for better understanding the technical solution of this embodiment, and does not serve as the sole limitation of this embodiment.

[0524] Thus, in the image data transmission method provided in this embodiment, before transmission, the image data packet is sub-packaged according to the above data structure, so that the tablet receiving the small image data packet can quickly and accurately determine whether the current small image data packet is the first packet, the middle packet, or the last packet according to the packet identification information in the identification field of each small image data packet.

[0525] That is to say, based on the image data transmission method provided in this embodiment, it is not necessary to parse the data body of the received small image data packet, and the first packet, the middle packet, and the last packet of each photo to be restored can be quickly and accurately determined, so as to quickly and accurately achieve packet assembly.

[0526] Furthermore, considering network fluctuations, the large size of image data, and factors such as the small image data packets obtained by the above sub-packaging method, it is very easy for the speed of the small image data packets put into the transmission queue on the desk lamp side to be greater than the speed of taking out the small image data packets from the transmission queue and sending them to the tablet during the image data transmission process. This will face the situation that the number of small image data packets cached in the transmission queue is greater than the maximum threshold supported by the transmission queue, such as 1000. In the face of this situation, the current processing standard followed by image data transmission is to directly reuse the processing standard of the preview stream, specifically to directly discard the subsequent small image data packets that need to be put into the transmission queue until the number of small image data packets in the transmission queue is less than the set threshold, and then put in the newly obtained small image data packets.

[0527] For the convenience of explanation, the following is an example:

[0528] Suppose that after the image data taken by the camera on the desk lamp side according to the resolution in the photo-taking request sent by the tablet side is sub-packaged, there are 1050 small image data packets, and the maximum threshold supported by the transmission queue is 1000. If the desk lamp side adds the 1040th small image data packet to the transmission queue, only 39 image data packets have been sent out from the transmission queue, and there are still 1000 that have not been taken out. At this time, the 1040th image data packet cannot be put into the transmission queue and needs to be discarded. Then, when preparing to put in the 1041st small image data packet, the above judgment still needs to be made. If the number of small image data packets in the transmission queue is still not less than 1000, the 1041st small image data packet also needs to be discarded.

[0529] Accordingly, if, following the above logic, at the 1050th small packet of image data, the number of small packets of image data in the transmission queue is less than 1000, the 1050th small packet of image data can be placed in the transmission queue at this time. Since all the small packets of image data for the currently captured image data have been placed in the transmission queue, there is only a process of retrieving small packets of image data from the transmission queue and sending them to the tablet subsequently. However, since the 1040th to 1049th small packets of image data are discarded, even if the tablet receives all the small packets of image data in the transmission queue and triggers the packet assembly operation subsequently, the image data cannot be restored.

[0530] For this processing standard, in the case of preview stream data, since the preview stream data is continuous, even if one or several frames are lost, it will not have a great impact on the final display effect. However, for image data, since the image data is large and there are many small packets of image data obtained by the above packet splitting method, the transmission time is long. If the above situation occurs, it will not only cause the tablet to be unable to restore the image data, but also waste device resources and network resources.

[0531] In view of this, the present application further provides an image data transmission method, aiming at when an abnormality occurs in the channel for transmitting small packets of picture data between an electronic device and an Internet of Things device, the Internet of Things device can directly empty the small packets of image data in the transmission queue, and the electronic device can directly reconstruct the transmission channel, ending the transmission of the small packets of image data this time in advance, reducing the occupation of device resources and network resources, and improving the transmission performance of the image data.

[0532] It should be noted that the image data transmission method involving image data retransmission and interruption given in this embodiment includes two parts, one part is the operation on the table lamp side, and the other part is the operation on the tablet side.

[0533] Among them, the operations on the table lamp side include operations of capturing image data, splitting the image data into packets, putting (adding) small packets of image data into the transmission queue, and retrieving small packets of image data from the transmission queue. To ensure that the operations of putting small packets of image data into the transmission queue and retrieving small packets of image data from the transmission queue do not interfere with each other, usually they are performed asynchronously, that is, completed by two independent threads respectively. For the convenience of description, in this embodiment, the thread for putting small packets of image data into the transmission queue is called thread 1, and the thread for retrieving small packets of image data from the transmission queue is called thread 2.

[0534] Furthermore, thread 1 can be used as the main thread, that is, in addition to performing the operation of putting small packets of image data into the transmission queue, it also needs to perform the operations of capturing image data and splitting the image data into packets.

[0535] On the tablet side, the operations to be performed are to determine the specific attributes of the data packet received from the table lamp side, which may be a small image data packet or an image data interruption feedback packet, and then perform corresponding processing when receiving the data packet.

[0536] For better understanding, the following combines Figure 22 to specifically describe the image data transmission method involving image data retransmission and interruption provided in this embodiment.

[0537] See Figure 22 , for the table lamp side, specifically including:

[0538] S2001. When thread 1 receives a photographing request, the camera takes a picture according to the resolution indicated in the photographing request to obtain image data.

[0539] It can be understood that thread 1 mentioned in this embodiment is, for example, the main thread started after the table lamp is started, which can be used to receive various requests sent by the tablet side, and perform photographing of image data or acquisition of preview streams according to the requests.

[0540] Regarding the resolution carried in the photographing request, such as 1080P mentioned above.

[0541] In addition, regarding the photographing request, for example, it is generated after the user triggers the control corresponding to the word query function, or the control corresponding to submitting homework, or the photographing control displayed in the user interface of the tablet side. For the generation details of the photographing request, reference can be made to the above, and details will not be elaborated here.

[0542] Correspondingly, the photographing request can be transmitted through the image data channel negotiated between the tablet and the table lamp.

[0543] S2002. Thread 1 divides the image data into N small image data packets according to the set size.

[0544] It can be understood that the size of the small image data packet can be 1394 bytes as mentioned above, or can be dynamically adjusted according to actual business requirements. This embodiment does not limit this.

[0545] Exemplarily, in some implementation manners, the image data obtained by photographing can be directly divided into packets, that is, each data body of the small image data packet only includes image data and does not include the above-mentioned data header (the first data header).

[0546] Exemplarily, in some other implementation manners, the packet division method given in the above embodiment can also be followed.

[0547] For the sake of convenience of description, in this embodiment, 128 bytes of data header is first added to the image data to form an image data packet, and then the image data packet is sub-packaged to obtain small image data packets with a data structure including an RTP data header, an identification field, and a data body. For specific implementation details, reference can be made to Figures 12 - 21 the corresponding content, which will not be elaborated here.

[0548] S2003, Thread 1 adds N small image data packets to the end of the queue to be transmitted.

[0549] It should be noted that in this embodiment, the queue to be transmitted is specifically used to cache the small image data packets obtained through the above sub-packaging process. Essentially, it is a data queue that follows the first-in, first-out characteristic of the queue. Therefore, the small image data packets need to be added to the end of the queue to be transmitted in order.

[0550] Correspondingly, when retrieving image data from the queue to be transmitted, it is retrieved from the head of the queue in order.

[0551] It can be understood that for the completion of the sub-packaging operation, when the small image data packets are first put into the queue to be transmitted, since there are no small image data packets in the queue to be transmitted at this time, there is no need to make a quantity judgment in the same way as the processing method followed by the existing preview stream, and the small image data packets can be directly added to the queue to be transmitted.

[0552] In addition, it can be understood that the process of adding small image data packets to the queue to be transmitted is to add one data packet at a time, rather than adding all the small image data packets to the end of the queue to be transmitted at once. Therefore, when the first small image data packet is put into the queue to be transmitted, Thread 2 will be started synchronously. That is, during the process of putting small image data packets into the queue to be transmitted, Thread 2 will perform the operation of retrieving small image data packets from the queue to be transmitted, that is, repeat step S2003. During the process of putting small image data packets into the queue to be transmitted, steps S3001 to S3005 will be synchronously executed.

[0553] Regarding the specific processing flow of steps S3001 to S3005, it is as follows:

[0554] S3001, Thread 2 checks whether the number of small image data packets in the queue to be transmitted is less than the set threshold.

[0555] Specifically in this embodiment, in order to avoid the problems of packet loss and frame loss in the corresponding processing method of the preview stream, each time Thread 2 retrieves small image data packets from the queue to be transmitted, it is necessary to first judge whether the number of small image data packets already cached in the queue to be transmitted is less than the set threshold.

[0556] Assume that the set threshold is 1000, that is, the transmission queue can support caching up to 1000 small packets of image data at most. If the number of small packets of image data currently cached in the transmission queue read this time is less than 1000, it indicates that new small packets of image data can still be added to the transmission queue. In this case, Thread 2 does not need to trigger an interruption, but directly takes out a small packet of image data from the head of the transmission queue and transmits the taken-out small packet of image data to the tablet, that is, step S3002 is executed.

[0557] It is understandable that after transmitting the small packet of image data taken out this time to the tablet through the image data channel negotiated with the tablet, when continuing to take out the next small packet of image data from the transmission queue, step S3001 also needs to be executed, that is, before each operation of taking out a small packet of image data from the head of the transmission queue, it is necessary to first determine whether the number of small packets of image data currently cached in the transmission queue is less than the set threshold.

[0558] Correspondingly, if it is less, step S3002 is executed; otherwise, step S3003 is executed.

[0559] S3002, Thread 2 takes out a small packet of image data from the head of the transmission queue and transmits the taken-out small packet of image data to the tablet.

[0560] S3003, Thread 2 notifies Thread 1 to stop adding small packets of image data to the transmission queue and waits for a preset duration.

[0561] S3004, when Thread 2 loops through steps S3001 to S3003 for a preset number of times, if the number of small packets of image data in the transmission queue is still less than the set threshold, the transmission queue is emptied and an image data interruption feedback packet is generated (the image data interruption feedback packet includes a data header and an identification field indicating that the packet is an image data interruption feedback packet).

[0562] Specifically in this embodiment, when it is determined that the number of small packets of image data cached in the transmission queue is not less than the set threshold, it indicates that the current speed of putting small packets of image data into the transmission queue is greater than the speed of taking out small packets of image data from the transmission queue. Therefore, in order to avoid packet loss and cause abnormalities in the finally synthesized image data, Thread 2 will notify Thread 1 to stop adding small packets of image data to the transmission queue within a preset duration, for example, pause putting small packets of image data into the transmission list within 50 ms, rather than directly discarding the small packets of image data. Since Thread 2 still executes the operation of taking out small packets of image data from the transmission queue and sending them to the tablet during the process of Thread 1 stopping putting small packets of image data into the transmission queue, that is, steps S3001 and S3002 are repeatedly executed.

[0563] Accordingly, if there is still a situation where the number of small packets of image data cached in the transmission queue is not less than the set threshold, step S3003 will be executed again.

[0564] Furthermore, to avoid network failures that may prevent small packets of image data from being sent out continuously, thereby causing steps S3001 to S3003 to be executed in a loop continuously, wasting the resources of the desk lamp and the tablet. In this embodiment, a loop count is set, for example, 10.

[0565] It should be understood that the 50 ms of each pause and the pause count of 10 given in this embodiment are only a specific implementation. In actual applications, they can be adjusted according to actual business requirements, and this embodiment does not limit this.

[0566] Accordingly, if the number of times of looping through steps S3001 to S3003 reaches the set number of times, such as 10 times, and the number of small packets of image data cached in the transmission queue is still not less than the set threshold, such as 1000, it means that within 500 ms (50 * 10), not a single image data packet has been sent out. In this case, it can be considered that the current network already fails to meet the transmission requirements. To promptly inform the tablet and then interrupt the current image data transmission, reducing the resource occupation on the desk lamp side and the tablet side, thread 2 will clear the transmission queue, that is, delete all the small packets of image data cached in the transmission queue, and generate an image data interruption feedback packet according to the set data structure.

[0567] Specifically in this embodiment, the data structure of the image data interruption feedback packet is as Figure 23 shown, specifically including an RTP data header and an identification field. That is, it does not include a data body carrying data, and the entire image data interruption feedback packet is only 13 bytes in size. In this way, even if the network fluctuates and causes small packets of image data to fail to be transmitted to the tablet, due to the small size of the image data interruption feedback packet, the probability of transmission to the tablet is relatively high.

[0568] Continue to refer to Figure 23 Regarding the structure of the RTP data header, it can be the same as the RTP data header of the small packet of image data shown in Figure 16 That is, fill the version number (V) information with 0 bytes, fill the data type (T) with 1 byte, fill the sequence number (SEQNUM) with 2 bytes and 3 bytes, fill the timestamp (RTPTIME) with bytes 3 to 7, and fill the synchronization source identifier (SSRC) with bytes 8 to 11; regarding the identification field located after the RTP data header, it can occupy only 1 byte, that is, at byte 12, or it can be the remaining bytes of the entire image data interruption feedback packet. Specifically in this embodiment, it is agreed that 0xE0 is used to indicate that the current data packet is an image data interruption feedback packet.

[0569] For the specific use of the content filled in the byte in the RTP data header, reference can be made to the above text, and details will not be elaborated here.

[0570] It should be understood that the above description is only an example listed for better understanding the technical solution of this embodiment, and does not serve as the sole limitation of this embodiment.

[0571] S3005, Thread 2 notifies Thread 1 to delete all small image data packets that have not been written into the transmission queue.

[0572] It can be understood that when Thread 2 clears the transmission queue, it simultaneously notifies Thread 1 to delete all small image data packets that have not been written into the transmission queue. In this way, Thread 1 will not continue to write small image data packets into the transmission queue when Thread 2 clears the transmission queue (the data of the cached small image data packets is less than the set threshold), and Thread 2 will not perform the operation of taking out small image data packets from the transmission queue, that is, transmitting to the tablet, thereby reducing the transmission pressure of the image data channel and enabling the image data interruption feedback packet generated by Thread 2 to be transmitted to the tablet side as much as possible.

[0573] S3006, Thread 2 sends an image data interruption feedback packet to the table lamp.

[0574] Continue to refer to Figure 22 , for the tablet side, specifically including:

[0575] S4001, The tablet determines that the received data packet is a small image data packet and adds the image data packet to the cache queue.

[0576] S4002, When the packet assembly condition is met, the tablet assembles N small image data packets in the cache queue to restore the image data.

[0577] S4003, The tablet displays the image data.

[0578] S4004, The tablet determines that the received data packet is an image data interruption feedback packet and triggers the reconstruction operation of the image data channel.

[0579] It can be understood that for the tablet, both the small image data packet and the image data interruption feedback packet transmitted from the table lamp side can be regarded as data packets. Therefore, in this embodiment, the tablet can know the data structures of these two types of data packets according to the standards for generating small image data packets and the standards for image data interruption feedback packets on the table lamp side, and then extract the content filled in the identification field from the byte where the identification field is located.

[0580] As can be seen from the above description, for the small packets of image data, the content filled in the 0th and 1st frames of the identification field is information indicating whether the small packet of image data is the first packet, the middle packet, or the last packet, while the 2nd to 7th frames are reserved frames, usually filled with "0"; for the image data interruption feedback packet, "0xE0" is directly filled in the bytes after the RTP data header. Therefore, based on the extracted content, it can be determined whether the currently received data packet is a small packet of image data or an image data interruption feedback packet, and then corresponding operations can be performed.

[0581] Exemplarily, when it is determined that the received data packet is a small packet of image data, step S4001 can be executed, that is, adding the image data packet to the cache queue on the tablet side.

[0582] Correspondingly, when it is determined that the packet assembly condition is met, step S4002 is executed to assemble N small packets of image data in the cache queue to restore the image data.

[0583] Correspondingly, after obtaining the restored image data, step S4003 can be executed, that is, displaying the image data in the user interface of the tablet.

[0584] Regarding the packet assembly condition, for example, the packet identification information extracted from the identification field indicates that the current small packet of image data is the last packet. The specific identification method and packet assembly method can be referred to above and will not be elaborated here.

[0585] Exemplarily, when it is determined that the received data packet is an image data interruption feedback packet, step S4004 can be executed, such as automatically triggering a photo-taking request, so that the currently used image data channel between the tablet and the table lamp is recycled, and the two negotiate to create an image data channel again, that is, the current image data transmission operation can be ended in advance, and the transmission channel can be rebuilt, without waiting for the locally set timeout duration to end the current image data transmission operation and rebuild the transmission channel.

[0586] Thus, for the image data transmission method involving image data interruption and retransmission provided in this embodiment, when the channel for transmitting small packets of picture data between the electronic device and the Internet of Things device is abnormal, the Internet of Things device deletes all the small packets of image data that have not been transmitted to the tablet locally and sends an image data interruption feedback packet that can indicate that the current transmission is abnormal to the electronic device, so that when the electronic device determines that the received data packet is an image data interruption feedback packet, it can directly rebuild the transmission channel and end the transmission of the small packets of image data in advance, thereby effectively reducing the occupation of device resources and network resources and improving the transmission performance of image data.

[0587] In addition, it can be understood that the above description of the image data transmission method provided by the present application is from the perspective of the overall two devices, namely the tablet and the desk lamp. In specific implementation, the implementation of the above process needs to involve educational applications, device management services, camera services, transmission management services, hardware virtualization services, virtual camera HAL on the tablet side, and device interconnection services, hardware abstraction services, and cameras on the desk lamp side. For the specific interactions between these modules, reference can be made to the descriptions of the word search scenario and the assignment submission scenario in the above text, which will not be elaborated here.

[0588] In addition, it can also be understood that in order for an electronic device to implement the above functions, it includes corresponding hardware and / or software modules for executing each function. Combining the algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of the present application.

[0589] In addition, it should be noted that in an actual application scenario, the methods provided by the above embodiments implemented by an electronic device can also be executed by a chip system included in the electronic device. Among them, the chip system can include a processor. The chip system can be coupled to a memory, so that when the chip system runs, it calls the computer program stored in the memory to implement the steps executed by the above electronic device. Among them, the processor in the chip system can be an application processor or a processor other than an application processor.

[0590] In addition, an embodiment of the present application also provides a computer-readable storage medium. Computer instructions are stored in the computer storage medium. When the computer instructions run on an electronic device, the electronic device is caused to execute the above-related method steps to implement the methods in the above embodiments.

[0591] In addition, an embodiment of the present application also provides a computer program product. When the computer program product runs on an electronic device, the electronic device is caused to execute the above-related steps to implement the methods in the above embodiments.

[0592] In addition, an embodiment of the present application also provides a chip (which can also be a component or a module). The chip can include one or more processing circuits and one or more transceiver pins; among them, the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the above-related method steps to implement the method in the above embodiment, so as to control the receiving pin to receive a signal and control the sending pin to send a signal.

[0593] In addition, from the above description, it can be seen that the electronic device, computer-readable storage medium, computer program product or chip provided by the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0594] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image data transmission method, characterized in that, Applied to an Internet of Things device, the Internet of Things device is provided with a camera, the camera is used to collect image data downward, the Internet of Things device is bound to a target application in an electronic device, and the image data transmission method includes: The first thread packets the image data obtained by the camera according to the resolution carried in the photo request to obtain N image data packets, where N is an integer greater than 0; After obtaining N image data packets, starting a second thread, the second thread is used to read the image data packets from the head of the queue to be transmitted; The first thread sequentially adds the obtained N image data packets to the tail of the queue to be transmitted; The second thread performs the following operations during the process in which the first thread adds the image data packet to the queue to be transmitted: Each time the image data packet is read from the queue to be transmitted, obtaining the number of the image data packets in the queue to be transmitted; When the number of image data packets in the queue to be transmitted is less than a set threshold, taking an image data packet from the head of the queue to be transmitted and sending it to the electronic device; Otherwise, instruct the first thread to stop adding the image data packet to the queue to be transmitted and wait for a preset time; After waiting for a preset number of times and a preset time length, if the number of image data packets in the queue to be transmitted is still not less than the set threshold, the second thread clears the queue to be transmitted and generates an image data interruption feedback packet, wherein the image data interruption feedback packet carries identification information that identifies it as an image data interruption feedback packet, not an image data packet; The generated image data interruption feedback packet is sent to the electronic device.

2. The method according to claim 1, characterized in that When the second thread clears the queue to be transmitted, the method further includes: The second thread notifies the first thread to delete all image data packets that have not been written into the queue to be transmitted.

3. The method according to claim 1, characterized in that, The preset number of times is 10 times, and the preset duration of each waiting is 50ms.

4. The method according to claim 1, wherein The first thread packages the image data obtained by the camera according to the resolution carried in the photo request, including: The first thread processes the image data into an image data packet according to a first data packet standard; The first thread packets the image data packet according to the second data packet standard; wherein each image data packet carries packet identification information identifying the image data packet as a first packet, a middle packet, or a last packet.

5. The method according to claim 4, wherein The first data packet standard indicates that the image data packet includes a first data header and a first data body; The first data body is used to store the image data; The first data header includes an image data identification field, and the image data identification information recorded in the image data identification field is used to identify whether the image data stored in the first data body is normal.

6. The method according to claim 5, wherein The first data header also includes an extension field.

7. The method according to claim 6, wherein The size of the first data header is 128 bytes, the identification field is located at 0 to 3 bytes, and the extension field is located at 4 to 127 bytes.

8. The method according to claim 7, characterized in that, The second data packet standard indicates that the image data packet includes a second data header, a packet identification field, and a second data body; The second data header occupies 12 bytes, and the packet identification field occupies 1 byte; The content after combining the binary data corresponding to the 0th frame and the 1st frame in the packet identification field is the packet identification information; The second data body is used to store the image data.

9. The method according to claim 8, characterized in that, When the packet identification information indicates that the small packet of image data is the first packet, the second data body is also used to store the first data header.

10. The method according to any one of claims 1 to 9, characterized in that The second thread generates an image data interruption feedback packet, including: The second thread generates an image data interruption feedback packet including a third data header and an interruption identification field according to the third data packet standard; Among them, the third data header occupies 12 bytes, and the interruption identification information is filled in the interruption identification field.

11. An image data transmission method, characterized in that, Applied to an electronic device, a target application in the electronic device is bound to an Internet of Things device, a camera is set on the Internet of Things device, the camera is used to collect image data downward, and the image data transmission method includes: Register a virtual camera corresponding to the camera in the system, and send a photographing request to the Internet of Things device by calling the virtual camera, and the resolution of the image data captured by the camera is carried in the photographing request; Receive a data packet sent by the Internet of Things device, and determine the attribute of the data packet according to the identification information carried in the data packet; When the data packet is a small packet of image data, add the small packet of image data to the cache queue. The small packet of image data is obtained by the Internet of Things device sub-packaging the image data packet according to the second data packet standard. The image data packet is obtained by the Internet of Things device processing the image data according to the first data packet standard. The image data is obtained by the camera shooting according to the resolution. The image data packet can be divided into N small packets of image data, N is an integer greater than 0, and each small packet of image data carries packet identification information indicating that the small packet of image data is the first packet, or the middle packet, or the last packet; When the packet assembly condition is met, assemble the N small packets of image data in the cache queue, restore the image data captured by the camera according to the resolution, and display the image data in the target application; When the data packet is an image data interruption feedback packet, trigger a reconstruction operation of the image data channel. The image data interruption feedback packet is generated by the Internet of Things device according to the second data packet standard or the third data packet standard. The image data interruption feedback packet carries identification information indicating that it is an image data interruption feedback packet and not a small packet of image data. The image data channel is the channel for the Internet of Things device to transmit the small packet of image data or the image data interruption feedback packet to the electronic device.

12. The method according to claim 11, wherein The triggering of the reconstruction operation of the image data channel includes: Call the virtual camera to send a photographing request to the Internet of Things device, and the resolution of the image data captured by the camera is carried in the photographing request; Among them, each time a photographing request is generated, the electronic device and the Internet of Things device need to negotiate to determine an image data channel.

13. The method according to claim 11, wherein The method further includes: When receiving an image data packet with the carried identification information being the last packet, it is determined that the packet assembly condition is met.

14. The method according to claim 11, wherein The first data packet standard indicates that the image data packet includes a first data header and a first data body; The first data body is used to store the image data; The first data header includes an image data identification field, and the image data identification field is used to identify whether the image data stored in the first data body is normal.

15. The method according to claim 14, wherein The first data header further includes an extension field.

16. The method according to claim 15, characterized in that, The size of the first data header is 128 bytes, the identification field is located at bytes 0 to 3, and the extension field is located at bytes 4 to 127.

17. The method according to claim 16, wherein The second data packet standard indicates that the image data packet includes a second data header, a packet identification field, and a second data body; The second data header occupies 12 bytes, and the packet identification field occupies 1 byte; The content obtained by combining the binary data corresponding to the 0th frame and the 1st frame in the packet identification field is the packet identification information; The second data body is used to store the image data.

18. The method according to claim 17, wherein When the packet identification information indicates that the image data packet is the first packet, the second data body is further used to store the first data header.

19. The method according to any one of claims 11 to 18, characterized in that When the image data interruption feedback packet is generated according to the second data packet standard, the image data interruption feedback packet includes a third data header and an interruption identification field; Wherein, the third data header occupies 12 bytes, and the interruption identification field is filled with the agreed interruption identification information.

20. An Internet of Things device, characterized in that, The Internet of Things device includes: a memory and a processor, and the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the Internet of Things device executes the image data transmission method according to any one of claims 1 to 10.

21. An electronic device, characterized in that, The electronic device includes: a memory and a processor, and the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the image data transmission method according to any one of claims 11 to 19.

22. A collaborative working system, characterized in that, Including the Internet of Things device according to claim 20 and the electronic device according to claim 21.

Images