Data transmission method, electronic device and collaborative working system

Abstract

Embodiments of the present application provide a data transmission method, an electronic device and a cooperative work system. The method is applied to an electronic device, the electronic device comprising a first process and a second process, the first process being configured to run a target application, the second process being configured to transmit data to the first process, the second process comprising a first module and a second module, the first module being configured to receive the first process and the second module, the second module being configured to receive data, the first process and the second process communicating through AIDL, the first module and the second module communicating through JNI, the first module creating shared memory for transmitting data between the first process and the second process, the second module copying the received data into the shared memory, the target application obtaining the data in the shared memory and copying the data into the user space of the first process, thereby realizing data transmission across processes, reducing the number of data copies, improving the speed of data transmission across processes and reducing transmission time.

Description

Technical Field

[0001] This application relates to the field of smart terminal technology, and in particular to a data transmission method, electronic device, and collaborative work system. Background Technology

[0002] Because processes in the Android system cannot share memory, mechanisms are needed for data communication between different processes. In traditional communication mechanisms, data transfer across processes requires four copy operations. Taking process A transferring data to process B as an example, the data is first copied from process A's user space to its kernel space, then from its kernel space to memory, then from memory to process B's kernel space, and finally from its kernel space back to its user space.

[0003] Such data copying operations undoubtedly reduce the speed of data transfer between processes and prolong the transmission time. Therefore, how to improve the speed of data transfer across processes and reduce the transmission time is a problem that needs to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application provide a data transmission method, an electronic device, and a collaborative work system. In this data transmission method, the number of data copies during cross-process data transmission is reduced, thereby increasing the speed of cross-process data transmission and decreasing transmission time.

[0005] In a first aspect, embodiments of this application provide a data transmission method, characterized in that it is applied in an electronic device, the electronic device including a first process and a second process, the first process being used to run a target application, and the second process being used to transmit the target data to the first process; the second process includes a first module implemented in JAVA and a second module implemented in C++, the second module being used to receive the target data; the first process and the second process communicate via AIDL, and the first module and the second module communicate via JNI. The method includes:

[0006] The second module obtains the frame size of the target data and sends the frame size to the first module;

[0007] The first module creates shared memory based on the frame size and sends the shared memory address to the second module;

[0008] The second module fills the target data according to the shared memory address and sends the filling result to the first module;

[0009] When the filling result indicates successful filling, the first module sends the shared memory address to the target application;

[0010] The target application obtains the target data based on the shared memory address.

[0011] The target application can be any application.

[0012] In this way, the first module creates shared memory for data transfer between the first and second processes, the second module copies the received target data into the shared memory, and the target application obtains the target data in the shared memory and copies it into the user space of the first process. This achieves cross-process data transfer with fewer data copies, improving the speed of cross-process data transfer and reducing the time consumption of cross-process data transfer.

[0013] According to the first aspect, the electronic device establishes a communication connection with the IoT device, and the target application is bound to the IoT device; wherein, before the second module obtains the frame size of the target data, the method further includes: the target application sending a data request to the IoT device through a second process; and the second module receiving the target data fed back by the IoT device according to the data request.

[0014] According to the first aspect, or any implementation of the first aspect above, the data request includes a task identifier, and the target data carries the task identifier; wherein, after the first module creates shared memory according to the frame size, the method further includes: the first module stores the task identifier and the shared memory address in the shared memory pool accordingly;

[0015] The second module sends the filling result to the first module, which may include: the second module sending the filling result and the task identifier to the first module;

[0016] Before the first module sends the shared memory address to the target application, the method further includes: the first module querying the corresponding shared memory address in the shared memory pool based on the task identifier.

[0017] Since the process of requesting to take a photo and transmitting the captured image data spans multiple modules, and each module is an asynchronous process, using the task identifier as the unique identifier for data transmission can ensure the accurate execution of the asynchronous process. This also ensures that data is transmitted safely and efficiently across multiple modules and threads, preventing the target application from being unable to obtain accurate data.

[0018] According to the first aspect, or any implementation of the first aspect above, the first module stores the task identifier and the shared memory address in the shared memory pool, which may include:

[0019] When the number of shared memory addresses stored in the shared memory pool is less than the shared memory maintenance threshold, the first module stores the task identifier and the corresponding shared memory address into the shared memory pool; the shared memory maintenance threshold is used to indicate the total number of memory addresses that the shared memory pool can maintain;

[0020] When the number of shared memory addresses stored in the shared memory pool is greater than or equal to the shared memory maintenance threshold, the first module removes the target shared memory address from the shared memory pool and stores the task identifier and the corresponding shared memory address into the shared memory pool. Among the multiple shared memory addresses currently stored in the shared memory pool, the target shared memory address was stored into the shared memory pool earliest.

[0021] In this way, by setting a threshold for the number of shared memory pools to maintain, the unlimited expansion of shared memory can be prevented.

[0022] According to the first aspect, or any of the above implementations of the first aspect, the data request also includes data parameters, and the target data carries the data parameters;

[0023] The first module stores the task identifier and the shared memory address in the shared memory pool, including: the first module stores the task identifier and the shared memory address in the shared memory pool that matches the data parameter value.

[0024] For example, data parameters can be modes, formats, or other parameters used to distinguish the target data. Taking image data as an example, data parameters could be shooting modes, etc.

[0025] In this way, different shared memory pools can be used to manage the shared memory addresses corresponding to different data requests, depending on the different data parameter values ​​carried in the data request.

[0026] According to the first aspect, or any of the implementations of the first aspect above, the threshold for the number of shared memory pools that are matched with different data parameter values ​​is different.

[0027] In this way, by setting different shared memory maintenance thresholds for different shared memory pools, it can be applied to more application scenarios.

[0028] According to the first aspect, or any implementation of the first aspect above, the frame size of the target data is greater than a preset threshold.

[0029] In this way, for cross-process transmission of big data or large files, the improvement in cross-process transmission speed is more obvious, and the reduction in transmission time is also more obvious.

[0030] According to the first aspect, or any implementation of the first aspect above, the IoT device is a desk lamp; the desk lamp is equipped with a camera; the second process is used to run a hardware virtualization service, the first module is a device management module, and the second module is a transmission channel module; the data request includes a photo-taking request, and the target data includes captured image data; the method further includes: the hardware virtualization service registers a virtual camera corresponding to the camera in the system; the target application generates a photo-taking request when it calls the virtual camera.

[0031] Thus, this method can be applied to scenarios where electronic devices and IoT devices work together. Collaboration between electronic devices and IoT devices provides a better user experience. Compared to users holding electronic devices to take pictures, IoT devices capture more stable images without interfering with the user's operation of the target application on the electronic device. Furthermore, IoT devices do not need to have a display function, which reduces their cost.

[0032] According to the first aspect, or any implementation of the first aspect above, the data request also includes data parameters, where the data parameters are shooting modes, including normal shooting mode and continuous shooting mode; wherein,

[0033] The threshold for the number of shared memory maintained in the first shared memory pool that matches the normal shooting mode is less than the threshold for the number of shared memory maintained in the second shared memory pool that matches the burst shooting mode.

[0034] In this way, the shared memory maintenance threshold of the shared memory pool matched for burst shooting mode is greater than that of the shared memory maintenance threshold of the shared memory pool matched for normal shooting mode. This is more suitable for situations where the lamp returns a large number of captured images in a short period of time during burst shooting. Conversely, for requests where the lamp returns a small number of captured images in a short period of time during normal shooting mode, the shared memory pool is configured to maintain more shared memory to avoid resource waste.

[0035] According to the first aspect, or any implementation of the first aspect above, electronic devices include mobile phones and tablet computers.

[0036] Secondly, embodiments of this application provide an electronic device. The electronic device includes: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when executed by the one or more processors, the electronic device performs the data transmission method described in the first aspect and any implementation thereof.

[0037] The second aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the second aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0038] Thirdly, embodiments of this application provide a collaborative work system, including an electronic device that performs the data transmission method in the first aspect and any implementation thereof, and an Internet of Things (IoT) device. The IoT device is equipped with a camera for collecting image data, and a target application in the electronic device is bound to the IoT device.

[0039] The electronic device is used to: register a virtual camera corresponding to the camera in the system, and send an image preview request to the Internet of Things device by calling the virtual camera;

[0040] IoT devices are used to: retrieve preview image data from the camera based on an image preview request from an electronic device, and send the preview image data to the target application on the electronic device for preview display;

[0041] Electronic devices are also used to: generate a photo-taking request when a virtual camera is invoked, and send the photo-taking request to an Internet of Things (IoT) device;

[0042] IoT devices are also used to: invoke the camera to capture images based on the electronic device's photo-taking request, and send the captured image data to the target application on the electronic device for display.

[0043] In this way, electronic devices and IoT devices work together to provide users with a better user experience. Furthermore, data transmission speeds are higher and time is shorter when data from IoT devices is transmitted across processes to the target application. At the same time, compared to users holding electronic devices to take pictures, IoT devices capture more stable images without interfering with users' operations on the target application within their electronic devices. Additionally, IoT devices do not need to have display functions, which also reduces their cost.

[0044] According to the third aspect, the IoT device is a desk lamp, and the camera is used to collect image data downwards.

[0045] In this way, compared to users taking pictures with their handheld electronic devices, IoT devices capture more stable images without affecting users' operation of the target application on their electronic devices.

[0046] Fourthly, embodiments of this application provide a computer-readable storage medium. This computer-readable storage medium includes a computer program that, when executed on an electronic device, causes the electronic device to perform the data transmission method described in the first aspect and any implementation thereof.

[0047] The fourth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fourth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0048] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when run, causes a computer to execute the data transmission method described in the first aspect and any implementation thereof.

[0049] The fifth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the fifth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here.

[0050] Sixthly, this application provides a chip including a processing circuit and transceiver pins. The transceiver pins and the processing circuit communicate with each other via an internal connection path. The processing circuit executes the data transmission method described in the first aspect and any implementation thereof, to control the receiving pin to receive signals and to control the transmitting pin to transmit signals.

[0051] The sixth aspect and any implementation thereof correspond to the first aspect and any implementation thereof, respectively. The technical effects of the sixth aspect and any implementation thereof are similar to those of the first aspect and any implementation thereof, and will not be repeated here. Attached Figure Description

[0052] Figures 1a-1b This is an example of an application scenario;

[0053] Figure 2a A schematic diagram of the hardware structure of an electronic device as an example;

[0054] Figure 2b A schematic diagram of the software structure of an electronic device as an example;

[0055] Figure 3aThis is a schematic diagram of the hardware structure of an IoT terminal as an example.

[0056] Figure 3b This is a schematic diagram of the software structure of an IoT terminal as an example.

[0057] Figure 4a A schematic diagram of module interaction provided for an embodiment of this application;

[0058] Figure 4b A schematic diagram of module interaction provided for an embodiment of this application;

[0059] Figures 5a-5b This is an example of an application scenario;

[0060] Figure 6a A schematic diagram of module interaction provided for an embodiment of this application;

[0061] Figure 6b A schematic diagram of module interaction provided for an embodiment of this application;

[0062] Figures 7a-7b This is an example of an application scenario;

[0063] Figure 8 A schematic diagram of module interaction provided for an embodiment of this application;

[0064] Figures 9a-9c This is an example of an application scenario;

[0065] Figure 10 A schematic diagram of module interaction provided for an embodiment of this application.

[0066] Figures 11a-11b This is an example of an application scenario;

[0067] Figure 12 This is a schematic diagram of a communication interface for inter-process data transmission, as exemplarily shown.

[0068] Figure 13 This is an example illustrating the data copying process involved in inter-process data transfer.

[0069] Figure 14 This application provides a schematic diagram of the data copying process involved in inter-process data transmission in an embodiment of the present application.

[0070] Figure 15 A schematic diagram of module interaction during inter-process data transmission provided in an embodiment of this application;

[0071] Figure 16 This is a schematic diagram illustrating a shared memory pool management method as an example.

[0072] Figure 17 This is a schematic diagram illustrating a shared memory pool management method as an example. Detailed Implementation

[0073] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0074] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0075] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0076] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0077] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0078] With the development of the internet, online education has become increasingly popular, and users (such as students) have a growing demand for it. In some applications, students can look up unfamiliar words online for explanations; in others, online reading aloud of textbook content facilitates learning and pronunciation; and still others require students to submit their assignments online. Therefore, how to meet users' online education needs using smart devices is a problem that needs to be solved.

[0079] Currently, users in online education scenarios typically use smart learning devices that combine camera and display functions. These devices require a camera or reflector in a specific position to photograph the book, limiting their versatility and ease of use. Furthermore, devices with both camera and display functions require robust hardware and system support, resulting in high costs. Therefore, how to provide users with a better online education experience based on smart devices, improve versatility and ease of use, and reduce the cost of online education are problems that need to be addressed.

[0080] This application provides a collaborative work system applicable to online education scenarios. The system includes an electronic device and a desk lamp with a communication connection. The desk lamp is equipped with a camera that can be used to photograph books. The electronic device uses the desk lamp's camera to capture images and combines them with online education resources on the platform to meet the user's online education needs. The electronic device can be a tablet or mobile phone, etc. Besides online education scenarios, the electronic device and desk lamp can also serve users based on their respective basic functions (i.e., communication and lighting). In this way, the system can create a better online education experience for users based on two commonly used smart devices, offering strong versatility and ease of use. Furthermore, since tablets or mobile phones are already essential products in most households, and the cost of a desk lamp with only a camera function is lower than that of a device with both camera and display functions, it significantly reduces the cost of online education for users.

[0081] The following explanation uses a tablet computer as an example to illustrate the technical solution provided in this application.

[0082] Figure 1a An example application scenario is shown. For example... Figure 1a As shown, the collaborative work system includes a tablet 100 and a desk lamp 200 that establish a communication connection. The desk lamp 200 includes a camera 201 for capturing images downwards, such as by taking a picture of text or images in a book. An educational app is installed on the tablet 100. This app can access the camera 201 of the desk lamp 200 to capture images and provide users with various online educational functions based on the images captured by the camera 201, such as online word lookup, online reading aloud, and online homework submission.

[0083] Although tablets have both front and rear cameras, regardless of which camera is used to photograph a book, the user needs to hold the tablet and point the camera at the book. This not only results in unstable images but also interferes with manual reading and pointing operations, failing to provide a good online education experience. Figure 1aAs shown, both the tablet 100 and the desk lamp 200 can be placed in a fixed position. The tablet 100 uses the desk lamp 200's camera 201 to photograph the book. The stable image ensures a high success rate for content recognition, and users can also flexibly perform finger-pointing and point-reading operations on the book. Therefore, the interaction between the tablet and the desk lamp can provide users with a better online education experience.

[0084] like Figure 1b As shown, the tablet 100 and the desk lamp 200 can perform near-field communication or far-field communication. Near-field communication can be achieved through devices such as routers, while far-field communication can be achieved through a cloud server. For example, the tablet 100 and the desk lamp 200 can achieve near-field communication based on Wi-Fi (wireless fidelity) network protocols.

[0085] like Figure 2a The diagram shown is a structural schematic of electronic device 100. Optionally, electronic device 100 can be a terminal, also referred to as a terminal device. A terminal can be a cellular phone or a tablet computer, etc., and this application does not limit the scope. It should be noted that the structural schematic of electronic device 100 is applicable to various devices. Figures 1a-1b The tablet in the middle. It should be understood that, Figure 2a The electronic device 100 shown is only one example of an electronic device, and the electronic device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. Figure 2a The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0086] Electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include pressure sensors, gyroscope sensors, accelerometers, temperature sensors, motion sensors, barometric pressure sensors, magnetic sensors, distance sensors, proximity sensors, fingerprint sensors, touch sensors, ambient light sensors, bone conduction sensors, etc.

[0087] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0088] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0089] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory.

[0090] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0091] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0092] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0093] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0094] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0095] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.

[0096] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0097] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology.

[0098] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0099] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. In some embodiments, electronic device 100 may include one or N displays screens 194, where N is a positive integer greater than 1.

[0100] In this embodiment, the display screen 194 can display a shooting preview interface and a captured image interface, etc. It should be noted that, in this embodiment, the shooting preview interface refers to the interface through which the user can view the images captured in real-time by the lamp camera on the display screen 194.

[0101] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to perform data storage functions.

[0102] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121, such as enabling electronic device 100 to implement the cooperative working method and / or data transmission method in the embodiments of this application. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (e.g., sound playback function, image playback function, etc.). The data storage area may store data created during the use of electronic device 100 (e.g., audio data, phone book, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0103] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0104] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0105] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A. In some embodiments, the electronic device 100 may be equipped with multiple speakers 170A.

[0106] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0107] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0108] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0109] A pressure sensor is used to sense pressure signals and can convert these signals into electrical signals. In some embodiments, the pressure sensor may be located on the display screen 194. The electronic device 100 may also calculate the position of a touch based on the detection signal from the pressure sensor.

[0110] A touch sensor, also known as a "touch panel," can be located on the display screen 194. The touch sensor and display screen 194 together form a touchscreen, also called a "touch screen." The touch sensor detects touch operations applied to or near it. It then transmits the detected touch operation to the application processor to determine the type of touch event.

[0111] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0112] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.).

[0113] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0114] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0115] Figure 2b This is a software structure block diagram of an electronic device 100 according to an embodiment of this application.

[0116] The layered architecture of the electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the system layer, the HAL layer (Hardware Abstraction Layer), and the kernel layer.

[0117] The application layer can include a series of application packages.

[0118] like Figure 2b As shown, the application package can include calling, video, Bluetooth, camera, WLAN, educational applications, device management applications, etc. The application package can also include applications such as calendar, maps, navigation, music, and SMS.

[0119] Educational applications can be used to provide users with online educational functions, such as online vocabulary learning, online reading aloud, and online homework submission.

[0120] In some examples, device management apps can be used to bind IoT (Internet of Things) devices such as desk lamps. In other examples, educational apps can enable the binding of IoT (Internet of Things) devices such as desk lamps.

[0121] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0122] like Figure 2b As shown, the application framework layer may include camera services, authentication services, hardware virtualization services, device management services, transmission management services, etc.

[0123] The camera service can be used to invoke the camera (including the front camera and / or the rear camera) in response to application requests.

[0124] In this embodiment of the application, the camera service can be used to call the virtual camera on the electronic device side in response to the application's request, that is, to call the camera in the IoT device.

[0125] The authentication service is used to provide secure access control capabilities.

[0126] Hardware virtualization services can be used to establish a logical channel between the electronic device side (i.e., the central device side) and the IoT device side, providing the ability to virtualize cameras.

[0127] Device management services can be used for IoT device discovery and management, providing information on far-field (i.e., cloud-based) IoT devices and near-field (i.e., nearby connectable) IoT devices for applications such as educational applications.

[0128] Transmission management services can be used to establish physical transmission channels and provide data transmission capabilities.

[0129] In addition, it can include window managers, content providers, view systems, phone managers, resource managers, notification managers, etc.

[0130] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0131] The content provider stores and retrieves data, making it accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon may include views for displaying text and views for displaying images. The phone manager provides communication functionality for the electronic device 100. For example, it manages call status (including connection, hang-up, etc.). The resource manager provides various resources to applications, such as localized strings, icons, images, layout files, video files, etc. The notification manager allows applications to display notification information in the status bar, which can be used to convey informational messages and can disappear automatically after a short pause without user interaction.

[0132] The system library and runtime layer (i.e., the system layer) includes the system library and the Android Runtime.

[0133] The Android Runtime consists of core libraries and a virtual machine. The Android Runtime is responsible for the scheduling and management of the Android system.

[0134] The core library consists of two parts: one part contains the functionalities that the Java language needs to call, and the other part is the Android core library. The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0135] In this embodiment, the Android Runtime also includes a virtual camera adaptation layer, providing the ability to register a virtual camera.

[0136] The system library in the system layer can include multiple functional modules. For example, multimedia platforms, graphics and image processing libraries, encoding and decoding, etc.

[0137] The multimedia platform can be used for multimedia management, supporting playback and recording of various common audio and video formats, as well as still image files. The multimedia platform supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0138] Graphics and image processing libraries can be used to implement graphics drawing, image rendering, compositing, and layer processing, etc.

[0139] Codecs can be used to perform encoding and decoding operations on audio and video data.

[0140] The HAL layer is the interface layer located between the operating system kernel and the hardware circuitry. The HAL layer includes, but is not limited to: audio HAL, sensor HAL, modem HAL, camera HAL, and virtual camera HAL.

[0141] The audio HAL is used to process audio streams, such as performing noise reduction and directional enhancement. The camera HAL is used to process the image stream corresponding to the camera on the electronic device side, and the virtual camera HAL is used to process the image stream corresponding to the virtual camera registered on the electronic device side, that is, to process the image stream captured by the camera on the IoT device side.

[0142] The kernel layer is the layer between hardware and software. It includes at least display drivers, camera drivers, audio drivers, network drivers (such as Wi-Fi drivers), CPU drivers, USB drivers, storage drivers, and printer drivers. The hardware includes at least a processor, a display screen, and a Wi-Fi module.

[0143] Understandable, Figure 2bThe layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0144] like Figure 3a The diagram shown illustrates the hardware structure of the IoT device 200. It should be noted that this schematic diagram of the IoT device 200 can be applied to... Figures 1a-1b The desk lamp in the middle. It should be understood that... Figure 3a The IoT device 200 shown is just one example of an electronic device, and the IoT device 200 may have more or fewer components than shown in the figure, may combine two or more components, or may have different component configurations. Figure 3a The various components shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0145] The Internet of Things (IoT) device 200 may include: a processor 210, a camera 201, a wireless communication module 202, a memory 203, an audio module 204, a USB interface 205, a charging management module 206, a power management module 207, a battery 208, a lighting device 209, a button 211, etc.

[0146] Processor 210 may include one or more processing units, such as a GPU, ISP, controller, memory, video codec, etc. These different processing units may be independent devices or integrated into one or more processors.

[0147] The controller can serve as the nerve center and command center of an IoT device 200. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0148] Camera 201 is used to capture still images or videos. An object is projected onto a photosensitive element through a lens, generating an optical image. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for further processing. Taking a desk lamp as an example, camera 201 can be mounted on the lamp stand to capture images downwards.

[0149] The Internet of Things (IoT) device 200 can perform shooting functions through ISP, camera 201, video codec, GPU, etc.

[0150] The ISP (Image Signal Processor) is used to process data fed back from the camera 201. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 201.

[0151] The wireless communication module 202 can provide wireless communication solutions, including WLAN (such as Wi-Fi networks) and Bluetooth (BT), for use on the IoT device 200. In some embodiments, the antenna of the IoT device 200 is coupled to the wireless communication module 202, enabling the IoT device 200 to communicate with networks and other devices via wireless communication technology.

[0152] The memory 203 can be used to store computer executable program code, which includes instructions. The processor 210 executes various functional applications and data processing of the Internet of Things (IoT) device 200 by running the instructions stored in the memory 203, such as enabling the IoT device 200 to implement the collaborative working method in the embodiments of this application.

[0153] The Internet of Things (IoT) device 200 can realize audio functions, such as music playback, through audio module 204, speaker 212, etc.

[0154] USB interface 205 is a USB standard compliant interface, specifically a Mini USB interface, Micro USB interface, USB Type-C interface, etc. USB interface 205 can be used to connect a charger to charge IoT device 200, and can also be used for data transfer between IoT device 200 and peripheral devices.

[0155] The charging management module 206 receives charging input from the charger. While charging the battery 208, the charging management module 206 can also supply power to the IoT device 200 through the power management module 207.

[0156] The power management module 207 is used to connect the battery 208, the charging management module 206, and the processor 210. The power management module 207 receives input from the battery 208 and / or the charging management module 206 to power the processor 210, memory 203, camera 201, wireless communication module 202, and lighting device 209, etc.

[0157] Button 211 includes the power button (or power button), etc.

[0158] The software system of the IoT device 200 can adopt a layered architecture or other architectures. This application embodiment uses a layered architecture as an example to illustrate the software structure of the IoT device 200.

[0159] Figure 3b This is a software structure block diagram of the Internet of Things device 200 according to an embodiment of this application.

[0160] The layered architecture of the IoT device 200 divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the IoT device 200 system is divided into three layers, from top to bottom: the application layer, the application framework layer, the system layer, and the kernel layer.

[0161] like Figure 3b As shown, the application layer can include device application services, which can be understood as system-level applications. The device application services are started after the IoT device 200 system starts.

[0162] like Figure 3b As shown, the application framework layer may include device interconnection services, hardware abstraction services, resource manager, etc.

[0163] Device interconnection services can be used to establish physical transmission channels, provide data transmission capabilities, and manage the activation / deactivation of hardware abstraction services.

[0164] Hardware abstraction services can be used to establish logical channels between electronic devices (i.e., central devices) and IoT devices, providing virtualized camera capabilities and open interfaces for IoT devices' cameras.

[0165] File Explorer can provide various resources for applications.

[0166] like Figure 3b As shown, the system layer may include a multimedia platform, graphics and image processing library, codec, device adaptation module, etc.

[0167] The multimedia platform can be used to manage multimedia files, supporting various commonly used audio, video, and still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0168] Graphics and image processing libraries can be used to implement graphics drawing, image rendering, compositing, and layer processing, etc.

[0169] Codecs can be used to perform encoding and decoding operations on audio and video data.

[0170] The device adaptation module can implement the interface of hardware abstraction service, provide device information and capability query, and also provide functions to perform related operations on the IoT device side, such as opening the camera, taking pictures, and previewing.

[0171] It is understood that, in order to implement the collaborative working method in the embodiments of this application, the electronic device 100 and the Internet of Things device 200 include hardware and / or software modules that perform various functions. Based on the algorithmic steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0172] like Figure 4a The diagram shows the interaction between each module. (Refer to...) Figure 4a This application provides a method flow for a tablet and a desk lamp to work together, specifically including:

[0173] 0. Device service initialization phase

[0174] S0.1 In response to the user's operation, the desk lamp's device application service is started, and the device interconnection service is loaded.

[0175] For example, a user action could be turning on the power to the desk lamp. In response to the user action, the desk lamp system starts, the device application service starts, and the device interconnection service is loaded. This device interconnection service can be used to establish a physical transmission channel between the tablet and the desk lamp, providing data transmission capabilities.

[0176] S0.2, The device interconnection service of the desk lamp loads the hardware abstraction service.

[0177] The Device Connectivity Service can also control the activation of the Hardware Abstraction Service. For example, after the Device Connectivity Service is started, it can load the Hardware Abstraction Service as a plugin. This Hardware Abstraction Service can be used to establish a logical channel between the tablet and the lamp, provide virtualized camera capabilities, and also provide an open interface for the lamp's camera.

[0178] Reference Figure 4bThe illustrated module interaction diagram shows that the hardware abstraction service can include at least basic components and a camera component. During the device service initialization phase, the device interconnection service first loads and initializes the basic components. After the basic components are initialized, it can interact with the lamp's device adaptation module to obtain device information and virtualization capability information. For example, device information includes, but is not limited to, device name, device identifier, and device type. For example, virtualization capability information includes, but is not limited to, whether virtualized camera and virtualized microphone are supported.

[0179] Among them, the desk lamp has the ability to support virtual cameras, which can be understood as the desk lamp's camera being accessible to other electronic devices (such as tablets), or in other words, the desk lamp's camera being used as a virtual camera for other electronic devices.

[0180] After the basic components obtain the device and capability information of the desk lamp, if the desk lamp supports virtualized cameras, the basic components load the camera component to provide virtualized camera capabilities. At this point, the basic components can prepare for establishing a negotiation channel to negotiate network connection information (including but not limited to IP address and port) with the tablet. Specifically, during the basic components' preparation for establishing the negotiation channel, a session service is created, and the session name of the session service is sent to the device interconnection service, pending the establishment of a negotiation channel between the tablet's transmission management service and the desk lamp's device interconnection service.

[0181] 1. Equipment discovery phase

[0182] S1.1 In response to user actions, the tablet's educational app sends a device discovery command to the device management service.

[0183] User actions can refer to actions performed by a user within an educational app that require access to the virtual camera. For example, user actions could include clicking on functions such as the point-and-read function, word lookup function, homework assignment function, or photo-taking function within the educational app.

[0184] The tablet's educational app receives a user action and, in response, sends a device discovery command to the tablet's device management service. This device discovery command instructs the search for IoT devices capable of connecting to the tablet. For example, the device discovery command may include, but is not limited to, the command type and the type of device to be discovered. In this embodiment, the device discovery command is specifically used to find a desk lamp capable of connecting to the tablet.

[0185] S1.2, the device management service in the tablet calls the authentication service to authenticate the education app and obtains the authentication result of the education app.

[0186] After receiving a device discovery command, the device management service can obtain the name (or identifier) ​​of the education app based on existing mechanisms in the Android system, and authenticate the education app based on its name. Specifically, the device management service can call the authentication service to authenticate the education app and obtain the authentication result.

[0187] In this embodiment, the tablet-side application framework layer also provides a device management API corresponding to the device management service, and a hardware virtualization API corresponding to the hardware virtualization service. To implement the technical solution provided in this embodiment, the educational app needs to register on a relevant platform (e.g., a platform provided by the tablet manufacturer), adapt to the framework of the device management service, hardware virtualization service, and transmission management service, and apply for permissions to the device management API and hardware virtualization API.

[0188] For example, the authentication service accesses the authentication server to authenticate the educational app, including but not limited to whether it is registered on the relevant platform, whether it is compatible with the relevant framework, and whether it has applied for the relevant API permissions.

[0189] For example, the authentication service can authenticate educational apps based on a local whitelist.

[0190] Once the authentication service receives the authentication result (successful or failed) from the educational app, it sends the authentication result to the device management service.

[0191] S1.3, when the education APP successfully authenticates, the device management service in the tablet sends a device search command to the transmission management service.

[0192] If the educational app successfully authenticates, the device management service sends a device search command to the transmission management service. This device search command may include, but is not limited to, the command type, the type of device to be searched, and the search method. For example, search methods may include, but are not limited to, near-field device scanning and obtaining device information from a cloud server. In this implementation, the device type to be searched is a desk lamp.

[0193] S1.4, the transmission management service in the tablet obtains the list of near and far field devices according to the device search command, and sends the list of near and far field devices to the device management service.

[0194] The far-field and near-field device lists include a far-field device list and a near-field device list. The far-field device list includes registered devices obtained from the cloud server, while the near-field device list includes devices scanned via near-field communication. Device information in both lists includes, but is not limited to, device name, device identifier, and device type.

[0195] When the transmission management service receives a device search instruction, it performs relevant device search operations based on the type of device to be searched and the search method carried in the device search instruction. These operations include performing near-field device scanning and retrieving relevant device information from the cloud server. The service then obtains a far-field device list and a near-field device list and sends these lists to the device management service.

[0196] S1.5, the device management service in the tablet filters devices based on the near and far field device list and reports the filtered device information to the education APP.

[0197] The device management service filters devices based on the far-field device list and the near-field device list to determine the desk lamp information that can interact with the tablet, and then sends this information to the educational app. Specifically, the device management service can perform an intersection operation on the far-field and near-field device lists, filtering out desk lamps that exist only in either the far-field or near-field device list, and identifying those that exist in both lists as desk lamps capable of interacting with the tablet. In this way, the device management service can filter out desk lamps not registered in the cloud server, as well as those that cannot establish near-field communication with the tablet.

[0198] In another optional implementation, regardless of whether the tablet and the desk lamp are on the same local area network, the tablet's transmission management service can obtain a list of communication devices and a list of registered devices based on device search commands. The devices included in the communication device list are those scanned via near-field communication or far-field communication, while the devices included in the registered device list are those already registered and obtained from the cloud server. The near-field devices included in the near-field device list are those scanned via near-field communication. Device information in the communication device list and the registered device list includes, but is not limited to, device name, device identifier, and device type.

[0199] The tablet's transmission management service sends the communication device list and registered device list to the device management service. The device management service filters devices based on these lists and reports the filtered device information to the educational app. Specifically, the device management service can perform an intersection operation on the communication device list and registered device list, filtering out desk lamps that exist only in either the communication device list or the registered device list. Desk lamps that exist in both lists are considered eligible for tablet interaction. This allows the device management service to filter out desk lamps not registered with the cloud server and those unable to establish near-field communication with the tablet.

[0200] 2. Virtual Camera Enablement Phase

[0201] S2.1, the tablet's educational app identifies the desk lamp to be linked.

[0202] The device management service can filter out one or more desk lamps that can be linked with the tablet. When there is only one desk lamp, the education app will use it as the default desk lamp to be linked. When there are multiple desk lamps, the education app can display a list of desk lamps to be linked for the user to choose from. In response to the user's selection, the education app will use the selected desk lamp as the desk lamp to be linked.

[0203] It should be noted that the step of the educational APP in determining the desk lamp to be linked can also be classified as the device discovery stage, and this embodiment does not limit this.

[0204] S2.2, the tablet's educational app verifies and connects the desk lamp, and confirms that the desk lamp has the capability to support a virtual camera.

[0205] S2.3, the tablet's educational app sends a virtual camera enable request to the hardware virtualization service.

[0206] After the tablet's educational app detects that the desk lamp supports a virtual camera, it sends a virtual camera enable request to the hardware virtualization service. This virtual camera enable request instructs the user to register the virtual camera in the virtual camera HAL (Hardware Ability Level). The virtual camera enable request may include, but is not limited to, request type, device name, device identifier, device type, and the virtual camera's identifier.

[0207] S2.4, the tablet's hardware virtualization service registers the virtual camera with the virtual camera HAL.

[0208] After receiving the virtual camera enable request, the hardware virtualization service registers the corresponding virtual camera with the virtual camera HAL according to the virtual camera enable request.

[0209] S2.5 After the virtual camera registration is completed, the tablet's virtual camera HAL sends a virtual camera enable success instruction to the educational app.

[0210] The following is combined Figure 4b The module interaction diagram shown provides a detailed explanation of the process during the virtual camera enablement phase. (Refer to...) Figure 4b The virtual camera enabling phase mainly includes the device verification sub-phase (S301-S309), the device connection sub-phase (S310-S321), the device service capability request sub-phase (S322-S325), and the virtual camera enabling sub-phase (S326-S331).

[0211] Reference Figure 4b The virtual camera enabling phase process specifically includes the following steps:

[0212] In the S301 tablet, the educational app sends a virtual camera enable command to the hardware virtualization API.

[0213] The virtual camera enable command is used to instruct the virtual camera to be enabled. The virtual camera enable command may include, but is not limited to, command type, device name, device identifier, and device type.

[0214] S302, after receiving the virtual camera enable command, the hardware virtualization API in the tablet sends a device verification command to the interface scheduling module of the hardware virtualization service.

[0215] The device verification command is used to instruct the verification of the device information carried in the virtual camera enable command. The device verification command may include, but is not limited to, command type, device name, device identifier, and device type.

[0216] S303, the interface scheduling module of the hardware virtualization service in the tablet sends the APP authentication command to the permission management module of the hardware virtualization service.

[0217] After receiving the device verification command, the interface scheduling module of the hardware virtualization service first sends an APP authentication command to the permission management module of the hardware virtualization service to authenticate the APP that initiated the virtual camera enable command. The APP authentication command may include, but is not limited to, the APP name.

[0218] S304, the permission management module of the hardware virtualization service in the tablet performs app authentication for educational apps.

[0219] For example, the permission management module can access the authentication server to authenticate the educational app, including but not limited to verifying whether it is registered on the relevant platform, whether it is compatible with the relevant framework, and whether it has applied for the relevant API permissions. The permission management module can access the authentication server through the authentication service; this embodiment does not limit this.

[0220] S305, when the hardware virtualization service permission management module in the tablet successfully authenticates the education APP, it sends an authentication success indication to the interface scheduling module.

[0221] After the permission management module receives the authentication result of the education app, if the education app is successfully authenticated, it sends an authentication success indication to the interface scheduling module. If the education app fails to authenticate, it sends an authentication failure indication to the hardware virtualization API, so that the hardware virtualization API returns an indication message to the education app that the app does not have permission based on the authentication failure indication.

[0222] S306, when the interface scheduling module of the hardware virtualization service in the tablet confirms that the education APP has been successfully authenticated, it sends a device verification command to the device management module.

[0223] Upon receiving the authentication success indication, the interface scheduling module of the hardware virtualization service, after confirming that the education app has been successfully authenticated, sends a device verification command to the device management module. This device verification command is used to verify the status of the device to be linked; in this embodiment, it is specifically used to verify the status of the desk lamp to be linked. For example, the device verification command may include, but is not limited to, command type, device name, device identifier, and device type.

[0224] S307, the device management module of the hardware virtualization service in the tablet sends a device information query command to the device profile module of the device management service.

[0225] The device profile module of the device management service stores information about currently online devices.

[0226] After receiving the device verification command, the device management module of the hardware virtualization service sends a device information query command to the device profile module of the device management service. The device information query command may include, but is not limited to, device name, device identifier, and device type.

[0227] S308, the device profile module of the device management service in the tablet returns device information to the device management module of the hardware virtualization service.

[0228] If the device description module of the device management service finds a corresponding device based on the device information query command, it returns the device information to the device management module of the hardware virtualization service. The returned device information may include, but is not limited to, device name, device identifier, device type, and online status.

[0229] If the device description module of the device management service does not find the corresponding device based on the device information query command, it returns a null value to the device management module of the hardware virtualization service to indicate that the corresponding device was not found. At this time, the device management module of the hardware virtualization service can send a device verification failure indication to the hardware virtualization API, so that the hardware virtualization API can return device verification failure indication information to the education APP based on the device verification failure indication.

[0230] S309, after receiving the device information, the device management module of the hardware virtualization service in the tablet sends a device verification success indication to the hardware virtualization API.

[0231] If the device management module of the hardware virtualization service receives the device information returned by the device profile module of the device management service, it will send a device verification success indication to the hardware virtualization API to indicate that the table lamp to be linked has been successfully verified.

[0232] S310, the hardware virtualization API in the tablet sends a device connection request to the device management module of the hardware virtualization service.

[0233] After confirming the successful verification of the desk lamp to be linked, the hardware virtualization API sends a device connection request to the device management module of the hardware virtualization service. This device connection request instructs the establishment of a network connection with the device to be linked; in this embodiment, it specifically instructs the establishment of a network connection with the desk lamp to be linked. The device connection request may include, but is not limited to, request type, device name, device identifier, and device type.

[0234] S311 After receiving the device connection request, the device management module of the hardware virtualization service performs the negotiation channel establishment preparation operation and sends a negotiation channel opening request to the transmission management service.

[0235] After receiving a device connection request, the device management module of the hardware virtualization service prepares a negotiation channel. During this preparation, the module creates a Session Server and sends the Session Name to the transport management service. After preparing the negotiation channel, it sends a negotiation channel open request to the transport management service. This request instructs the establishment of a negotiation channel and may include, but is not limited to, the peer device identifier (i.e., the lamp identifier) ​​and the Session Name.

[0236] In this embodiment, the request to open the negotiation channel is initiated by the tablet, meaning the tablet needs to establish a connection with the desk lamp. In this case, the desk lamp can be understood as the server, and the tablet as the client that needs to access the server.

[0237] S312, the transmission management service in the tablet and the device interconnection service in the desk lamp establish a negotiation channel.

[0238] After receiving the negotiation channel opening request, the transmission management service interacts with the device interconnection service in the lamp based on the Session Name to establish a negotiation channel. Establishing a negotiation channel can specifically involve creating a session and determining the session identifier.

[0239] S313, The Device Interconnection Service in the desk lamp sends a successful negotiation channel establishment indication to the camera component in the Hardware Abstraction Service.

[0240] After the negotiation channel is established, the device interconnection service in the desk lamp sends a successful negotiation channel establishment indication to the camera component in the hardware abstraction service, indicating that the negotiation channel has been established and that there are devices that need to establish a connection. This successful negotiation channel establishment indication may include, but is not limited to, information about the device that needs to establish a connection (i.e., tablet device information) and a session identifier.

[0241] S314, The transmission management service in the tablet sends a successful negotiation channel establishment indication to the device management module of the hardware virtualization service.

[0242] After the negotiation channel is established, the tablet's transmission management service sends a successful negotiation channel establishment indication to the device management module of the hardware virtualization service, indicating that the negotiation channel has been established and that there are devices that need to establish a connection. This successful negotiation channel establishment indication may include, but is not limited to, information about the devices that need to establish a connection (i.e., the desk lamp device information) and a session identifier.

[0243] This embodiment does not limit the execution order of S313 and S314.

[0244] S315, the device management module of the hardware virtualization service in the tablet sends a device information negotiation request to the camera component of the hardware abstraction service in the desk lamp based on the negotiation channel.

[0245] The equipment negotiation request may include, but is not limited to, equipment information (such as equipment name, equipment identifier, equipment type, etc.) and control channel connection requests.

[0246] S316 After receiving the device negotiation request, the camera component of the hardware abstraction service in the desk lamp prepares the control channel and returns the device negotiation information to the device management module of the hardware virtualization service in the tablet.

[0247] After receiving a device negotiation request, the camera component of the hardware abstraction service in the desk lamp parses the device negotiation request, obtains and records the device information of the other end, determines an IP address and port to be listened to based on the control channel connection request, adds the IP address and port to the device negotiation information, and returns it to the device management module of the hardware virtualization service in the tablet.

[0248] It should be noted that device negotiation requests and device negotiation information are transmitted based on the established negotiation channel.

[0249] S317. After receiving the returned device negotiation information, the device management module of the hardware virtualization service in the tablet closes the negotiation channel.

[0250] Specifically, closing the negotiation channel can mean closing the session. Once the device management module of the hardware virtualization service in the tablet receives the returned device negotiation information, the session ends, and the device management module of the hardware virtualization service can close the corresponding session based on the session identifier.

[0251] S318, the device management module of the hardware virtualization service in the tablet sends a control channel opening request to the transmission management service.

[0252] A control channel open request is used to instruct the establishment of a network communication connection with the desk lamp. This request may include, but is not limited to, a communication protocol, source IP address, source port, destination IP address, and destination port. The destination IP address and destination port are the IP address and port that the camera component in the desk lamp's hardware abstraction service is listening on.

[0253] S319, the transmission management service in the tablet connects to the control channel of the camera component in the hardware abstraction service of the desk lamp, and sends a control channel successful connection indication to the device management module of the hardware virtualization service in the tablet.

[0254] Upon receiving a control channel opening request, the tablet's transmission management service establishes a control channel connection with the desk lamp based on the information carried in the request, thus establishing a network communication connection between the tablet and the desk lamp. Subsequently, the device management module of the tablet's hardware virtualization service and the camera component of the desk lamp's hardware abstraction service can communicate via this control channel.

[0255] After the control channel is successfully established, the transmission management service in the tablet sends a control channel successful connection indication to the device management module of the hardware virtualization service in the tablet. The control channel successful connection indication may include, but is not limited to, a connection success identifier and control channel related information.

[0256] S321, the device management module of the hardware virtualization service in the tablet sends a device connection success indication to the hardware virtualization API.

[0257] The device connection success indication may include, but is not limited to, a connection success identifier and connected device information.

[0258] S322, the hardware virtualization API in the tablet sends a device capability request to the device management module of the hardware virtualization service.

[0259] Upon receiving a successful device connection indication, the hardware virtualization API sends a device capability request to the device management module of the hardware virtualization service. This device capability request can be used to request virtualization capability information of the peer device (i.e., the desk lamp). For example, virtualization device capability information includes, but is not limited to, whether it supports a virtual camera or a virtual microphone.

[0260] S323, the device management module of the hardware virtualization service in the tablet sends a device capability request to the camera component of the hardware abstraction service in the desk lamp in the control channel.

[0261] S324, the camera component of the hardware abstraction service in the desk lamp returns device capability information to the device management module of the hardware virtualization service in the tablet via the control channel.

[0262] In this embodiment, the returned device capability information of the desk lamp may include at least the ability to support virtualized cameras and the camera identifier of the desk lamp.

[0263] S325, the device management module of the hardware virtualization service in the tablet sends device capability information to the hardware virtualization API.

[0264] The device management module of the hardware virtualization service in the tablet sends the received device capability information to the hardware virtualization API so that the hardware virtualization API can know whether the desk lamp has the capability to support virtualized cameras.

[0265] S326, the hardware virtualization API in the tablet sends a virtual camera enable request to the device management module of the hardware virtualization service.

[0266] The tablet's hardware virtualization API learns that the desk lamp has the capability to support a virtual camera and sends a virtual camera enable request to the device management module of the hardware virtualization service. This virtual camera enable request may include, but is not limited to, the request type and the desk lamp's camera identifier.

[0267] S327, the device management module of the hardware virtualization service in the tablet registers the virtual camera in the virtual camera HAL.

[0268] Upon receiving a virtual camera enable request, the device management module of the hardware virtualization service sends a virtual camera registration request to the virtual camera HAL. The virtual camera registration request may include, but is not limited to, the request type and the lamp's camera identifier. After receiving the virtual camera registration request, the virtual camera HAL registers a virtual camera driver for the lamp's camera and assigns a camera ID (i.e., virtual camera ID) to it, registering this camera ID in the system. This establishes a mapping relationship between the lamp's camera and the virtual camera within the virtual camera HAL.

[0269] S328, the device management module of the hardware virtualization service in the tablet sends a service status update instruction to the camera component of the hardware abstraction service in the desk lamp.

[0270] A service status update indicator is used to instruct the camera component of the hardware abstraction service in the desk lamp to update its virtualization service status. The virtualization service status can include an occupied state, an unoccupied state, or a registered state, an unregistered state. For example, the service status update indicator may include, but is not limited to, the device information of the peer device (i.e., the desk lamp), a hardware identifier (such as the desk lamp camera identifier), and the virtualization service status corresponding to the hardware identifier.

[0271] S329, the camera component of the hardware abstraction service in the desk lamp updates its service status according to the service status update instruction.

[0272] When the service status update indicator indicates that the virtualization service status corresponding to the lamp camera is in an occupied state (or registered state), the camera component will update the virtualization service status corresponding to its lamp camera to an occupied state (or registered state).

[0273] In the S330 tablet, the device management module of the hardware virtualization service sends a virtual camera enable success indication to the hardware virtualization API.

[0274] The virtual camera enable success indication may include, but is not limited to, an enable success identifier (or virtualization success identifier), a camera identifier for the desk lamp, and a camera ID corresponding to the virtual camera (or a camera ID corresponding to the desk lamp camera).

[0275] This embodiment does not limit the execution order of S328 and S330.

[0276] S331, the tablet's hardware virtualization API sends a virtual camera enable success indication to the educational app.

[0277] 3. Virtual camera preview access stage

[0278] S3.1, the hardware virtualization API in the tablet sends a virtual camera access command to the camera service.

[0279] Virtual camera access commands refer to commands used to invoke a virtual camera. These commands may include, but are not limited to, command type, virtual camera ID, and camera configuration parameters, including but not limited to camera resolution and frame rate.

[0280] S3.2, the camera service in the tablet sends an image preview request to the virtual camera HAL according to the virtual camera access command.

[0281] After receiving the virtual camera access command, the camera service generates a corresponding image preview request based on the virtual camera ID and sends it to the virtual camera HAL. This image preview request is used to request a preview of the image data stream. For example, the image preview request may include, but is not limited to, a request identifier, the virtual camera ID, and camera configuration parameters.

[0282] S3.3, the virtual camera HAL in the tablet sends an image preview request to the hardware virtualization service.

[0283] After receiving an image preview request, the virtual camera HAL determines the matching virtualized hardware identifier based on the virtual camera ID carried in the image request. In this embodiment, the virtual camera HAL determines the linked desk lamp camera based on the virtual camera ID and the mapping relationship between the virtual camera ID and the desk lamp camera, and generates a corresponding image request based on the determined virtualized hardware identifier, sending it to the hardware virtualization service. For example, the image preview request may include, but is not limited to, a request identifier, device information (i.e., desk lamp information), a virtualized hardware identifier (i.e., desk lamp camera identifier), and camera configuration parameters.

[0284] S3.4, the hardware virtualization service in the tablet sends an image preview request to the transmission management service.

[0285] The hardware virtualization service sends the image preview request to the transmission management service. The image preview request may include, but is not limited to, a request identifier, device information (i.e., lamp information), virtualization hardware identifier (i.e., lamp camera identifier), and camera configuration parameters.

[0286] When the hardware virtualization service in the tablet sends an image preview request to the transmission management service, if it finds that no data channel has been established with the desk lamp, it generates a data channel establishment request and sends it to the transmission management service. The data channel establishment request is used to indicate the transmission of data with the desk lamp. The data channel establishment request may include, but is not limited to, session identifiers, connection information, and data encoding / decoding modes.

[0287] Upon receiving a data channel establishment request, the tablet's transmission management service establishes a data channel connection with the desk lamp based on the information carried in the request, thus establishing a data channel between the tablet and the desk lamp. Subsequently, the tablet's transmission management service and the desk lamp's device interconnection service can transmit various types of data, including but not limited to image data, through this data channel.

[0288] After the data channel is successfully established, the transmission management service in the tablet sends a data channel successful connection indication to the hardware virtualization service in the tablet, and the device interconnection service in the desk lamp sends a data channel successful connection indication to the camera component in the hardware abstraction service. The data channel successful connection indication may include, but is not limited to, a connection success identifier and data channel-related information.

[0289] S3.5, the tablet's transmission management service transmits an image preview request to the desk lamp's device interconnection service.

[0290] The tablet's transmission management service determines the appropriate control channel based on the device information carried in the image preview request, and transmits the image preview request to the lamp's device interconnection service through that control channel.

[0291] In S3.6, the device interconnection service in the desk lamp sends an image preview request to the camera driver.

[0292] After receiving the image preview request, the device interconnection service in the desk lamp determines the corresponding hardware driver (in this embodiment, the camera driver) based on the virtualized hardware identifier, and sends the corresponding image preview request to the camera driver.

[0293] S3.7, the camera driver in the desk lamp drives the camera to capture images and transmits the preview image data to the tablet's hardware virtualization service through the data channel.

[0294] The camera driver activates the camera and captures an image using the camera configuration parameters carried in the image preview request, obtaining a preview image data stream. This preview image data stream is then sent to the device interconnect service via the hardware abstraction service, enabling the device interconnect service to continuously transmit the preview image data stream to the tablet's hardware virtualization service in the data channel. The packetization, assembly, and encoding / decoding processes for the preview image data stream are not detailed here.

[0295] In S3.8, the hardware virtualization service in the tablet sends preview image data to the virtual camera HAL.

[0296] The hardware virtualization service continuously receives preview image data streams and sends them to the virtual camera HAL.

[0297] In S3.9, the virtual camera HAL in the tablet sends preview image data to the camera service.

[0298] At this time, the virtual camera HAL continuously acquires the preview image data captured by the lamp camera and continuously sends the preview image data to the camera service.

[0299] In S3.10, the tablet's camera service sends preview image data to the educational app.

[0300] S3.11, the educational app on the tablet displays a preview image.

[0301] Once the educational app receives the preview image data stream through the camera service, it can display the preview image on the corresponding interface.

[0302] 4. Virtual camera photo taking stage

[0303] S4.1 In response to the received user operation, the educational app on the tablet sends a photo-taking request to the hardware virtualization service.

[0304] User actions could include, for example, clicking the camera option. In response to the received user action, the educational app on the tablet sends a photo request to the hardware virtualization service. This photo request may include, but is not limited to, the image sequence number, device information (i.e., lamp information), virtualization hardware identifier (i.e., lamp camera identifier), and camera configuration parameters. Camera configuration parameters include, but are not limited to, image resolution.

[0305] In addition, the photo request can also carry a task identifier to ensure the orderly management of multiple photo tasks.

[0306] S4.2, the hardware virtualization service in the tablet sends a photo-taking request to the transmission management service.

[0307] In S4.3, the tablet's transmission management service will transmit the photo-taking request to the desk lamp's device interconnection service.

[0308] The tablet's transmission management service determines the appropriate control channel based on the device information carried in the image preview request, and transmits the image preview request to the lamp's device interconnection service through that control channel.

[0309] In S4.4, the device interconnection service in the desk lamp sends a photo-taking request to the camera driver.

[0310] After receiving the image preview request, the device interconnection service in the desk lamp determines the corresponding hardware driver (in this embodiment, the camera driver) based on the virtualized hardware identifier, and sends the corresponding photo-taking request to the camera driver.

[0311] S4.5, the camera driver in the desk lamp drives the camera to capture images and transmits the captured image data to the tablet's hardware virtualization service through the data channel.

[0312] The camera driver uses the camera configuration parameters carried in the photo capture request to acquire images, obtain captured image data, and send the captured image data to the device interconnect service through the hardware abstraction service. This allows the device interconnect service to continuously transmit the captured image data to the tablet's hardware virtualization service in the data channel. The packetization, assembly, and encoding / decoding process of the captured image data will not be elaborated here.

[0313] In S4.6, the hardware virtualization service in the tablet sends the captured image data to the educational app.

[0314] The S4.7 tablet displays captured images in an educational app.

[0315] Once the educational app receives the captured image through the hardware virtualization service, it can display the captured image on the corresponding interface.

[0316] In this embodiment, the virtual camera preview access stage is implemented based on the native Android camera framework, while the virtual camera capture stage is implemented based on a proprietary virtualized camera framework. This results in a shorter processing path and lower shooting latency in the virtual camera capture stage. Furthermore, since the image preview is still implemented based on the native Android camera framework, the educational app requires fewer modifications to adapt to the technical solution provided in this embodiment.

[0317] It should be noted that the stage divisions in the above process are merely illustrative and are not intended to limit the scope of this application. Furthermore, after the virtual camera preview access stage process displays the preview image on the tablet, the real-time preview image display process and the virtual camera capture stage process can be executed simultaneously. Any aspects of the above process not explained in detail can be referred to existing technologies, and will not be elaborated upon here.

[0318] Figure 4a The paper presents the communication architecture of the collaborative work system, which is used to manage the virtual camera (i.e., the lamp camera) on the tablet, the control command interaction between the tablet and the lamp, and the transmission and processing of image data.

[0319] It should be noted that instructions and requests transmitted across devices (i.e., between the tablet and the lamp) need to be encapsulated based on communication protocols and parameter sequences, which will not be elaborated upon in this embodiment. The hardware virtualization service in the tablet can also manage the lifecycle of preview image streams and captured images through dynamic memory allocation and dynamic memory destruction.

[0320] Additionally, it should be noted that before implementing the collaborative working method provided in this embodiment, the educational APP is bound to the desk lamp, and the desk lamp is registered in the cloud server.

[0321] This application provides a framework scheme for Android system devices to take pictures using the camera of an external device. This scheme can be applied not only in educational scenarios, but also to other devices equipped with cameras. These devices can share their camera capabilities with Android system devices such as mobile phones and tablets to achieve interconnection and interoperability between Android system devices and these devices.

[0322] Figures 5a-5b An example application scenario is shown. For example... Figure 5a As shown in Figure (1), the tablet displays interface 401, which shows multiple application icons. The user clicks the education application icon 4011. In response to the received user operation, the tablet opens the education application and displays the education application interface, which can be referred to... Figure 5a As shown in (2). Figure 5a As shown in Figure (2), the tablet displays an educational application interface 402, which displays various functional options of the educational application, including but not limited to word lookup, point-and-read function, homework function, and photo-taking function. When the user clicks the photo-taking function option 4021, in response to the user's operation, the tablet executes the device discovery phase, the virtual camera service enable phase, and the virtual camera preview access phase.

[0323] During the device discovery phase, if the tablet's device management service filters out only one desk lamp that can be linked with the tablet, the tablet automatically executes the virtual camera service enabling phase and the virtual camera preview access phase, and displays, for example... Figure 5b The interface shown in (1) is as follows. During the device discovery phase, if the number of desk lamps that can be linked with the tablet is multiple as filtered by the tablet's device management service, the tablet displays a desk lamp selection interface. For example, a list of desk lamps to be linked is displayed on the desk lamp selection interface, and the user can perform a selection operation. In response to the user's selection operation, the educational application determines a desk lamp to be linked and continues to execute the virtual camera service enable phase and the virtual camera preview access phase to display, for example... Figure 5b The interface shown in (1).

[0324] like Figure 5b As shown in Figure (1), interface 403 displays an image preview window 4031 and a photo option 4032. The image preview window 4031 displays a preview image captured in real time by the lamp's camera. At this time, if the user clicks the photo option 4032, in response to the user's operation, the tablet executes the virtual camera photo-taking process, displaying, for example... Figure 5b The interface shown in (2). Figure 5b(2) As shown in the interface 404, the image captured by the lamp's camera is displayed in the image preview window 4041. If the user clicks the confirmation option 4041, the tablet responds to the user's operation, saves the captured image, and continues to display, for example... Figure 5b The preview interface shown in (1) is shown in the image. If the user clicks the cancel option 4042, the tablet will respond to the user's operation and may display, for example... Figure 5b The preview interface shown in (1) is shown in the middle.

[0325] It should be pointed out that, as Figure 5b (2) The interface shown is merely an example. The image captured by the lamp camera may not be displayed in the image preview window 4041, but in other areas of the interface. The image preview window 4041 continues to display the preview image captured by the lamp camera in real time. This application does not limit this.

[0326] This embodiment provides a collaborative working method, primarily illustrating a low-cost technical solution for implementing online education functions based on a combination of a tablet device and a desk lamp device. The technical solution provided in this embodiment is further explained below in conjunction with several different functions involved in online education.

[0327] Scene 1

[0328] Reference Figure 1a The illustration shows an application scenario, using a word lookup function as an example to illustrate the technical solution. When students encounter unfamiliar words, they can point their finger under the word, and the tablet's camera will take a picture. The tablet will then recognize the image to identify the word and provide feedback on its meaning via the display screen after the online word lookup is completed, such as displaying and reading out the word's explanation on the interface.

[0329] like Figure 6a The diagram shows the interaction between each module. (Refer to...) Figure 6a This application provides a method flow for a tablet and a desk lamp to work together, specifically including:

[0330] S501, in response to the user's click on the word search function, the tablet and desk lamp execute the device discovery phase, the virtual camera enable phase, and the virtual camera preview access phase, with the tablet displaying the preview interface.

[0331] The processes for the device discovery phase, the virtual camera enable phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0332] It should be noted that during the virtual camera preview access phase, the hardware virtualization API in the tablet sends a virtual camera access command to the camera service, which includes camera configuration parameters. These parameters may include, but are not limited to, image resolution and image capture frame rate. The desk lamp camera is configured according to the received parameters and then captures preview image data at the corresponding image resolution and frame rate.

[0333] In this scenario, the tablet needs to accurately recognize the preview image to determine the text content the user is pointing to. Therefore, the preview image quality is crucial; for example, the image resolution can be set to 1080P. This ensures a high success rate for word lookup.

[0334] The S502 tablet's educational app uses finger recognition to scan preview images.

[0335] For example, the educational app can perform finger recognition on each frame of the received preview image, or it can periodically perform finger recognition on the latest received preview image. This embodiment does not limit this.

[0336] For example, an educational app can integrate an image recognition algorithm to perform image recognition operations, or the educational app can call an image recognition service to perform image recognition operations. This embodiment does not limit this.

[0337] For image recognition algorithms, please refer to existing technologies; this embodiment will not elaborate further.

[0338] In response to the user pointing to words with their finger, the educational app on the tablet recognizes the user's finger in the preview image.

[0339] When a user points to a word in a book with their finger, the lamp's camera can capture a preview image of the finger pointing to the word, and then the educational app on the tablet can recognize the user's finger in the preview image.

[0340] When an educational app performs finger recognition on a preview image, if a finger is detected, it can obtain the finger's position information in the preview image, such as coordinates.

[0341] In the S504 tablet, the educational app determines the ROI (region of interest) image based on the position of the finger in the preview image.

[0342] After the educational app recognizes the user's finger in the preview image, it can determine the Region of Interest (ROI) image based on the finger's position information within the preview image. Specifically, the educational app can determine the ROI information based on the finger's coordinates within the preview image. ROI information includes, but is not limited to, the center point coordinates and the region's extent (such as width and height). Furthermore, the educational app can crop the ROI image from the preview image based on this ROI information.

[0343] The S505 tablet's educational app accurately identifies ROI images and determines the new words to be explained.

[0344] For example, an educational app can integrate an image recognition algorithm to accurately identify ROI images, or the educational app can call an image recognition service to accurately identify ROI images in order to determine the new words to be explained. This embodiment does not limit this.

[0345] For image recognition algorithms, please refer to existing technologies; this embodiment will not elaborate further.

[0346] On the S506 tablet, the educational app allows users to look up unfamiliar words and displays their definitions.

[0347] After identifying the unfamiliar word to be explained, the educational app can perform an online word lookup or a lookup in a database to obtain the definition of the word. The app can then display the definition for the user to view. In addition, the app can read the displayed definition aloud; this embodiment does not limit this aspect.

[0348] Similarly, users can also use pointing tools (or indicator tools) such as reading pens to perform word pointing operations, and this embodiment does not limit this. Accordingly, the educational app performs word pointing tool recognition on the preview image to determine whether the user intends to search for words, and determines the ROI image based on the position information of the pointing tool in the preview image.

[0349] Similarly, users can point to images in a book using their fingers or reading pens. The educational app then determines the Region of Interest (ROI) based on the location information of the finger or reading pen, performs image content recognition on the ROI, displays the corresponding definition, and can also read the displayed definition aloud. This embodiment will not elaborate further on this scenario.

[0350] Figure 1a , Figures 7a-7b An example application scenario is shown. For example... Figure 7aAs shown in Figure (1), the tablet displays an educational app interface 701, which displays various function options of the educational application, including but not limited to word lookup, point-and-read function, homework function, and photo function. When the user clicks the word lookup function option 7011, in response to the user's operation, the tablet executes the device discovery phase, the virtual camera service enable phase, and the virtual camera preview access phase.

[0351] During the device discovery phase, if the tablet's device management service filters out only one desk lamp that can be linked with the tablet, the tablet automatically executes the virtual camera service enabling phase and the virtual camera preview access phase, and displays, for example... Figure 7a The interface shown in (2) is as follows. During the device discovery phase, if the number of desk lamps that can be linked with the tablet is multiple as filtered by the tablet's device management service, the tablet displays a desk lamp selection interface. For example, a list of desk lamps to be linked is displayed on the desk lamp selection interface, and the user can perform a selection operation. In response to the user's selection operation, the educational application determines a desk lamp to be linked and continues to execute the virtual camera service enable phase and the virtual camera preview access phase to display, for example... Figure 7a The interface shown in (2) is shown in the middle.

[0352] like Figure 7a As shown in Figure (2), interface 702 displays an image preview window 7021 and a word lookup function operation diagram 7022. The image preview window 7021 displays a preview image captured in real-time by the desk lamp's camera. Users can refer to the word lookup function operation diagram 7022 to perform a pointing-to-word or pointing-to-image operation to trigger the word lookup function. The educational app performs pointing-to-word or pointing-to-image recognition based on the preview image. (Continue referring to...) Figure 7a In section (2), when a user points to a word in a book, the desk lamp camera captures a preview image of the finger pointing to the word, which is displayed in the image preview window 7021. Furthermore, the educational app can identify the user's finger in the preview image and determine the finger's position information in the preview image, such as coordinates. The educational app determines the ROI image based on the finger's position in the preview image and performs accurate recognition of the ROI image to identify the new word to be explained. After the educational app finds the definition of the new word, it displays the corresponding definition on the interface for reference. Figure 7b As shown.

[0353] However, in the above process, to ensure a high success rate for word lookup, the desk lamp's camera needs to continuously return a high-resolution (e.g., 1080p) preview image stream, which places higher demands on bandwidth, requiring 4-8 Mbps. This also increases the hardware requirements for the desk lamp, thereby raising its cost.

[0354] To achieve a collaborative word lookup function between a tablet and a desk lamp, and to reduce the hardware cost of the desk lamp, this embodiment also provides a technical solution. Since finger recognition, word pointing tool recognition, and location recognition in images do not require high image resolution, during the virtual camera preview access phase, the desk lamp's camera continuously returns a low-resolution (e.g., 480p) preview image stream. If the educational app recognizes the user's word or image pointing action based on the preview image, it triggers the desk lamp to capture a high-resolution (e.g., 1080p) image, allowing the educational app to accurately identify the word or image to be explained based on this high-resolution image. Thus, using the desk lamp's camera to continuously return a low-resolution preview image stream requires minimal bandwidth, only 0.5-1 Mbps, with higher bandwidth only needed when transmitting the high-resolution image. Therefore, this technical solution not only reduces the hardware chip requirements of the desk lamp and lowers its cost, but also ensures a high word lookup success rate.

[0355] like Figure 6b The diagram shows the interaction between each module. (Refer to...) Figure 6b This application provides a method flow for a tablet and a desk lamp to work together, specifically including:

[0356] S601, in response to the user's click on the word search function, the tablet and desk lamp execute the device discovery phase, the virtual camera enable phase, and the virtual camera preview access phase, with the tablet displaying the preview interface.

[0357] The processes for the device discovery phase, the virtual camera enable phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0358] It should be noted that during the virtual camera preview access phase, the hardware virtualization API in the tablet sends a virtual camera access command to the camera service, which includes camera configuration parameters. These parameters may include, but are not limited to, image resolution and image capture frame rate. The desk lamp camera is configured according to the received parameters and then captures preview image data at the corresponding image resolution and frame rate.

[0359] In this scenario, to reduce the bandwidth consumed by the preview image stream, the lamp camera can capture preview images at a low resolution during the virtual camera preview access phase. For example, the hardware virtualization API in the tablet sends a virtual camera access command to the camera service, which includes a first configuration parameter, such as a first image resolution (e.g., 480P). The lamp camera then configures itself according to the received configuration parameters and captures preview image data at the first image resolution and the corresponding image capture frame rate.

[0360] The S602 tablet's educational app uses finger recognition to scan preview images.

[0361] In response to the user pointing to words with their finger, the educational app on the tablet recognizes the user's finger in the preview image.

[0362] In the S604 tablet, the educational app determines the ROI information based on the position of the finger in the preview image and generates a photo request based on the ROI information.

[0363] ROI information refers to the information used to determine the ROI, which may include, but is not limited to, the coordinates of the center point and the area range (such as width and height information).

[0364] S605, the tablet transmits the photo-taking request to the desk lamp.

[0365] The photo request may include, but is not limited to, the virtual camera ID corresponding to the desk lamp camera, the second configuration parameters of the desk lamp camera, and ROI information. The second configuration parameters include, but are not limited to, a second image resolution, which is higher than the first image resolution, such as setting the second image resolution to 1080P. This allows the educational app to accurately identify and determine the vocabulary or image to be explained based on this high-resolution image.

[0366] S606, the desk lamp's camera is set according to the second configuration parameters carried in the photo request, takes an image at the second image resolution, and sends the captured image to the hardware abstraction service.

[0367] S607, the Hardware Abstraction Service determines the ROI image based on the ROI information.

[0368] The hardware abstraction service can crop out the ROI image from the captured image based on the ROI information.

[0369] In one optional implementation, the photo-taking request includes, but is not limited to, the second configuration parameters, but excludes ROI information. In this way, the tablet transmits the photo-taking request to the desk lamp, whose camera is configured according to the second configuration parameters carried in the request, captures an image at the second image resolution, and returns the captured image to the educational app on the tablet. The educational app can then determine the ROI image based on the ROI information, for example, by cropping the ROI image from the captured image.

[0370] The S608 desk lamp transmits the ROI image to the educational app on the tablet.

[0371] Compared to a desk lamp directly sending high-resolution images to an educational app on a tablet, sending cropped ROI images to the educational app on a tablet reduces data transfer and bandwidth usage.

[0372] In the S609 tablet, the educational app accurately identifies the ROI image and determines the new words to be explained.

[0373] On the S610 tablet, the educational app allows users to look up unfamiliar words and displays their definitions.

[0374] For any parts of this process that are not explained in detail, please refer to the previous text; they will not be repeated here.

[0375] Similarly, users can also use pointing tools such as reading pens to perform word-pointing operations, and this embodiment does not limit this. Accordingly, the educational app performs word-pointing tool recognition on the preview image to determine whether the user intends to search for words, and determines the ROI information based on the position information of the pointing tool in the preview image.

[0376] Similarly, users can use their fingers or reading pens to point at images in a book. Accordingly, the educational app determines the Region of Interest (ROI) based on the location information of the finger or reading pen, identifies the ROI image within the captured image, performs image content recognition on the ROI, displays the corresponding definition, and can also read the displayed definition aloud. This embodiment will not elaborate further on this scenario.

[0377] For application scenarios of this process, please refer to... Figure 1a , Figures 7a-7b The application scenarios shown are illustrated. (Refer to...) Figure 7a In step (2), when a user points to a word in a book, the lamp camera captures a preview image of the finger pointing to the word, which is displayed in the image preview window 7021. The educational app can then recognize the user's finger in the preview image, determine the ROI information, and generate a photo request based on the ROI information and the high resolution of the image, triggering the lamp camera to capture the image. The lamp camera captures the image at high resolution, and the lamp side crops the high-resolution image based on the ROI information to obtain the ROI image, returning the ROI image to the educational app on the tablet. The educational app accurately identifies the ROI image and determines the new word to be explained. After the educational app finds the definition of the new word, it displays the corresponding definition on the interface for reference. Figure 7b As shown.

[0378] Scene 2

[0379] Reference Figure 1a The illustration shows an application scenario, using the homework function as an example to illustrate the technical solution. When students need to submit homework online, they can click to take a picture in the educational app, using the desk lamp's camera to capture an image of the homework, and then upload the image to the database through the educational app.

[0380] like Figure 8 The diagram shows the interaction between each module. (Refer to...) Figure 8 This application provides a method flow for a tablet and a desk lamp to work together, specifically including:

[0381] In response to the user clicking the homework function, the education app on the tablet displays a list of submitted homework.

[0382] It should be noted that the assignment submission list refers to a list containing multiple assignment submission options. Each assignment submission option corresponds to one assignment. (See reference...) Figure 9a The interface is shown in section 704. The assignment options can be categorized by subject or by time; this embodiment does not limit this.

[0383] If an educational app's assignment function only requires submitting an image for one assignment, the app will not display an assignment submission list. In this case, in response to the user clicking the assignment function, the tablet and desk lamp execute the device discovery phase, the virtual camera enable phase, and the virtual camera preview access phase, with the tablet displaying the preview interface.

[0384] In response to the user clicking the "Submit Job" option, the tablet and desk lamp execute the device discovery phase, the virtual camera enable phase, and the virtual camera preview access phase, with the tablet displaying the preview interface.

[0385] The processes for the device discovery phase, the virtual camera enable phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0386] It should be noted that during the virtual camera preview access phase, the hardware virtualization API in the tablet sends a virtual camera access command to the camera service, which includes camera configuration parameters. These parameters may include, but are not limited to, image resolution and image capture frame rate. The desk lamp camera is configured according to the received parameters and then captures preview image data at the corresponding image resolution and frame rate.

[0387] In this scenario, to reduce the bandwidth consumed by the preview image stream, the lamp camera can capture preview images at a low resolution during the virtual camera preview access phase. For example, the hardware virtualization API in the tablet sends a virtual camera access command to the camera service, which includes a first configuration parameter, such as a first image resolution (e.g., 480P). The lamp camera then configures itself according to the received configuration parameters and captures preview image data at the first image resolution and the corresponding image capture frame rate.

[0388] In response to the user clicking the camera option, the educational app on the tablet generates a photo request.

[0389] When a user places homework or books within the camera's capture area, the user can click the photo option to trigger the camera to capture an image of the homework.

[0390] The photo request may include, but is not limited to, the virtual camera ID corresponding to the desk lamp camera and the desk lamp camera's second configuration parameters. The second configuration parameters include, but are not limited to, a second image resolution, which is higher than the first image resolution, such as setting the second image resolution to 1080P. This allows the educational app to upload high-resolution images of assignments.

[0391] S804, the tablet transmits the photo-taking request to the desk lamp.

[0392] S805, the desk lamp's camera is set according to the second configuration parameters carried in the photo capture request, and the image is captured at the second image resolution.

[0393] The S806 desk lamp transmits captured images to the educational app on the tablet.

[0394] The S807 tablet displays captured images via an educational app.

[0395] The educational app on the tablet receives and displays the homework image captured by the desk lamp's camera. If the user is satisfied with the image, they can click the submit option to upload it to the database. If the user is not satisfied, they can click the photo option again to trigger the desk lamp's camera to retake the picture.

[0396] In response to the user clicking the submit option, the educational app on the S808 uploads captured images to the database.

[0397] For any parts of this process that are not explained in detail, please refer to the previous text; they will not be repeated here.

[0398] It should be noted that the homework images mentioned above are merely illustrative examples. Users can click the photo option to trigger the desk lamp's camera to capture other images. After the desk lamp sends the captured images to the educational app on the tablet, the educational app can then upload the received images to the corresponding database.

[0399] Figure 1a , 9a - Figure 9c An example application scenario is shown. For example... Figure 9aAs shown in Figure (1), the tablet displays the educational app interface 701, which shows various function options of the educational application, including but not limited to word lookup, point-and-read function, homework function, and photo function. When the user clicks the word lookup function option 7012, the educational app on the tablet displays the homework submission list in response to the user's operation. (See Figure 1 for reference.) Figure 9a As shown in (2). Figure 9a The assignment submission list interface 704 shown in (2) displays multiple assignment submission options (such as Submit Assignment 1, Submit Assignment 2, Submit Assignment 3, Submit Assignment 4, etc.), with different assignment submission options corresponding to different assignments. Taking the user needing to upload an image for Assignment 4 as an example, the user clicks the Submit Assignment 4 option 7042. In response to the user's operation, the tablet can display the following: Figure 9b The job submission interface 705 shown in (1) is shown in the middle.

[0400] Continuing with reference to 9b(1), the homework submission interface 705 displays an image preview window 7041, a photo option 7051, and a submission option 7052. The image preview window 7041 displays a preview image captured in real-time by the desk lamp camera. When the user places homework or books within the desk lamp camera's capture area, the user can click the photo option 7051 to trigger the desk lamp camera to capture the homework image. In response to the user's operation, the education app generates a photo request and sends it to the desk lamp to invoke the desk lamp camera to capture the homework image. The desk lamp camera captures the homework image at high resolution according to the image carried in the photo request and returns the captured homework image to the education app on the tablet for display. See also: Figure 9b Interface 706 is shown in (2).

[0401] Continuing with reference to 9b (2), the interface 706 displays an image preview window 7041, a photo option 7051, a submit option 7052, and a work image 7061 captured by the desk lamp camera. The work image 7061 also displays a close option 7062. If the user is not satisfied with the work image 7061, they can click the close option 7062, and the work image 7061 will no longer be displayed on the interface. At this time, the user can click the photo option 7051 to trigger the desk lamp camera to retake the work image. If the user is satisfied with the work image 7061, they can click the submit option 7052. In response to the user's operation, the tablet can display the following... Figure 9cThe pending confirmation interface 701 is shown. This interface displays the assignment image 7061 to be submitted, and a confirmation / submission window 7071. If the user clicks the confirmation option 7072 in the confirmation / submission window 7071, the educational app responds to the user's action by uploading the assignment image 7061 to the database. If the user clicks the cancel option 7071 in the confirmation / submission window 7071, the educational app responds to the user's action by displaying... Figure 9b The interface shown in (1) is for the user to click the photo option 7051 to trigger the desk lamp camera to retake the work image.

[0402] It should be noted that, continuing to refer to 9b(2), if the user is satisfied with the assignment image 7061, they can click the submit option 7052. In response to the user's action, the educational app may no longer display the following: Figure 9c The interface shown directly uploads the work image 7061 to the database. This embodiment does not limit this process.

[0403] Scene 3

[0404] Reference Figure 1a The illustration shows an application scenario, using the point-and-read function (or finger-read function) as an example to illustrate the technical solution. When a student needs the educational app to read content from a book aloud, the lamp's camera can capture a real-time image of the book. The educational app can then load the corresponding book content based on the image and determine the content to be read aloud based on the student's finger position or page-turning action.

[0405] like Figure 10 The diagram shows the interaction between each module. (Refer to...) Figure 10 This application provides a method flow for a tablet and a desk lamp to work together, specifically including:

[0406] S901, in response to the user's click-to-read function operation, the tablet and desk lamp execute the process of device discovery phase, virtual camera enable phase and virtual camera preview access phase, and the tablet displays the preview interface.

[0407] The processes for the device discovery phase, the virtual camera enable phase, and the virtual camera preview access phase can be found in the previous text and will not be repeated here.

[0408] It should be noted that during the virtual camera preview access phase, the hardware virtualization API in the tablet sends a virtual camera access command to the camera service, which includes camera configuration parameters. These parameters may include, but are not limited to, image resolution and image capture frame rate. The desk lamp camera is configured according to the received parameters and then captures preview image data at the corresponding image resolution and frame rate.

[0409] In this scenario, to reduce the bandwidth consumed by the preview image stream, the lamp camera can capture preview images at a low resolution during the virtual camera preview access phase. For example, the hardware virtualization API in the tablet sends a virtual camera access command to the camera service, which includes a first configuration parameter, such as a first image resolution (e.g., 480P). The lamp camera then configures itself according to the received configuration parameters and captures preview image data at the first image resolution and the corresponding image capture frame rate.

[0410] In the S902 tablet, the educational app recognizes the preview image and determines the book title.

[0411] For example, the educational app can identify book information for each frame of the received preview image, or it can periodically identify book information for the latest received preview image. This embodiment does not limit this.

[0412] For example, an educational app can integrate an image recognition algorithm to perform image recognition operations, or the educational app can call an image recognition service to perform image recognition operations. This embodiment does not limit this.

[0413] For image recognition algorithms, please refer to existing technologies; this embodiment will not elaborate further.

[0414] The S903 tablet's educational app searches the database based on book titles and loads the corresponding book content.

[0415] If an educational app on a tablet searches its database based on book titles to identify different versions of a book, it can display a list of those books for the user to choose from. Then, in response to the user's selection of a particular version, the educational app loads the content corresponding to that version of the book.

[0416] In response to user page turning or finger tapping, the educational app on the S904 recognizes the preview image, identifies the paragraph to be read aloud, and reads the corresponding paragraph aloud.

[0417] After loading the book content, the educational app can recognize the pointing tool (finger or reading pen) for each frame of the received preview image. It can also periodically recognize the pointing tool for the latest received preview image; this embodiment does not limit this. In response to the user's pointing operation, the educational app can recognize the book page number and the user's click location information, such as coordinates, based on the preview image. Then, based on the book page number and the user's click location information, it can determine the paragraph to be read aloud from the loaded book content and read the corresponding paragraph aloud.

[0418] After loading the book content, the educational app can also perform page-turning recognition based on the preview image stream. In response to the user's page-turning operation, the educational app can identify the paragraph to be read aloud from the loaded book content based on the identified page number, and then read the corresponding paragraph aloud.

[0419] For any parts of this process that are not explained in detail, please refer to the previous text; they will not be repeated here.

[0420] Figure 1a , Figures 11a-11b An example application scenario is shown. For example... Figure 11a As shown in Figure (1), the tablet displays an educational app interface 701, which displays various function options of the educational application, including but not limited to word lookup, point-and-read function, homework function, and photo function. When the user clicks the point-and-read function option 7013, in response to the user's operation, the tablet executes the device discovery phase, the virtual camera service enable phase, and the virtual camera preview access phase.

[0421] During the device discovery phase, if the tablet's device management service filters out only one desk lamp that can be linked with the tablet, the tablet automatically executes the virtual camera service enabling phase and the virtual camera preview access phase, and displays, for example... Figure 11a The interface shown in (2) is as follows. During the device discovery phase, if the number of desk lamps that can be linked with the tablet is multiple as filtered by the tablet's device management service, the tablet displays a desk lamp selection interface. For example, a list of desk lamps to be linked is displayed on the desk lamp selection interface, and the user can perform a selection operation. In response to the user's selection operation, the educational application determines a desk lamp to be linked and continues to execute the virtual camera service enable phase and the virtual camera preview access phase to display, for example... Figure 11a The interface shown in (2) is shown in the middle.

[0422] like Figure 11a As shown in Figure (2), an image preview window 7081 is displayed in interface 708. The image preview window 7081 displays a preview image captured in real-time by the desk lamp camera. The educational app identifies the preview image to determine the book title. After identifying the book title, the educational app searches the database based on the book title. If the corresponding book is found, the book content loading operation is performed, which can be referred to... Figure 11b As shown in (1).

[0423] Continue to refer to Figure 11bAs shown in Figure (1), interface 709 displays an image preview window 7081, the identified book name 7091, and a book content loading progress indicator 7092. After the book content is loaded, the educational app can recognize the user's point-and-read or page-turning operations based on the preview image. Taking the user's page-turning operation as an example, refer to Figure (2). Figure 11b In the interface 710 shown in (2), the user's page-turning action can be displayed in the image preview window 7081. In response to the user's operation, the educational app recognizes the book page number based on the preview image. Then, the educational app can determine the paragraph to be read aloud in the loaded book content based on the recognized book page number, and read the corresponding paragraph aloud.

[0424] The collaborative working method provided in this application embodiment enables a professional online education experience that can be achieved by combining a home tablet and a desk lamp with a camera. The desk lamp's camera, in conjunction with the tablet's educational app, facilitates scenarios requiring photo capture, such as students using their fingers to look up words, submitting assignments, and pointing to books for reading.

[0425] In the aforementioned photography application scenario, after the transmission channel module of the hardware virtualization service in the tablet receives the captured image from the desk lamp camera, it transmits the captured image to the education APP through the device management module of the hardware virtualization service.

[0426] Continue to refer to Figure 12 In this tablet, the educational app and hardware virtualization service are implemented in separate processes. For example, the educational app runs in the first process (which can be called the third-party application layer), while the hardware virtualization service runs in the second process. The second process can be further divided into a device management layer implemented in Java and a native transmission channel layer implemented in C++. Specifically, the device management module of the aforementioned hardware virtualization service runs in the Java-based device management layer, while the transmission channel module runs in the C++-based native transmission channel layer. The first and second processes communicate via AIDL (Android Interface Definition Language), while the device management module and transmission channel module communicate via JNI (Java Native Interface) within the second process. The native transmission channel layer receives data, and the device management layer connects the third-party application layer and the native transmission channel layer.

[0427] In the aforementioned photo-taking application scenario, the user requests image data at the third-party application layer. The photo-taking request is transmitted to the native transmission channel layer through the device management layer, which then interacts with the lamp. After receiving the data from the lamp, the native transmission channel layer transmits it to the device management layer, which then transmits the image data to the third-party application layer process for display to the user.

[0428] Thus, when the image data (or image files) captured by the desk lamp's camera is transmitted to the educational app, it crosses three layers: the native transmission channel layer, the device management layer, and the third-party application layer. Whether from the native transmission channel layer to the device management layer or from the device management layer to the third-party application layer, the data transmission inevitably involves copying.

[0429] Because processes in the Android system cannot share memory, mechanisms are needed to facilitate data communication between different processes. In traditional communication mechanisms, cross-process data transfer requires four copy operations, data transfer between the native transport channel layer and the device management layer via JNI communication requires one copy operation, and data transfer from the lamp side to the native transport channel layer also requires one copy operation. Thus, data transfer from the transport channel layer and device management layer to the third-party application layer requires six copy operations.

[0430] Reference Figure 13 The six data copy operations involved in transferring data (or files) from the transmission channel layer and device management layer to the third-party application layer include:

[0431] 1. The hardware virtualization service's transmission channel module copies the received captured image data into the transmission channel buffer.

[0432] 2. The hardware virtualization service's transmission channel module copies the captured image data from the transmission channel cache to the device management cache.

[0433] The two copy operations mentioned above were performed in the user space of the second process.

[0434] 3. The hardware virtualization service copies the captured image data from the device management cache to the kernel space of the second process.

[0435] 4. The hardware virtualization service copies the captured image data from the kernel space of the second process into memory.

[0436] 5. The educational app copies the captured image data from memory to the kernel space of the first process.

[0437] 6. The educational app copies the captured image data from the kernel space of the first process to the user space of the first process.

[0438] In this way, the educational app can display the captured image data.

[0439] In the scenario of collaborative operation of a flat panel lamp provided in this application embodiment, the captured image resolution is generally high (e.g., 1080P), resulting in a relatively large file size. However, when large image files need to be transferred across layers, the more times data is copied, the lower the transfer rate and the longer the time required. In traditional solutions, large image files require six copy operations to be transferred from the transmission channel layer and device management layer to the third-party application layer, which undoubtedly reduces the transfer speed of large image files and prolongs the time consumption. Moreover, since the captured image data is a shared resource operated by multiple processes, the process is prone to memory errors, making memory management more complex.

[0440] Therefore, improving the transmission rate of large image files across layers and reducing communication time are technical problems that need to be solved.

[0441] This application also provides a data transmission method that improves the transmission rate of large image files across layers by establishing a shared memory mechanism, reduces communication time, and solves the problem of excessive copying and long transmission time in traditional solutions.

[0442] In the data transmission method provided in this application embodiment, large image files only require two copy operations to be transferred from the transmission channel layer and the device management layer to the third-party application layer, thereby improving the cross-layer transmission rate of large image files and reducing communication time. Figure 14 As shown, the two data copy operations involved in the data transfer from the transmission channel layer and the device management layer to the third-party application layer include:

[0443] 1. The hardware virtualization service's transmission channel module copies the received captured image data into shared memory.

[0444] 2. The educational app copies the captured image data from shared memory to the user space of the first process.

[0445] like Figure 15 The diagram shows the interaction between each module. (Refer to...) Figure 15 The data transmission method provided in this application embodiment specifically includes the following steps:

[0446] S1001, the tablet education app sends a photo-taking request to the device management module in the hardware virtualization service.

[0447] For example, in a homework submission scenario, in response to the user clicking the photo option, the tablet education app sends a photo request to the device management module in the hardware virtualization service.

[0448] For example, in a word lookup scenario, in response to the user pointing to a word with their finger, the educational app on the tablet recognizes the user's finger in the preview image, determines the ROI information based on the position of the finger in the preview image, and generates a photo request to send to the device management module in the hardware virtualization service.

[0449] In this embodiment, the photo-taking request may include, but is not limited to, a task identifier (such as PictureNo), a callback function, and shooting parameters (CamPara). Shooting parameters include, but are not limited to, shooting modes, such as continuous shooting mode (also known as burst mode) and normal shooting mode.

[0450] Since each photo request involves different image data, a task identifier can be used as a unique identifier for the photo request.

[0451] S1002, the device management module of the hardware virtualization service in the tablet stores the task identifier and callback function carried in the photo-taking request into the callback cache area, and sends the photo-taking request to the transmission channel module of the hardware virtualization service.

[0452] After receiving a photo-taking request, the device management module parses the task identifier and callback function written in the request. Using the task identifier as an identifier, it stores the corresponding callback function in the callback buffer. The callback function information stored in the callback buffer can be in key-value pair format, where the key is the task identifier and the value is the callback function.

[0453] Since the process of requesting to take a picture and transmitting the captured image data is asynchronous, it involves the transmission channel layer, the device management layer, and the third-party application layer on the tablet side. Therefore, after the device management module receives the shooting data from the desk lamp, it can look up the matching callback function in the callback buffer through the task identifier and pass the captured image data to the third-party application layer through the callback function.

[0454] S1003, the hardware virtualization service transmission channel module in the tablet transmits the photo-taking request to the desk lamp side in the control channel.

[0455] S1004, the desk lamp calls the camera to capture an image based on the photo request, and returns the captured image data to the transmission channel module of the hardware virtualization service in the tablet in the data channel.

[0456] The captured image data may include, but is not limited to, task identifiers, image frame sizes, shooting parameters, and image data.

[0457] For example, captured image data can be transmitted back to the hardware virtualization service transmission channel module in the tablet in the form of data packets. The data packets include, but are not limited to, task identifiers, image frame sizes, shooting parameters, and image data.

[0458] It should be noted that for the image data returned by a certain photo request, the task identifier and shooting parameters carried in the corresponding data packet are the same as those carried in the photo request, so that the image data returned by the desk lamp can be accurately transmitted to the photo request sent by the educational APP.

[0459] S1005, the transmission channel module of the hardware virtualization service in the tablet obtains the task identifier, image frame size, and shooting parameters corresponding to the captured image data, and sends the task identifier, image frame size, and shooting parameters to the device management module.

[0460] S1006, the device management module of the hardware virtualization service in the tablet creates shared memory based on the image frame size and task identifier, and stores the task identifier and shared memory address in the shared memory pool corresponding to the shooting parameters.

[0461] In this embodiment, a shared memory segment (which can be identified by ShareM) is created for each photo-taking request for inter-process communication. The size of the shared memory address is determined according to the image frame size, and the identifier of the shared memory address can be determined by the unique identifier (i.e., task identifier) ​​of the photo-taking request passed by the third-party application layer.

[0462] After receiving the captured image from the desk lamp, the transmission channel module uses JNI to pass the image frame size (frameSize) and task identifier (PictureNo) of the captured image to the device management module. The device management module creates a shared memory location with PictureNo as the identifier and frameSize as the size, and stores this shared memory address in the shared memory pool. PictureNo serves as the unique identifier for the shared memory address.

[0463] Since tablets have limited hardware space and shared memory cannot be created indefinitely, a shared memory pool management module can be set up in the device management layer implemented in JAVA to manage the shared memory pool, so as to ensure that third-party applications can obtain image data in a timely manner and clean up unnecessary shared memory in real time according to the number of requests.

[0464] For example, the shared memory pool management module can be integrated into the device management module, or it can be an independent module in the device management layer. This application does not limit this.

[0465] For example, the device management module can call the shared memory pool management module to complete the operation of storing shared memory addresses in the shared memory pool.

[0466] As an optional implementation, after the device management module creates shared memory based on the image frame size and task identifier, it sends the task identifier and shared memory address to the shared memory pool management module, which then stores the task identifier and shared memory address in the shared memory pool.

[0467] For example, in a shared memory pool, task identifiers and shared memory addresses exist as key-value pairs, where the key is the task identifier and the value is the shared memory address.

[0468] For example, the shared memory pool can be created during the virtual camera enabling phase. For instance, the device management module in the application virtualization service creates a shared memory pool when it receives a virtual camera enabling request from an educational app, or the device management module in the application virtualization service creates a shared memory pool when it confirms that the virtual camera is successfully enabled.

[0469] As another example, the shared memory pool can also be created by the device management module in the application virtualization service during the virtual camera preview access phase.

[0470] As another example, the shared memory pool can also be created by the shared memory pool management module in the device management layer during the virtual camera enable phase or the virtual camera preview access phase.

[0471] Regarding the timing of the shared memory pool creation, this embodiment does not impose specific limitations; it only needs to be created before the user's first initiation and before the educational app's first automatic photo-taking request. Regarding the entity responsible for creating the shared memory pool, this embodiment also does not impose specific limitations; relevant modules in the device management layer implemented in Java can complete the shared memory pool creation operation.

[0472] The following explanation uses the shared memory pool management module to create and manage a shared memory pool as an example.

[0473] To prevent the unlimited expansion of shared memory, the shared memory pool management module can limit the number of shared memory addresses maintained by the shared memory pool when creating it. For example, assuming the threshold for the number of shared memory addresses maintained by the shared memory pool is n+1, when the shared memory pool stores n+1 shared memory addresses, whenever a new task identifier and its corresponding shared memory address are stored in the shared memory pool, the earliest task identifier and shared memory address stored in the shared memory pool are removed, and the corresponding shared memory is released.

[0474] like Figure 16As shown, assume the threshold for the number of shared memory segments maintained by the shared memory pool is n+1. After the device management module creates a new shared memory segment based on the image frame size and task identifier, and sends the shared memory information (PictureNo x, ShareM x) to the shared memory pool management module, the shared memory pool management module performs the operation of storing (PictureNo x, ShareM x) into the shared memory pool.

[0475] For example, after receiving shared memory information, the shared memory pool management module first determines the number of shared memory information currently stored in the shared memory pool. If the number of shared memory information currently stored does not reach the shared memory maintenance threshold, the shared memory pool management module directly stores the shared memory information (PictureNo x, ShareM x) into the shared memory pool.

[0476] When the amount of shared memory information currently stored reaches the shared memory maintenance threshold, such as Figure 16 As shown, the shared memory pool stores n+1 shared memory address information (e.g., (PictureNo 0, ShareM 0) to (PictureNo n, ShareM n)). The shared memory pool management module first removes the earliest shared memory address information from the shared memory pool and releases the shared memory corresponding to that address. For example, when storing shared memory information (PictureNo x, ShareM x) into the shared memory pool, shared memory information (PictureNo 0, ShareM 0) is removed from the shared memory pool, and the shared memory corresponding to ShareM0 is released.

[0477] As an optional implementation, to ensure compatibility with different shooting modes, such as burst mode and normal shooting mode, the shared memory pool management module can create a shared memory pool corresponding to each shooting module. The threshold number of shared memory pools maintained by each shared memory pool is different. For example, the threshold number of shared memory pools maintained for burst mode is n1, and the threshold number of shared memory pools maintained for normal shooting mode is n2, where n1 is greater than n2.

[0478] For example, after the device management module creates shared memory based on the image frame size and task identifier, it sends the task identifier, shared memory address, and shooting parameters (including but not limited to shooting mode) to the shared memory pool management module. The shared memory pool management module stores the task identifier and shared memory address in a shared memory pool that matches the shooting mode.

[0479] For example, in this embodiment, n1 can be set to 15 and n2 can be set to 5.

[0480] like Figure 17 As shown, assuming the first shared memory pool is the one corresponding to the normal shooting mode, and the maintenance threshold for the number of shared memory pools is 5, and the first shared memory pool is the one corresponding to the continuous shooting mode, and the maintenance threshold for the number of shared memory pools is 15. After the device management module creates a new shared memory segment based on the image frame size and task identifier, and sends the shared memory information (PictureNo x, ShareM x) and shooting parameter information to the shared memory pool management module, the shared memory pool management module performs the operation of storing (PictureNo x, ShareM x) into the matching shared memory pool according to the shooting parameters.

[0481] When the shooting parameters indicate normal shooting mode, the shared memory pool management module performs the operation of storing (PictureNo x, ShareM x) into the first shared memory pool. For example, the shared memory pool management module first determines the number of shared memory information currently stored in the first shared memory pool. If the number of shared memory information currently stored in the first shared memory pool does not reach the shared memory maintenance threshold of 5, the shared memory pool management module directly stores the shared memory information (PictureNo x, ShareM x) into the first shared memory pool. If the number of shared memory information currently stored in the first shared memory pool reaches the shared memory maintenance threshold of 5, such as... Figure 17 As shown, the shared memory pool management module first removes the earliest shared memory address from the first shared memory pool out of the five shared memory address information, and then releases the shared memory corresponding to that shared memory address. For example, when storing shared memory information (PictureNo x, ShareM x) into the first shared memory pool, the shared memory information (PictureNo 0, ShareM 0) is removed from the first shared memory pool, and the shared memory corresponding to ShareM 0 is released.

[0482] When the shooting parameters indicate continuous shooting mode, the shared memory pool management module performs the operation of storing (PictureNo x, ShareMx) into the second shared memory pool. For example, the shared memory pool management module first determines the amount of shared memory information currently stored in the second shared memory pool. If the amount of shared memory information currently stored in the second shared memory pool has not reached the shared memory maintenance threshold of 15, the shared memory pool management module directly stores the shared memory information (PictureNo x, ShareMx) into the second shared memory pool. If the amount of shared memory information currently stored in the second shared memory pool reaches the shared memory maintenance threshold of 15, such as... Figure 17 As shown, the shared memory pool management module first removes the earliest shared memory address from the second shared memory pool out of the five shared memory address information, and then releases the shared memory corresponding to that shared memory address. For example, when storing shared memory information (PictureNo x, ShareM x) into the second shared memory pool, the shared memory information (PictureNo 0, ShareM 0) is removed from the second shared memory pool, and the shared memory corresponding to ShareM 0 is released.

[0483] In this way, the shared memory pool management module can maintain the size and lifecycle of the shared memory.

[0484] S1007, the device management module of the hardware virtualization service in the tablet returns the shared memory address corresponding to the task identifier to the transmission channel module through the file descriptor.

[0485] The device management module creates a shared memory with PictureNo as the identifier and frameSize as the size, and returns the file descriptor to the transmission channel module. At the same time, it stores the shared memory address in the shared memory pool, with PictureNo as the unique identifier.

[0486] S1008, the transmission channel module of the hardware virtualization service in the tablet fills the captured image data according to the shared memory address.

[0487] The transmission channel module obtains the shared memory address through the file descriptor and fills in the captured image data according to the shared memory address.

[0488] S1009, the hardware virtualization service transmission channel module in the tablet returns the task identifier, shooting parameters, and shared memory filling results to the device management module.

[0489] The shared memory filling result may include, but is not limited to, an indicator of whether the filling was successful.

[0490] S1010, when the shared memory filling result indicates successful filling, the device management module of the hardware virtualization service in the tablet queries the shared memory address corresponding to the task identifier in the matching shared memory pool based on the task identifier and shooting parameters.

[0491] When there is only one shared memory pool, the device management module can query the shared memory address corresponding to the task identifier in the shared memory pool based on the task identifier.

[0492] When there are multiple shared memory pools, and each shared memory pool is associated with the shooting parameters, the device management module can query the shared memory address corresponding to the task identifier in the shared memory pool that matches the shooting parameters. For example, when the shooting parameters indicate normal shooting mode, the device management module can query the shared memory address corresponding to the task identifier in the first shared memory pool; when the shooting parameters indicate continuous shooting mode, the device management module can query the shared memory address corresponding to the task identifier in the second shared memory pool.

[0493] S1011, when the shared memory filling result indicates successful filling, the device management module of the hardware virtualization service in the tablet queries the callback function corresponding to the task identifier in the callback cache based on the task identifier.

[0494] This embodiment does not limit the execution order of S1010 and S1011.

[0495] S1012, the device management module of the hardware virtualization service in the tablet returns the shared memory address to the education APP through a callback function.

[0496] After the device management module obtains the shared memory address from the matching shared memory pool using PictureNo as the key, and obtains the callback function from the callback cache using PictureNo as the key, it can return the shared memory address to the education APP through the callback function.

[0497] In the S1013 tablet, the educational app reads the captured image data according to the shared memory address and copies it to the user space of the process it is running.

[0498] At this point, in response to the photo request initiated by the educational app, the desk lamp retrieves the image data captured by its camera and returns it to the educational app. The educational app can then display the corresponding image to the user or perform image recognition based on the image to determine the text to be interpreted.

[0499] The creation of shared memory, the filling of shared memory with data, and the retrieval of data from shared memory are all asynchronous processes across processes. Therefore, maintaining a shared memory pool and managing its lifecycle not only ensures that third-party applications can accurately obtain data but also prevents the shared memory from expanding indefinitely.

[0500] For any parts of this process that are not explained in detail, please refer to the previous text; they will not be repeated here.

[0501] Thus, in the data transmission method provided in the embodiment, file cross-process transmission only requires two data copies, namely from the user space of the second process to memory, and then from memory to the user space of the first process, which improves the speed of file cross-process transmission and reduces communication time.

[0502] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the collaborative working method or data transmission method in the above embodiment.

[0503] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the collaborative work method or data transmission method in the above embodiments.

[0504] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the cooperative working method or data transmission method in the above-described method embodiments.

[0505] In this embodiment, the electronic devices (such as tablets, mobile phones, IoT devices, etc.), computer storage media, computer program products or chips are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0506] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0507] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0508] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data transmission method, characterized in that, The invention is applied in an electronic device, which includes a first process and a second process. The first process is used to run a target application, and the second process is used to transmit target data to the first process. The second process includes a first module implemented in Java and a second module implemented in C++. The first module is used to take over the first process and the second module, and the second module is used to receive the target data. The first process and the second process communicate with each other via AIDL, and the first module and the second module communicate with each other via JNI. The method includes: The second module obtains the frame size of the target data and sends the frame size to the first module; The first module creates shared memory based on the frame size and sends the shared memory address to the second module; The second module fills the target data according to the shared memory address and sends the filling result to the first module; When the filling result indicates that the filling was successful, the first module sends the shared memory address to the target application; The target application obtains the target data based on the shared memory address.

2. The method according to claim 1, characterized in that, The electronic device establishes a communication connection with the IoT device, and the target application is bound to the IoT device; Before the second module obtains the frame size of the target data, the following is also included: The target application sends a data request to the IoT device through the second process; The second module receives the target data fed back by the IoT device based on the data request.

3. The method according to claim 2, characterized in that, The data request includes a task identifier, and the target data carries the task identifier; After the first module creates shared memory based on the frame size, the method further includes: The first module stores the task identifier and the shared memory address in the shared memory pool; The second module sends the filling result to the first module, including: The second module sends the filling result and the task identifier to the first module; Before the first module sends the shared memory address to the target application, the method further includes: The first module queries the shared memory address in the shared memory pool based on the task identifier.

4. The method according to claim 3, characterized in that, The first module stores the task identifier and the shared memory address in a shared memory pool, including: When the number of shared memory addresses stored in the shared memory pool is less than the shared memory maintenance threshold, the first module stores the task identifier and the shared memory address into the shared memory pool; the shared memory maintenance threshold is used to indicate the total number of memory addresses that the shared memory pool can maintain. When the number of shared memory addresses stored in the shared memory pool is greater than or equal to the shared memory maintenance threshold, the first module removes the target shared memory address from the shared memory pool and stores the task identifier and the shared memory address into the shared memory pool accordingly; wherein, among the multiple shared memory addresses currently stored in the shared memory pool, the target shared memory address was stored into the shared memory pool earliest.

5. The method according to claim 3, characterized in that, The data request also includes data parameters, and the target data carries the data parameters. The first module stores the task identifier and the shared memory address in a shared memory pool, including: The first module stores the task identifier and the shared memory address in a shared memory pool that matches the data parameter values.

6. The method according to claim 5, characterized in that, The threshold for the number of shared memory pools that are matched with different data parameter values ​​is different.

7. The method according to claim 1, characterized in that, The frame size of the target data is greater than a preset threshold.

8. The method according to any one of claims 2-6, characterized in that, The IoT device is a desk lamp; the desk lamp is equipped with a camera; the second process is used to run hardware virtualization services, the first module is a device management module, the second module is a transmission channel module; the data request includes a photo request, and the target data includes captured image data; The method further includes: The hardware virtualization service registers a virtual camera corresponding to the camera in the system. The target application generates a photo-taking request when it invokes the virtual camera.

9. The method according to claim 8, characterized in that, The data request also includes data parameters, which are shooting modes, including normal shooting mode and continuous shooting mode; wherein... The threshold for the number of shared memory maintained in the first shared memory pool that matches the normal shooting mode is less than the threshold for the number of shared memory maintained in the second shared memory pool that matches the burst shooting mode.

10. The method according to claim 8, characterized in that, The electronic devices include mobile phones and tablet computers.

11. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is coupled to the processor; The memory stores program instructions that, when executed by the processor, cause the electronic device to perform the data transmission method according to any one of claims 1-10.

12. A collaborative work system, characterized in that, include: An electronic device for performing the data transmission method as described in any one of claims 1-10, and an Internet of Things (IoT) device, wherein the IoT device is equipped with a camera for acquiring image data, and a target application in the electronic device is bound to the IoT device; The electronic device is used to: register a virtual camera corresponding to the camera in the system, and send an image preview request to the Internet of Things device by calling the virtual camera; The IoT device is used to: invoke the camera to collect preview image data according to the image preview request of the electronic device, and send the preview image data to the target application of the electronic device for preview display; The electronic device is also used to: generate a photo-taking request when the virtual camera is invoked, and send the photo-taking request to the Internet of Things device; The IoT device is also used to: call the camera to capture an image according to the electronic device's photo capture request, and send the captured image data to the target application of the electronic device for display.

13. The system according to claim 12, wherein the IoT device is a desk lamp, and the camera is used to collect image data downwards.

14. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, the electronic device performs the data transmission method according to any one of claims 1-10.

Images