Human intelligence-based image processing method, edge device and storage medium

Abstract

The invention provides an image processing method based on human intelligence, an edge device and a storage medium, and the method comprises the steps: receiving image data of a collected image through a first language process, storing the image data in a local shared memory, and storing the image data in the form of image matrix elements; sending a memory address of the image matrix element in the local shared memory and preset image description information corresponding to the acquired image to a second language process through a cross-language bridging tool; the second language process reads image matrix elements from the local shared memory according to the memory address; according to preset image description information, the image matrix elements are reconstructed, a reconstructed image consistent with the collected image is obtained through reconstruction, image processing is carried out, and a processing result is returned to the first language process through a cross-language bridging tool; and the first language process sends the image processing result to the host device. According to the invention, the data transmission delay and the memory resource pressure can be reduced while the system response speed is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, and in particular to an image processing method based on human factor intelligence, an edge device and a storage medium. BACKGROUND

[0002] In the physiological state assessment using physiological signals, image recognition technology is often combined to improve the assessment accuracy, such as identifying the facial expressions of the test personnel to assess concentration, or detecting the closed-eye situation to judge fatigue. For this purpose, the existing system usually introduces a computer vision-based function module in the physiological device, such as an AI function-based facial expression recognition module, which can deeply analyze the facial expressions of the personnel to assist in physiological state assessment.

[0003] At the system architecture level, in order to save computing power, a "host + edge device" collaborative mode can be used to deploy feature extraction, real-time analysis and other computing tasks on the edge device end, and run on the basis of retaining the original system interface and interaction logic. Since the edge device needs to undertake image processing and data analysis tasks, it usually needs to contain an efficient application architecture constructed by multiple programming languages in its internal, such as a C# process responsible for system control and a Python process executing image AI recognition. In the face of such a multi-language collaborative architecture, how to realize efficient and low-delay transmission of image data between different language processes becomes a key problem affecting system performance.

[0004] At present, the traditional cross-process image processing method includes: based on network communication (such as TCP / IP protocol), file exchange or inter-process communication (IPC) mechanism, serializing image data into byte stream, transmitting through Socket, writing into shared file or exchanging by using named pipe, message queue and other IPC methods.

[0005] However, in the traditional cross-process image data processing process, image data usually needs to be copied multiple times, especially when facing high-resolution or high-frame-rate images, repeated memory copy operations will significantly consume CPU and memory resources, thereby reducing the overall throughput of the system. In addition, the data exchange method based on network communication will also introduce additional protocol processing, I / O waiting and context switching overhead, further aggravating resource occupation and transmission delay, resulting in the problem of large resource occupation and high transmission delay in cross-process image data processing. SUMMARY

[0006] In view of this, the embodiments of the present application provide an image processing method based on human factor intelligence, an edge device and a storage medium to eliminate or improve one or more defects in the prior art. The problem of large resource occupation and high transmission delay in the traditional cross-process image processing method can be solved.

[0007] An aspect of the present application provides a human factor intelligent-based image processing method, applicable to an edge device, the edge device being pre-configured with a first language process, a second language process and a cross-language bridging tool; the method comprises: receiving, by the first language process, image data of a collected image, and storing the image data in a local shared memory in the form of image matrix elements; wherein the collected image is collected by a sensor in a host device; sending, by the cross-language bridging tool, a memory address of the image matrix elements in the local shared memory and preset image description information corresponding to the collected image to the second language process; reading, by the second language process, the image matrix elements from the local shared memory according to the memory address; reconstructing the image matrix elements according to the preset image description information to obtain a reconstructed image consistent with the collected image; performing image processing on the reconstructed image according to a preset image processing algorithm, and returning the image processing result to the first language process through the cross-language bridging tool; sending, by the first language process, the image processing result to the host device.

[0008] In some embodiments of the present application, after receiving, by the first language process, the image data of the collected image and storing the image data in the local shared memory, the method further comprises: determining, by the first language process, the memory address of the image matrix elements in the local shared memory; adding a lock identifier at a starting bit of the memory address, and performing a page lock operation on the local shared memory storing the image matrix elements; sending, by the cross-language bridging tool, the locked memory address to the second language process.

[0009] In some embodiments of the present application, after performing image processing on the reconstructed image according to a preset image processing algorithm, and returning the image processing result to the first language process through the cross-language bridging tool, the method further comprises: releasing, by the first language process, the page lock of the local shared memory storing the image matrix elements.

[0010] In some embodiments of the present application, the preset image description information comprises an image width, an image height and an image channel number corresponding to the collected image; The second language process is pre-provisioned with an image reconstruction process, wherein the reconstruction process is used to convert the memory address into a memory pointer, access the local shared memory through the memory pointer, read the image matrix elements, and reconstruct the reconstructed image in combination with the image size indicated by the image width and the image height, and the channel number indicated by the image channel number; Before sending the memory address of the image matrix element in the local shared memory and the preset image description information corresponding to the captured image to the second language process through the cross-language bridging tool, the method further includes: adding a second language script file defining the image reconstruction process to a project directory of the first language project, and configuring the second language script file as an additional file of the first language analyzer; triggering the first language analyzer to analyze the additional file and obtain an analysis result, and triggering the first language source code generator to generate a first language static class corresponding to the image reconstruction process as a calling interface, by the cross-language bridging tool, during compilation of the first language project.

[0011] In some embodiments of the present application, sending the memory address of the image matrix element in the local shared memory and the preset image description information corresponding to the captured image to the second language process through the cross-language bridging tool includes: inputting the memory address as a parameter into the first language static class, and passing the memory address to the image reconstruction process through the first language static class.

[0012] In some embodiments of the present application, when the captured image is at least two, the image data of each captured image is independently allocated local shared memory; the preset image description information further includes an image index for representing a capturing order, an image identifier for uniquely identifying the captured image, and a mapping relationship table between the image index, the image identifier, and the memory address; Before sending the memory address of the image matrix element in the local shared memory and the preset image description information corresponding to the captured image to the second language process through the cross-language bridging tool, the method further includes: determining the memory address of the image matrix element in the local shared memory by the first language process; adding a lock identifier at a starting bit of the memory address, and performing a page lock operation on the local shared memory storing the captured image; generating a corresponding image identifier and image index for each captured image; constructing a mapping relationship table with the memory address of the image matrix element corresponding to each captured image in the local shared memory as a value, and the image identifier and image index corresponding to each captured image as a key.

[0013] In some embodiments of the present application, after performing image processing on the reconstructed image according to a preset image processing algorithm and returning the image processing result to the first language process through the cross-language bridging tool, the method further includes: extracting the corresponding image index and / or image identifier from the image processing result by the first language process; According to the extracted image index and / or image identifier, a mapping relationship table is queried to determine a memory address where the image matrix element corresponding to the image processing result is stored, and the page lock is released.

[0014] In some embodiments of the present application, the memory address of the image matrix element in the local shared memory and the preset image description information corresponding to the captured image are sent to the second language process through the cross-language bridging tool, including: An image index list and an image identifier list are constructed as parameters input into the first language static class; Through the first language static class, a memory address list is found based on the mapping relationship table and is passed to the image reconstruction process preset in the second language process.

[0015] Another aspect of the present application provides an edge device including a processor, a memory, and a computer program or instructions stored on the memory, the processor being configured to execute the computer program or instructions, and the device implementing the steps of the human factor intelligent image processing method as described above when the computer program or instructions are executed.

[0016] Another aspect of the present application provides a computer readable storage medium having a computer program or instructions stored thereon, the computer program or instructions being executed by a processor to implement the steps of the human factor intelligent image processing method as described above.

[0017] The human factor intelligent image processing method and the edge device of the present application can solve the problems of large resource occupation and high transmission delay in the conventional cross-process image transmission method. After the first language process of the edge device receives the image data of the captured image sent by the host device, the image matrix element is saved in the local shared memory in the form of an image matrix element. The memory address and the preset image description information of the captured image are sent to the second language process through the cross-language bridging tool. The second language process reads the image matrix element in the local shared memory according to the memory address, and reconstructs the reconstructed image consistent with the captured image according to the image layout indicated by the preset image description information, thereby realizing zero-copy image transmission, avoiding the large resource occupation problem caused by image data copy, and avoiding the additional processing steps introduced by image data transmission, which can improve the system response speed while reducing data transmission delay and memory resource pressure. At the same time, the parameter transmission and method call of the second language process source file are realized through the cross-language bridging tool, which can improve the calling efficiency and further improve the system response speed.

[0018] In addition, the memory area where the image matrix element is stored is subjected to a page lock operation to prevent the memory area from being moved or released by the garbage collection mechanism, so that the second language process can successfully access and read the image matrix element corresponding to the captured data, and the success rate of the second language process processing the image is ensured.

[0019] In addition, the parameter passing and method calling of the second language process source file are realized through the cross-language bridging tool, so that the calling efficiency is improved, and the response speed and the image processing efficiency are further improved.

[0020] In addition, the parameter passing and method calling of the second language process source file are realized through the cross-language bridging tool, so that the calling efficiency is improved, and the response speed and the image processing efficiency are further improved.

[0021] Additional advantages, objects, and features of the application will be set forth in part in the description which follows, and will become apparent to those skilled in the art upon examination of the following or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

[0022] It will be understood by those skilled in the art that the objects and advantages of the present application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings: Figure 1 The system structure schematic diagram of the image matrix element transmission system based on human factor intelligence provided for an embodiment of the present application.

[0024] Figure 2 The flow chart of the image processing method based on human factor intelligence provided for another embodiment of the present application.

[0025] Figure 3 The flow chart of the image processing method based on human factor intelligence provided for another embodiment of the present application.

[0026] Figure 4 The device structure schematic diagram of the edge device provided for another embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with embodiments and drawings. Herein, the illustrative embodiments of the present application and the description thereof are used to explain the present application, but not as a limitation to the present application.

[0028] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0029] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0030] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0031] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0032] Figure 1 This is a schematic diagram of an image processing system based on human factors intelligence provided in one embodiment of this application. Figure 1 As shown, the system includes at least: a host device 110 and an edge device 120.

[0033] The host device 110 refers to the device that acquires images through sensors and sends the image data of the acquired images to the edge device 120, including but not limited to mobile phones, tablets, PCs, or servers.

[0034] The sensors include, but are not limited to, charge-coupled devices (CCDs) and complementary metal-oxide-semiconductors (CMOS). This embodiment does not limit the type of sensor.

[0035] In practice, the sensor can be installed in the host device 110 or communicate with the host device 110 as an independent device. This embodiment does not limit the connection relationship between the sensor and the host device 110.

[0036] like Figure 1 As shown, a wired network connection or a wireless network connection based on the TCP / IP protocol is established between the host device 110 and the edge device 120 for transmitting the acquired images.

[0037] The edge device 120 refers to a device that receives image data of the collected image sent by the host device 110 and performs image processing. The edge device 120 can be a mobile phone, a tablet computer, a computer PC, a workstation, or the like. The edge device 120 can also be a smart home device (for example, a smart speaker, a smart home appliance), a vehicle-mounted device, a gateway, or an edge server, and the like. The implementation of the edge device 120 is not limited in the present embodiment.

[0038] Specifically, the host device 110 is configured as a TCP client, and the edge device 120 is configured as a TCP server. In the connection establishment stage, the edge device 120 continuously listens to the IP address and port number of the host device 110, and waits for an incoming connection request. The host device 110 initiates a connection request to the IP address and port number of the edge device 120. After receiving the connection request, the edge device 120 responds and accepts the request, and then completes the three-way handshake process, thereby establishing a stable and full-duplex TCP network connection between the host device 110 and the edge device 120.

[0039] After the host device 110 collects the collected image through the sensor, the image data of the collected image is transmitted to the edge device 120 through the network connection based on the TCP / IP protocol.

[0040] As shown in FIG. 1, the edge device 120 runs a pre-configured first language process, a second language process, and a cross-language bridging tool. Figure 1 The programming language for implementing the first language process is different from the programming language for implementing the second language process.

[0041] In some embodiments of the present application, the first language process is a process implemented by C# language, and the second language process is a process implemented by Python language.

[0042] In actual implementation, the first language process and the second language process can also be processes implemented based on other programming languages, including but not limited to C++ programming language or JAVA programming language, and the like. The implementation of the first language process and the implementation of the second language process are not limited in the present embodiment.

[0043] A cross-language bridging tool is a tool used to achieve zero-copy image acquisition and transmission between a first-language process and a second-language process. The host device 110 acquires images through sensors and digitizes them into image data (i.e., a set of pixel values) before transmitting them to the edge device 120. The edge device 120 continuously listens to and receives the image data from the host device 110 through its first-language process, storing the image data in local shared memory as image matrix elements, where each matrix element corresponds to the intensity or channel value of a pixel.

[0044] Specifically, the edge device 120 receives image data of the acquired images through a first language process and stores the image data in local shared memory in the form of image matrix elements. The acquired images are obtained by sensors in the host device 110.

[0045] In addition, to ensure that the second language process can successfully access and read the image matrix elements corresponding to the acquired images in local memory before performing image processing, after storing the image data of the acquired images in local shared memory, it is also necessary to perform a page locking operation on the local shared memory address storing the image matrix elements using a fixed statement. This prevents the memory area from being moved or released by the garbage collection mechanism. Simultaneously, the locked memory address is recorded for use in subsequent release operations.

[0046] For example, in a process where the first language is C#, page locking operations require enabling unsafe code. By adding a locking flag, such as an unsafe flag, to a code block, memory pointers and direct memory access are allowed in managed code to bypass C#'s default type safety and memory management restrictions. At the same time, the garbage collector (GC) is instructed not to move or release the memory region during this operation.

[0047] In some embodiments of this application, after receiving image data of the acquired image through the first language process and storing the image data in local shared memory, the method further includes: determining the memory address of the image matrix element in the local shared memory through the first language process; adding a locking flag at the beginning of the memory address and performing a page locking operation on the local shared memory storing the image matrix element; and sending the locked memory address to the second language process through a cross-language bridging tool.

[0048] Subsequently, the edge device 120 initiates the image processing flow, and through the cross-language bridging tool, it transmits the memory address of the image matrix element corresponding to the acquired image in the local shared memory and the preset image description information of the acquired image to the second language process, thereby realizing zero-copy transmission of the acquired image across language processes.

[0049] Specifically, the edge device 120 uses a cross-language bridging tool to send the memory addresses of the image matrix elements in the local shared memory, as well as the preset image description information corresponding to the acquired image, to the second language process.

[0050] The preset image description information is used to indicate the image layout of the acquired image, including the image width, image height and number of image channels corresponding to the acquired image.

[0051] In some embodiments of this application, an image reconstruction process is pre-configured in the second language process. The image reconstruction process is used to convert memory addresses into memory pointers, access local shared memory through the memory pointers, read image matrix elements, and reconstruct the image by combining the image size indicated by the image width and image height, and the number of channels indicated by the number of image channels.

[0052] Edge device 120 sends the memory address to the cross-language bridging tool through the first language process. After receiving the memory address, the cross-language bridging tool passes the memory address to the image reconstruction process through a pre-generated first language static class with the same name as the image reconstruction process in the second language process.

[0053] Among them, the first language static class is automatically generated by the cross-language bridging tool during the compilation phase by parsing the second language source files during the compilation of the first language project through the first language analyzer and the first language source generator.

[0054] Specifically, the runtime component of the cross-language bridging tool is installed in the first language project; then, the second language script file defining the image reconstruction process is added to the project directory of the first language project and configured as an attachment file for the first language analyzer; during the compilation of the first language project, the cross-language bridging tool triggers the first language analyzer to perform syntactic and semantic analysis on the attachment file in the project directory of the first language project, generating structured analysis results (such as symbol tables and abstract syntax trees); subsequently, the first language source code generator is triggered, receiving the analysis results as input, parsing the semantic information of the attachment file, and generating a first language static class corresponding to the image reconstruction process as a calling interface.

[0055] In some embodiments of this application, before sending the memory addresses of the image matrix elements in the local shared memory and the preset image description information corresponding to the acquired images to the second language process via the cross-language bridging tool, the method further includes: adding the second language script file defining the image reconstruction process to the project directory of the first language project and configuring it as an attachment file of the first language analyzer; triggering the first language analyzer to parse the attachment file and obtain the analysis results during the compilation of the first language project via the cross-language bridging tool, and triggering the first language source code generator to generate a first language static class corresponding to the image reconstruction process as a calling interface by taking the analysis results as input.

[0056] Specifically, after receiving the memory address passed by the static class in the first language, the image reconstruction process in the second language process reads the image matrix elements from the local shared memory to reconstruct the image. Simultaneously, to ensure the integrity of the reconstructed image and avoid image errors or inconsistent image sizes, it needs to combine the image dimensions indicated by the image width and height, as well as the number of channels indicated by the number of image channels, to reconstruct an image consistent with the acquired image.

[0057] For example, when the actual image width, image height, and number of image channels of the acquired image are (1920, 1080, 4), the edge device 120 will process it as a 4-channel image instead of forcibly processing it as a 3-channel image to ensure image integrity. It will then reconstruct an image consistent with the acquired image according to the image layout indicated by the preset image description information.

[0058] In some embodiments of this application, the cross-language bridging tool is the CSnakes tool.

[0059] This explanation uses a process implemented in C# as the first language and Python as the second language as the second. The second language process triggers the call to the image reconstruction process by passing the received memory address as a parameter through the interface function provided by the CSNakes tool. Before calling the Python method, the CSNakes tool is used as an intermediary to generate C# type code based on the Python method, so that the Python method can be used and recognized by C#. Specifically, first, CSNakes.Runtime is installed on the edge device 120; then, the ".py" file corresponding to the Python method is placed in the program directory and set as an attachment file for the C# analyzer. During project compilation, the corresponding C# type file and method are automatically generated based on the ".py" file corresponding to the Python method, resulting in a C# static class; finally, the Process method defined in Python is called through the C# static class, passing the memory address as a parameter. After the second language process obtains the image matrix elements based on the memory address, it constructs the corresponding image array using the Python numerical computing library (Numerical Python, NumPy) to reconstruct the acquired image. The code for the image reconstruction process receiving the memory address and reconstructing the image is shown below: def process(ptr:int)->str: np_array = np_array.reshape((height, width, channels)) Where process is the function name; ptr:int is the function parameter, ptr represents an integer variable (int), i.e., the memory address; np_array represents a NumPy array; height represents the height of the acquired image; width represents the width of the acquired image; and channels represents the number of channels of the acquired image.

[0060] In practice, cross-language bridging tools can also be tools based on the Remote Procedure Call (gRPC) framework, tools based on the Zero Message Queue (ZeroMQ) message middleware, or tools based on a custom TCP / IP protocol. This embodiment does not limit the implementation method of the cross-language bridging tool.

[0061] After the second language process reconstructs the image to be consistent with the acquired image, the edge device 120 performs image processing on the reconstructed image according to the preset image processing algorithm, obtains the image processing result, and returns it to the host device 110.

[0062] The preset image processing algorithm may include image preprocessing algorithms (such as data cleaning algorithms or image format conversion algorithms) and image processing algorithms pre-set according to preset tasks.

[0063] In some embodiments of this application, the preset task may include an image enhancement task, an image segmentation task, or an image feature extraction task. Accordingly, the image processing algorithm pre-set according to the preset task may include the algorithm corresponding to the image enhancement task, the algorithm corresponding to the image segmentation task, and the image algorithm corresponding to the image feature extraction task.

[0064] The algorithms for image enhancement tasks include, but are not limited to, contrast adjustment algorithms, spatial domain filtering algorithms, or frequency domain filtering algorithms. The algorithms for image segmentation tasks include, but are not limited to, threshold-based image segmentation algorithms, edge-based image segmentation algorithms, or region-based image segmentation algorithms. The image algorithms for image feature extraction tasks include, but are not limited to, color feature extraction algorithms, texture feature extraction algorithms, or keypoint feature extraction algorithms.

[0065] In other embodiments of this application, the preset task includes an object detection task or an image classification task. Accordingly, the image processing algorithm includes image processing algorithms based on deep learning frameworks and image processing algorithms based on classifiers. The classifiers include, but are not limited to, Adaptive Boosting (AdaBoost) classifiers and Support Vector Machine (SVM) classifiers.

[0066] For example, taking a face detection task and image processing algorithms, including image processing methods based on facial expression recognition models, as a preset task, after the second language process reconstructs an image consistent with the acquired image, the reconstructed image is converted into the standard format required for subsequent model processing, including color space conversion, image size adjustment, and pixel data normalization; the pre-trained face detection model and expression recognition model are loaded, and the processed data is input into the face detection model and expression recognition model for inference, outputting the bounding box coordinates of each face, the expression recognition result, and the corresponding confidence score as the image processing result.

[0067] After the edge device 120 obtains the image processing result through the second language process, it returns the image processing result to the host device 110 via a network connection. The host device 110 also includes a display component for displaying the image processing result after receiving it.

[0068] Specifically, the edge device 130 is also used to perform image processing on the reconstructed image according to a preset image processing algorithm, and return the image processing result to the first language process through a cross-language bridging tool; and send the image processing result to the host device 110 through the first language process.

[0069] In some embodiments of this application, after processing the reconstructed image according to a preset image processing algorithm and returning the image processing result to the first language process through a cross-language bridging tool, the method further includes: releasing the page lock of the local shared memory storing the image matrix elements through the first language process.

[0070] Edge device 120 releases the page lock on the local shared memory storing image matrix elements through a first-language process to free the memory address. Part of its code is shown below: public static string process(int ptr){ / / Receive image processing results / / Release memory address } In summary, the human-centric intelligent image processing system provided in this embodiment can solve the problems of high resource consumption and high transmission latency in traditional cross-process image processing methods. After receiving the image data of the acquired image sent by the host device, the first language process of the edge device saves the image data in the form of image matrix elements in local shared memory and performs page locking operation on the local shared memory storing the image matrix elements. Through the cross-language bridging tool, the memory address and the preset image description information of the acquired image are sent to the second language process. The cross-language bridging project passes the memory address to the image reconstruction process with the same name as the first language static class through a pre-generated first language static class. The image reconstruction process reads the image matrix elements in the local shared memory according to the memory address and reconstructs the image consistent with the acquired image according to the image layout indicated by the preset image description information. This achieves zero-copy image transmission, avoids the large resource consumption problem caused by copying image data, and also avoids the additional processing steps introduced by image data transmission. It can improve the system response speed while reducing data transmission latency and memory resource pressure. At the same time, the parameter passing and method calling of the source file of the second language process through the cross-language bridging tool can improve the calling efficiency and further improve the system response speed.

[0071] The image processing method based on human factors intelligence provided in this application will be described in detail below.

[0072] like Figure 2 As shown, an embodiment of this application provides an image processing method based on human factors intelligence. This embodiment uses this method for... Figure 1The method is illustrated using an edge device as an example, wherein a first language process, a second language process, and a cross-language bridging tool are pre-configured in the edge device. The method includes at least the following steps S201-S206: Step S201: The image data of the acquired image is received through the first language process and stored in local shared memory in the form of image matrix elements. The acquired image is obtained by a sensor in the host device.

[0073] In some embodiments of this application, after the host device acquires the captured image, it sends the image data of the captured image to the edge device sequentially according to the acquisition order through a wired network connection or a wireless network connection with the edge device. Alternatively, the host device sends the image data of the captured image to the edge device at preset frame intervals (e.g., 5 frames, 10 frames, etc.).

[0074] The edge device continuously listens to and receives image data from the host device through the first language process, and stores the image data in local shared memory. It stores and accesses the data in the form of image matrix elements, where each matrix element corresponds to the intensity value or channel value of a pixel.

[0075] Subsequently, the edge device initiates the image processing flow through the first language process. Using a cross-language bridging tool, it passes the memory addresses of the image matrix elements corresponding to the acquired image in local shared memory, along with the preset image description information of the acquired image, to the second language process. The second language process reads the image matrix elements corresponding to the acquired image according to the memory addresses, and, combined with the preset image description information, reconstructs a reconstructed image consistent with the acquired image, and then performs image processing.

[0076] To ensure that the second language process can successfully access and read the image matrix elements corresponding to the acquired image before performing image processing, after storing the image data of the acquired image in local shared memory, it is necessary to use a fixed statement to fix the address of the local shared memory storing the image matrix elements to prevent this memory area from being moved or released by the garbage collection mechanism. At the same time, the locked memory address is recorded for use in subsequent release operations.

[0077] For example, in a process where the first language is C#, by adding a locking flag, such as an unsafe flag, to a code block, memory pointers and direct memory access can be used in managed code to break through the default type safety and memory management restrictions of C#. At the same time, the garbage collector (GC) is instructed not to move or release the memory region during this operation.

[0078] In some embodiments of this application, after receiving image data of the acquired image through a first language process and storing the image data in local shared memory, the method further includes: determining the memory address of the image matrix element in the local shared memory through the first language process; adding a locking flag at the beginning of the memory address and performing a page locking operation on the local shared memory storing the image matrix element; and sending the locked memory address to the second language process through a cross-language bridging tool.

[0079] In practice, the host device may include at least two sensors. These sensors independently acquire images, and the image data of at least two acquired images are simultaneously transmitted to the edge device via a network connection between the host device and the edge device. Alternatively, after acquiring multiple images (e.g., two or three) through a single sensor, the host device simultaneously transmits the image data of all multiple images to the edge device. Upon receiving image data from at least two acquired images, the edge device independently allocates local shared memory for each acquired image.

[0080] In some embodiments of this application, when at least two images are acquired, the image data of each acquired image is independently allocated to local shared memory.

[0081] In step S202, the memory address of the image data in the local shared memory and the preset image description information corresponding to the acquired image are sent to the second language process through the cross-language bridging tool.

[0082] The preset image description information is used to indicate the image layout of the acquired image, including the image width, image height and number of image channels corresponding to the acquired image.

[0083] In some embodiments of this application, the second language process includes a pre-configured image reconstruction process. The image reconstruction process is used to convert memory addresses into memory pointers, access local shared memory through the memory pointers, read image matrix elements, and reconstruct the image by combining the image size indicated by the image width and image height, and the number of channels indicated by the number of image channels.

[0084] The edge device sends the memory address to the cross-language bridging tool through the first language process. After receiving the memory address, the cross-language bridging tool passes the memory address to the image reconstruction process through a pre-generated first language static class with the same name as the image reconstruction process in the second language process.

[0085] In some embodiments of this application, the memory address of the image matrix elements in the local shared memory and the preset image description information corresponding to the acquired image are sent to the second language process through a cross-language bridging tool. This includes: inputting the memory address as a parameter into the first language static class, and passing the memory address to the image reconstruction process through the first language static class.

[0086] Among them, the first language static class is a cross-language bridging tool that, during the compilation phase of the first language project, parses the second language source file and automatically generates it through the first language analyzer and the first language source code generator.

[0087] Specifically, the runtime component of the cross-language bridging tool is installed in the first language project; then, the second language script file defining the image reconstruction process is added to the project directory of the first language project and configured as an attachment file for the first language analyzer; during the compilation of the first language project, the cross-language bridging tool triggers the first language analyzer to perform syntactic and semantic analysis on the attachment file in the project directory of the first language project, generating structured analysis results (such as symbol tables and abstract syntax trees); subsequently, the first language source code generator is triggered, receiving the analysis results as input, parsing the semantic information of the attachment file, and generating a first language static class corresponding to the image reconstruction process as a calling interface.

[0088] In some embodiments of this application, before sending the memory addresses of the image matrix elements in the local shared memory and the preset image description information corresponding to the acquired images to the second language process via the cross-language bridging tool, the method further includes: adding the second language script file defining the image reconstruction process to the project directory of the first language project and configuring it as an attachment file of the first language analyzer; using the cross-language bridging tool, when the first language project is compiled, triggering the first language analyzer to analyze the attachment file, obtaining the analysis results, and triggering the first language source code generator to use the analysis results as input to generate a first language static class corresponding to the image reconstruction process as a calling interface.

[0089] In some embodiments of this application, the cross-language bridging tool is the CSNakes tool. In actual implementation, the cross-language bridging tool can also be a tool based on the gRPC framework, a tool based on the ZeroMQ message middleware, or a tool based on a custom TCP / IP protocol. This embodiment does not limit the implementation method of the cross-language bridging tool.

[0090] In some embodiments of this application, when at least two images are acquired, the preset image description information further includes an image index representing the acquisition order, an image identifier uniquely identifying the acquired image, and a mapping table between the image index, the image identifier, and memory addresses. The image identifier can be a character, a number, or a combination of characters and numbers; this embodiment does not limit the implementation method of the image identifier. The image indices of images acquired in different batches are different, while the image indices of images acquired in the same batch are the same, and the image identifiers of different acquired images are different.

[0091] Before sending the memory addresses of the image matrix elements in the local shared memory and the preset image description information corresponding to the acquired images to the second language process via the cross-language bridging tool, the process further includes: determining the memory addresses of the image matrix elements in the local shared memory through the first language process; adding a locking flag at the beginning of the memory address to perform page locking operations on the local shared memory storing the image matrix elements; generating a corresponding image identifier and image index for each acquired image; and constructing a mapping table using the memory address of the image matrix element corresponding to each acquired image in the local shared memory as the value and the image identifier and image index corresponding to each acquired image as the key.

[0092] Accordingly, through the cross-language bridging tool, the memory addresses of the image matrix elements in the local shared memory, as well as the preset image description information corresponding to the acquired images, are sent to the second language process. This includes: constructing an image index list and an image identifier list, which are input as parameters into the first language static class; and using the first language static class, based on the mapping table, finding the memory address list and passing it to the preset image reconstruction process in the second language process.

[0093] Step S203: The image matrix elements are read from the local shared memory according to the memory address by the second language process.

[0094] In some embodiments of this application, the acquired image is a single image. In this case, the parameter passed by the first language static class is a memory address. The image reconstruction process in the second language process converts the memory address passed by the first language static class into a memory pointer, accesses the local shared memory through the memory pointer, and reads the image matrix elements corresponding to the acquired image.

[0095] In some other embodiments of this application, the acquired images include at least two images. In this case, the parameter passed by the first language static class is a list of memory addresses. The image reconstruction process in the second language process is used to convert the list of memory addresses into memory pointers, access local shared memory through the memory pointers, and read the image matrix elements of multiple acquired images in parallel.

[0096] Step S204: Based on the preset image description information, the image matrix elements are reconstructed to obtain a reconstructed image that is consistent with the acquired image.

[0097] In some embodiments of this application, the second language process is a process implemented in Python. After the second language process obtains the image matrix elements, it constructs the corresponding image array using NumPy (Numerical Python) to reconstruct the acquired images.

[0098] Step S205: Perform image processing on the reconstructed image according to the preset image processing algorithm, and return the image processing result to the first language process through the cross-language bridging tool.

[0099] After obtaining the reconstructed image, the second language process processes the reconstructed image according to a preset image processing algorithm to obtain the processing results and corresponding confidence information. Then, according to the preset return data type, the processing results and related information are encapsulated into an image processing result and returned to the first language process.

[0100] The preset image processing algorithm includes an image preprocessing algorithm and an image processing algorithm pre-set according to the preset task.

[0101] In some embodiments of this application, the preset task includes an image enhancement task, an image segmentation task, or an image feature extraction task. Accordingly, the image processing algorithms pre-set according to the preset task include algorithms corresponding to the image enhancement task, algorithms corresponding to the image segmentation task, and image algorithms corresponding to the image feature extraction task.

[0102] In other embodiments of this application, the preset task includes an object detection task or an image classification task. Accordingly, the image processing algorithm includes image processing algorithms based on deep learning frameworks and image processing algorithms based on classifiers.

[0103] After the second language process returns the image processing results to the first language process, the second language process continues to wait for the cross-language bridging tool to transmit new image description data and memory addresses. Alternatively, it terminates the image processing flow upon receiving an image processing termination command from the first language process.

[0104] In some embodiments of this application, when the acquired image is a single image, after receiving the image processing result, the first language process calls the Dispose method to notify the Garbage Collector (GC) to release the fixed memory resources based on the previously recorded memory address. After processing the reconstructed image according to a preset image processing algorithm and returning the image processing result to the first language process through a cross-language bridging tool, the process further includes: releasing the page lock of the local shared memory storing the image matrix elements through the first language process.

[0105] In some other embodiments of this application, when multiple images are acquired, the image processing result also includes the image index and image identifier corresponding to the processed image. The first language process, based on the image index and image identifier extracted from the image processing result, locates the target memory region where the image matrix element related to the result is located, and unlocks the page lock of that region, thereby safely reclaiming memory resources. After processing the reconstructed image according to a preset image processing algorithm and returning the image processing result to the first language process through a cross-language bridging tool, the process further includes: extracting the corresponding image index and / or image identifier from the image processing result through the first language process; querying the mapping table based on the extracted image index and / or image identifier to determine the memory address storing the image matrix element corresponding to the image processing result, and unlocking the page lock.

[0106] Step S206: The image processing result is sent to the host device through the first language process.

[0107] The host device includes a display component, which displays the interface or performs subsequent processing after receiving the image processing results returned by the edge device.

[0108] In summary, the human-centric intelligence-based image processing method provided in this embodiment can solve the problems of high resource consumption and high transmission latency in traditional cross-process image transmission methods. After receiving the image data of the acquired image sent by the host device, the first language process of the edge device stores it in local shared memory in the form of image matrix elements. Through a cross-language bridging tool, the memory address and the preset image description information of the acquired image are sent to the second language process. The second language process reads the image matrix elements in the local shared memory according to the memory address and reconstructs the image consistent with the acquired image according to the image layout indicated by the preset image description information. This achieves zero-copy image transmission, avoiding the large resource consumption caused by copying image data and avoiding the additional processing steps introduced by image data transmission. It can improve the system response speed while reducing data transmission latency and memory resource pressure. At the same time, the parameter passing and method calling of the source file of the second language process through the cross-language bridging tool can improve the calling efficiency and further improve the system response speed.

[0109] In addition, page locking is performed on the memory area storing image matrix elements to prevent the memory area from being moved or released by the garbage collection mechanism, so that the second language process can successfully access and read the image matrix elements corresponding to the acquired data, thus ensuring the success rate of the second language process in processing images.

[0110] In addition, by using cross-language bridging tools to pass parameters and call methods from second-language process source files, calling efficiency can be improved, thereby further improving response speed and image processing efficiency.

[0111] To better understand the human-centric intelligence-based image processing method provided in this application, an example is given below. In this example, this embodiment uses the method for... Figure 1 The system shown is used as an example for illustration. Figure 3 The method includes steps S301 to S309: In step S301, the edge device, through a cross-language bridging tool, triggers the parsing of the image reconstruction process source file preset in the second language process during the compilation of the first language project, and generates a static class in the first language.

[0112] In step S302, the host device acquires images through sensors.

[0113] In step S303, the host device sends the image data of the acquired image to the edge device through the network connection with the edge device.

[0114] In step S304, the edge device receives the image data of the acquired image through the first language process and stores it in local shared memory in the form of image matrix elements.

[0115] In step S305, the edge device sends the memory address of the image matrix elements in the local shared memory and the preset image description information corresponding to the acquired image to the second language process through the cross-language bridging tool.

[0116] In step S306, the edge device reads the image matrix elements from the local shared memory according to the memory address through the second language process.

[0117] In step S307, the edge device reconstructs the image matrix elements based on the image description information through a second language process, and obtains a reconstructed image that is consistent with the acquired image.

[0118] In step S308, the edge device performs image processing on the reconstructed image according to a preset image processing algorithm through the second language process, and returns the image processing result to the first language process through a cross-language bridging tool.

[0119] In step S309, the edge device sends the image processing results to the host device through the first language process.

[0120] For relevant details, please refer to the above methods and system implementation examples; these details will not be repeated here.

[0121] In summary, the human-centric intelligent image processing method provided in this embodiment can solve the problems of high resource consumption and high transmission latency in traditional cross-process image processing methods. After receiving the image data of the acquired image sent by the host device, the first language process of the edge device saves the image data in the form of image matrix elements in local shared memory and performs page locking operation on the local shared memory storing the image matrix elements. Through the cross-language bridging tool, the memory address and the preset image description information of the acquired image are sent to the second language process. The cross-language bridging project passes the memory address to the image reconstruction process with the same name as the first language static class through a pre-generated first language static class. The image reconstruction process reads the image matrix elements in the local shared memory according to the memory address and reconstructs the image consistent with the acquired image according to the image layout indicated by the preset image description information. This achieves zero-copy image transmission, avoids the large resource consumption caused by copying image data, and avoids the additional processing steps introduced by image data transmission. It can improve the system response speed while reducing data transmission latency and memory resource pressure. At the same time, the parameter passing and method calling of the source file of the second language process through the cross-language bridging tool can improve the calling efficiency and further improve the system response speed.

[0122] Corresponding to the above methods, such as Figure 4As shown, the present invention also provides an edge device, which includes a computer device including a processor 401 and a memory 402. The memory 402 stores computer instructions, and the processor 401 is used to execute the computer instructions stored in the memory 402. When the computer instructions are executed by the processor 401, the device implements the steps of the human-centric intelligence-based image processing method as described above.

[0123] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned human-factor intelligence-based image processing method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0124] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0125] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0126] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0127] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A human factor intelligence-based image processing method, characterized by, The method is suitable for an edge device, wherein a first language process, a second language process and a cross-language bridging tool are pre-configured in the edge device; the method comprises: receiving, by the first language process, image data of a collected image, and storing the image data in a local shared memory in the form of image matrix elements; wherein the collected image is collected by a sensor in a host device; sending, by the cross-language bridging tool, a memory address of the image matrix elements in the local shared memory and preset image description information corresponding to the collected image to the second language process; reading, by the second language process, the image matrix elements from the local shared memory according to the memory address; reconstructing the image matrix elements according to the preset image description information to obtain a reconstructed image consistent with the collected image; performing image processing on the reconstructed image according to a preset image processing algorithm, and returning image processing results to the first language process through the cross-language bridging tool; sending, by the first language process, the image processing results to the host device.

2. The method of claim 1, wherein, After the first language process receives the image data of the collected image and stores the image data in the local shared memory, the method further comprises: determining, by the first language process, a memory address of the image matrix elements in the local shared memory; adding a lock identifier at a starting bit of the memory address, and performing a page lock operation on the local shared memory storing the image matrix elements; sending, by the cross-language bridging tool, the locked memory address to the second language process.

3. The method of claim 2, wherein, After the first language process performs image processing on the reconstructed image according to a preset image processing algorithm, and returns image processing results to the first language process through the cross-language bridging tool, the method further comprises: releasing, by the first language process, the page lock of the local shared memory storing the image matrix elements.

4. The method of claim 1, wherein, The preset image description information comprises an image width, an image height and an image channel number corresponding to the collected image; The second language process is pre-provisioned with an image reconstruction process, wherein the image reconstruction process is configured to convert the memory address into a memory pointer, access the local shared memory through the memory pointer, read the image matrix elements, and reconstruct the reconstructed image according to an image size indicated by the image width and the image height, and a channel number indicated by the image channel number; Before the cross-language bridging tool sends the memory address of the image matrix elements in the local shared memory and the preset image description information corresponding to the collected image to the second language process, the method further comprises: adding a second language script file defining the image reconstruction process to a project directory of a first language project, and configuring the second language script file as an additional file of a first language analyzer; The cross-language bridging tool triggers the first language analyzer to analyze the additional file during the first language project compilation, obtains an analysis result, and triggers a first language source code generator to generate a first language static class corresponding to the image reconstruction process as a calling interface.

5. The method of claim 4, wherein, The cross-language bridging tool sends the memory address of the image matrix element in the local shared memory and preset image description information corresponding to the captured image to the second language process, including: The memory address is input as a parameter into the first language static class, and the memory address is transmitted to the image reconstruction process through the first language static class.

6. The method of claim 4, wherein, In the case of at least two captured images, local shared memories are independently allocated for image data of each captured image; the preset image description information further includes an image index for representing a capturing sequence, an image identifier for uniquely identifying the captured image, and a mapping relationship table between the image index, the image identifier, and the memory address; Before the cross-language bridging tool sends the memory address of the image matrix element in the local shared memory and the preset image description information corresponding to the captured image to the second language process, further comprising: The first language process determines the memory address of the image matrix element in the local shared memory; A lock identifier is added at a starting bit of the memory address, and a page locking operation is performed on the local shared memory storing the image matrix element; An image identifier and an image index corresponding to each captured image are generated respectively; The mapping relationship table is constructed by taking the memory address of the image matrix element corresponding to each captured image in the local shared memory as a value and taking the image identifier and the image index corresponding to each captured image as keys.

7. The method of claim 6, wherein, After the reconstructed image is processed according to the preset image processing algorithm and the image processing result is returned to the first language process through the cross-language bridging tool, further comprising: The first language process extracts the corresponding image index and / or image identifier from the image processing result; According to the extracted image index and / or image identifier, the mapping relationship table is queried to determine the memory address storing the image matrix element corresponding to the image processing result, and the page locking is released.

8. The method of claim 6, wherein, The cross-language bridging tool sends the memory address of the image matrix element in the local shared memory and the preset image description information corresponding to the captured image to the second language process, including: An image index list and an image identifier list are constructed and input as parameters into the first language static class; The first language static class finds a memory address list based on the mapping relationship table and transmits the memory address list to the image reconstruction process pre-stored in the second language process.

9. An edge device comprising a processor, a memory, and a computer program or instructions stored on the memory, wherein the computer program or instructions are configured to cause the processor to perform the method of any one of claims 1-8. The processor is configured to execute the computer program or instructions, and when the computer program or instructions are executed, the device implements the steps of the method according to any one of claims 1 to 8.

10. A computer readable storage medium having stored thereon a computer program or instructions, characterized in that, The computer program or instructions implement the steps of the method according to any one of claims 1 to 8 when executed by a processor.

Images