Sensing data processing method based on plug-in software architecture

Abstract

The invention relates to the technical field of sensing data processing, in particular to a sensing data processing method based on plug-in software architecture. The method comprises the steps that a sensing layer drives a sensor device to acquire dynamic data, the sensing layer comprises an image acquisition unit, the image acquisition unit is connected with an image sensing device, and the dynamic data at least comprises an image; the business layer is used for processing dynamic data of the sensor and comprises a plug-in module which is used for processing an image; the plug-in module is compiled according to a GSteam plug-in specification, and the plug-in module comprises an image processing unit, a visual reasoning unit, a business processing unit, an image rendering unit, a video coding unit and a video pushing unit; the management layer performs parameter configuration management on each plug-in module of the sensing layer and the service layer; and the network layer is responsible for network communication and pushing of the processed dynamic data of the sensor. According to the method, the implementation cost is low, unnecessary system resource consumption caused by data copying can be avoided, and reasoning results can be fused.

Description

Technical Field

[0001] The present invention relates to the field of sensor data processing technology, and in particular to a sensor data processing method, device and medium based on a plug-in software architecture. Background Art

[0002] Intelligent visual perception technology can detect and analyze the operating conditions of key equipment in real time, accurately and stably, so as to solve the problems of unreliability, low precision and single function of traditional physical sensors in harsh underground environments. The SOC chip is the core hardware for the realization of intelligent visual perception technology. Generally, chip design manufacturers integrate deep learning acceleration units and audio and video hardware codec units in their respective SOC chips. However, there are no unified standards or protocols between manufacturers, resulting in compatibility issues between devices and solutions from different manufacturers. As a result, every time a user selects a SOC chip from a manufacturer, they need to rebuild a set of platform software based on the SDK package provided by the manufacturer, which is costly to implement.

[0003] In addition, when performing multi-path reasoning on the same image data, the existing technology generally adopts a method where one camera corresponds to one reasoning engine, that is, the image collected by one camera is only loaded by one reasoning engine to perform reasoning calculations. When performing multi-path reasoning on the same image, it is often necessary to copy the data of the same image, which will occupy computing memory and waste resources; when performing multi-path reasoning on the same image, due to time synchronization, the reasoning results cannot be fused. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art.

[0005] To this end, the present invention provides a sensor data processing method based on a plug-in software architecture, which has low implementation cost, can avoid unnecessary system resource consumption due to data copying, and can fuse the results of reasoning.

[0006] According to the sensor data processing method based on the plug-in software architecture of the embodiment of the present invention, the software architecture framework includes a perception layer, a management layer, a business layer and a network layer from top to bottom, and the method includes the following steps:

[0007] The perception layer drives the sensor device to obtain dynamic data of the sensor. The perception layer includes an image acquisition unit, which is connected to the image sensor device. The dynamic data at least includes an image.

[0008] The business layer processes the dynamic data of the sensor, and the business layer includes at least one plug-in module, and the plug-in module is used to process the image;

[0009] The plug-in module is written according to the GSteamer plug-in specification, and includes an image processing unit, a visual reasoning unit, a business processing unit, an image rendering unit, a video encoding unit, and a video push unit;

[0010] The management layer performs parameter configuration management on each plug-in module of the perception layer and the business layer;

[0011] The network layer is responsible for network communication and pushing the processed dynamic data of the sensor.

[0012] The beneficial effects of the present invention are:

[0013] (1) The present invention modularizes each function and independently encapsulates each module according to the GStreamer plug-in mechanism, which facilitates the testing, management and transplantation of each plug-in and increases the stability and maintainability of the software framework;

[0014] (2) In the present invention, an image can be loaded by multiple algorithm models in a read-only manner at the same time, and each model runs synchronously during reasoning, and finally the reasoning results are integrated, thereby improving the utilization rate of the image. Each model jointly references the same set of data, avoiding unnecessary system resource consumption caused by data copying;

[0015] (3) The present invention can bind the tasks of the perception layer and the business layer by modifying the configuration file. The software automatically loads the sensor devices of the perception layer according to the configuration information, constructs the inference data flow channel, loads the algorithm model, and calls the business processing class interface;

[0016] (4) The present invention can uniformly package GPUs, NPUs, and hardware codecs of mainstream manufacturers. When the software is ported between different devices, it is only necessary to specify the model of the edge device in the compilation process to complete the software compilation work, and the adaptability is good.

[0017] According to an embodiment of the present invention, after the method is executed, firstly, a configuration file is loaded in the management layer, and parameters of the configuration file are configured;

[0018] The parameters in the configuration file include:

[0019] (1) The model of the hardware platform on which the software runs;

[0020] (2) Model and configuration parameters of image sensing equipment;

[0021] (3) The name and path of the algorithm model and the hyperparameters of the algorithm;

[0022] (4) Definition of the port number and data stream name of the RTSP server.

[0023] According to an embodiment of the present invention, the image acquisition unit acquires the model and configuration parameters of the image sensor device in the configuration file to collect the image;

[0024] The image processing unit preprocesses the image transmitted by the image acquisition unit according to the requirements in the configuration file, and the preprocessing includes image shearing, adjusting width and height, image flipping, image filtering and morphological image processing;

[0025] The visual reasoning unit loads the reasoning engine according to the model of the hardware platform on which the software in the configuration file runs, performs visual reasoning on the image after image processing, and performs accelerated calculation based on the hyperparameters of the algorithm in the configuration file to form a reasoning result;

[0026] The business processing unit performs business logic processing on the inference result based on the business requirements in the configuration file to obtain business layer data;

[0027] The image rendering unit obtains the business layer data and renders it in the image;

[0028] The video encoding unit loads an encoder according to the model of the hardware platform on which the software runs in the configuration file, and performs encoding processing on the rendered image;

[0029] The video push unit pushes the encoded image according to the port number of the RTSP server and the address information of the video stream in the configuration file.

[0030] According to an embodiment of the present invention, the business layer includes at least three threads when performing two-way reasoning when processing the image through a plug-in module, namely a main thread, a first algorithm reasoning thread, and a second algorithm reasoning thread;

[0031] The main thread is responsible for image acquisition, image processing, and business processing;

[0032] The first algorithm reasoning thread and the second algorithm reasoning thread are responsible for visual reasoning.

[0033] According to one embodiment of the present invention, the execution of three threads specifically includes the following steps:

[0034] The main thread drives the image acquisition unit to acquire the image, and processes the image through the image processing unit to obtain a pre-processed image, and transmits the pre-processed image;

[0035] The visual reasoning unit executes the first algorithm reasoning thread and the second algorithm reasoning thread, and during the execution, the main thread locks the first mutex lock and the second mutex lock, and the two mutex locks are locked, so that the main thread is blocked;

[0036] When the first algorithm reasoning thread infers the preprocessed image and obtains a first reasoning result, an unlocking operation is performed on the first mutex lock;

[0037] When the second algorithm reasoning thread infers the preprocessed image and obtains a second reasoning result, an unlocking operation is performed on the second mutex lock;

[0038] When both mutexes are unlocked, the blocked main thread is unblocked;

[0039] The business processing unit processes the acquired pre-processed image, the first reasoning result and the second reasoning result based on business requirements to obtain business layer data.

[0040] According to an embodiment of the present invention, each of the main threads corresponds to a business processing layer in the business processing unit and is responsible for user logic and business processing.

[0041] According to one embodiment of the present invention, the dynamic data also includes sensor data, and the business layer also includes a data filtering processing unit, which processes the sensor data and transmits the filtered sensor data to the business processing unit.

[0042] According to an embodiment of the present invention, the management layer further includes: log management and thread management, wherein the log management is used to save operation logs and exception logs, and the thread management is used to create and manage thread pools to optimize software structure.

[0043] A computer device comprising:

[0044] processor;

[0045] A memory for storing executable instructions;

[0046] The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the sensor data processing method based on the plug-in software architecture as described above.

[0047] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the processor implements the above-mentioned sensor data processing method based on the plug-in software architecture.

[0048] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0051] Figure 1 It is a structural diagram of the software architecture of the first embodiment of the present invention.

[0052] Figure 2 It is a flowchart of an image processing process according to a first embodiment of the present invention.

[0053] Figure 3 This is a diagram of the inference result data structure of the first embodiment of the present invention.

[0054] Figure 4 This is a flowchart of a two-way image reasoning process according to an embodiment of the present invention.

[0055] Figure 5 This is a flowchart of multi-sensor data processing according to a first embodiment of the present invention.

[0056] Figure 6 It is a structural block diagram of a computer device according to Embodiment 2 of the present invention.

[0057] In the figure, 10 is a computer device; 1002 is a processor; 1004 is a memory; and 1006 is a transmission device. DETAILED DESCRIPTION

[0058] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0059] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0060] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0061] Example 1

[0062] The present application embodiment provides a sensor data processing method based on a plug-in software architecture, such as Figure 1 As shown, the software architecture framework includes a perception layer, a management layer, a business layer, and a network layer from top to bottom, and the method includes the following steps:

[0063] The perception layer drives the sensor device to obtain dynamic data of the sensor. The perception layer includes an image acquisition unit, which is connected to the image sensor device. The dynamic data at least includes an image.

[0064] The business layer processes the dynamic data of the sensor, and the business layer includes at least one plug-in module, and the plug-in module is used to process the image;

[0065] Among them, the plug-in module is written according to the GSteamer plug-in specification, and the plug-in module includes an image processing unit, a visual reasoning unit, a business processing unit, an image rendering unit, a video encoding unit, and a video push unit;

[0066] The management layer manages the parameter configuration of each plug-in module in the perception layer and the business layer;

[0067] The network layer is responsible for network communication and pushing the processed dynamic data of sensors. The network layer supports protocols such as http, rtsp, tcp and udp.

[0068] In the embodiment, after the method is executed, firstly, a configuration file is loaded in the management layer, and parameters of the configuration file are configured;

[0069] The parameters in the configuration file include:

[0070] (1) The model of the hardware platform on which the software runs, such as "jetson-Xavier_NX" or "Rockchip-RK3588". The software needs to load the corresponding inference engine plug-in and encoder / decoder plug-in according to the hardware platform model.

[0071] (2) The model and configuration parameters of the image sensing device. The software needs to drive and adjust the camera according to the camera configuration information to obtain the best image.

[0072] (3) The name and path of the algorithm model and the hyperparameters of the algorithm, so that the software's inference engine plug-in can load the algorithm model and parameters to complete the algorithm reasoning;

[0073] (4) The port number and data stream name of the RTSP server are determined. The RTSP server plug-in needs to load configuration parameters to complete the push of the video stream.

[0074] In an embodiment, Figure 2 As shown, the image processing step is processed in sequence by acquiring images, image processing, visual reasoning, business processing, image rendering, video encoding, and video push. Specifically, the image acquisition unit acquires the model and configuration parameters of the image sensor device in the configuration file to collect the image. In other words, the image acquisition unit calls the image sensor device through the API interface based on the model of the image sensor device, and adjusts the parameters of the image sensor device according to the configuration parameters of the image sensor device to obtain the image of the corresponding image sensor device model. The image formats of the final outputs of image sensor devices of different models may be different, some output RGB format, and some output YUV format. For the convenience of management, the image acquisition unit will also convert the image output format and output I420 format. As an independent plug-in, the image acquisition unit can be directly run by the gst-launch tool.

[0075] The image processing unit pre-processes the image transmitted by the image acquisition unit according to the requirements in the configuration file. Furthermore, the image processing unit obtains the configuration information about image processing in the configuration file. The configuration information includes an enable switch for processing each frame of the image in the data stream pipeline. If the image processing function is turned on in the configuration information, it is also necessary to configure the image pre-processing method and the configuration parameters involved in the method. Supported image pre-processing methods include image cropping, adjusting width and height, image flipping, image filtering and morphological image processing. The image processing unit is used to highlight the target features, remove interference factors that have an adverse effect on the algorithm results, and improve the accuracy of the algorithm.

[0076] The visual reasoning unit loads the inference engine according to the model of the hardware platform on which the software runs in the configuration file to perform visual reasoning on the processed image, and performs acceleration calculations based on the hyperparameters of the algorithm in the configuration file to form the reasoning result. The formed reasoning result is stored in the structure infer_result. The main hierarchical relationship of the structure is as follows Figure 3 The main data definitions are shown in Table 1:

[0077]

[0078] The business processing unit performs business logic processing on the inference results to obtain business layer data. The business processing unit uses the class method for encapsulation. Each business class has a unique monitor_id identifier. The unique monitor_id identifier and the business processing class object form a key-value pair and are added to the hash table. The business processing unit performs business logic processing on the inference results based on the business requirements in the configuration file, specifically including: according to the value of "monitor_id" in the inference result, search for the business processing class object in the business hash table, that is, find the corresponding business requirements, and then call the business processing method of the class object to perform business logic processing. The corresponding business requirements are such as: when anchor rods, large pieces of gangue and other targets are detected on the main transport belt, an alarm message is sent to the server via http to remind the staff near the belt that foreign objects appear in the belt, etc.

[0079] It should be noted that the business processing class encapsulates a variety of communication protocols, including CAN communication, modbus_tcp communication, modbus_rtu communication, http communication, and ftp communication, which facilitates the transmission of business layer data. Business layer data includes the name of the target, the detection frame of the target, the coordinate information of the target, and the alarm message.

[0080] The image rendering unit obtains the business layer data and renders it in the image. Specifically, the image obtained by the image acquisition unit is first converted from I420 format to RGB format, and then the open source tool OPENCV is used to draw the corresponding target object detection frame at the target object position of the image, and the corresponding target object name and target object coordinate information are displayed.

[0081] The video encoding unit loads the encoder according to the model of the hardware platform on which the software in the configuration file runs, and performs encoding processing on the rendered image; the encoding processing is to use the encoding algorithm to filter and remove redundant information from multiple groups of continuously rendered images. The encoding algorithm is divided into software encoding and hardware encoding. Software encoding refers to the encoder directly using CPU resources for encoding. The GStreamer framework provides a software encoding plug-in "x264enc" to use the CPU to encode image data into H.264 format data. This plug-in has the advantages of being simple to use and cross-platform, but it has the problem of occupying a large amount of CPU resources and is not suitable for use in edge devices. Hardware encoding refers to the use of non-CPU devices to compress images. Most edge devices integrate hardware encoders and decoders to improve performance and reduce CPU load. In this embodiment, the video encoding unit uses the API interface provided by the edge device to encapsulate the hardware encoding function as a plug-in. During use, the encoder and decoder plug-ins are loaded according to the model of the hardware platform on which the software in the configuration file runs, and software encoding or hardware encoding is selected.

[0082] The video push unit pushes the encoded image according to the port number of the RTSP server and the address information of the video stream in the configuration file. For example, the image collected underground is encoded as a video stream and transmitted to the rtsp server. When the user uses the rtsp client software on the well surface to connect to the underground rtsp server, the real-time video stream can be obtained, which is convenient for the user to monitor the working condition of underground equipment or illegal operation of personnel on the well surface.

[0083] In an embodiment, Figure 4 As shown, when the business layer processes an image through a plug-in module and performs two-way reasoning, it includes at least three threads, namely the main thread, the first algorithm reasoning thread and the second algorithm reasoning thread; the main thread is responsible for image acquisition, image processing, and business processing; the first algorithm reasoning thread and the second algorithm reasoning thread are responsible for visual reasoning. This improves resource utilization.

[0084] Furthermore, the execution of the three threads specifically includes the following steps:

[0085] The main thread drives the image acquisition unit to acquire the image, and processes the image through the image processing unit to obtain the pre-processed image and transmit it;

[0086] The visual reasoning unit executes the first algorithm reasoning thread and the second algorithm reasoning thread. During the execution, the main thread locks the first mutex lock and the second mutex lock. The two mutex locks are locked, so that the main thread is blocked.

[0087] When the first algorithm reasoning thread reasons the preprocessed image and obtains a first reasoning result, an unlocking operation is performed on the first mutex lock;

[0088] When the second algorithm reasoning thread infers the preprocessed image and obtains a second reasoning result, an unlocking operation is performed on the second mutex lock;

[0089] When both mutexes are unlocked, the blocked main thread is unblocked;

[0090] The business processing unit processes the acquired pre-processed image, the first reasoning result and the second reasoning result based on business requirements to obtain business layer data.

[0091] In the embodiment, each main thread corresponds to a business processing layer in the business processing unit, which is responsible for user logic and business processing.

[0092] In an embodiment, the dynamic data also includes sensor data, and the business layer also includes a data filtering processing unit, which processes the sensor data and transmits the filtered sensor data to the business processing unit.

[0093] In an embodiment, the perception layer and the business layer support the modbus_TCP communication protocol, the modbus_ITU communication protocol or the CAN communication protocol to realize the processing of the sensor data. The configuration file is modified according to the requirements, and the binding relationship between the image acquisition unit, the visual processing unit and the business processing unit is established in the configuration file through camera_id, monitor_id and infer_id. Figure 5 Demonstrates the implementation process of driving two image acquisition devices, a sensor using CAN communication and a sensor using modbus_RTU communication. Obtain dynamic data, process the data and then perform data fusion in the business layer. That is, the business layer includes two plug-in modules and two data filter processing units. Each plug-in module processes the image of the corresponding image acquisition device, and the two data filter processing units process the sensor data collected by the corresponding sensor. During processing, each sensor is responsible for at least one main thread. The main threads are executed asynchronously, and the data is fused in a certain business processing layer. Each main thread contains a business processing layer, which is responsible for user logic and business processing. The business processing layer is actually a member function of a business processing subclass. The business logic is all performed in this member function. In order to facilitate data interaction and fusion processing between different main threads, all business processing subclasses are required to inherit from a parent class. The parent class at least contains the data queue of the business processing subclass, and the data of each business processing subclass is managed according to the id number of the business processing subclass. The business processing subclass can view all the data in the data queue, but can only modify its own data.

[0094] In the embodiment, the management layer also includes: log management and thread management, the log management is used to save the operation log and the abnormal log, the thread management is used to create and manage the thread pool and optimize the software structure.

[0095] Log management includes log classification, such as error level, warning level, reminder level and general information, etc. Log rollback function, when the space occupied by the log file exceeds a certain threshold, the current log file will be backed up, the previous backup log file will be deleted, and a new log file will be created to record the log information.

[0096] In order to facilitate thread management and improve program execution efficiency, a thread pool is created in the thread management of this embodiment. When creating a thread pool, it is necessary to set the thread data to be accommodated to prevent too many threads from affecting the performance of the system, because thread switching consumes certain system resources. At the same time, the thread pool can effectively avoid the overhead caused by frequent creation and destruction of the program during operation.

[0097] In summary:

[0098] (1) The present invention modularizes each function and independently encapsulates each module according to the GStreamer plug-in mechanism, which facilitates the testing, management and transplantation of each plug-in and increases the stability and maintainability of the software framework;

[0099] (2) In the present invention, an image can be loaded by multiple algorithm models in a read-only manner at the same time, and each model runs synchronously during reasoning, and finally the reasoning results are integrated, thereby improving the utilization rate of the image. Each model jointly references the same set of data, avoiding unnecessary system resource consumption caused by data copying;

[0100] (3) The present invention can bind the tasks of the perception layer and the business layer by modifying the configuration file. The software automatically loads the sensor devices of the perception layer according to the configuration information, constructs the inference data flow channel, loads the algorithm model, and calls the business processing class interface;

[0101] (4) The present invention can uniformly package GPUs, NPUs, and hardware codecs of mainstream manufacturers. When the software is ported between different devices, it is only necessary to specify the model of the edge device in the compilation process to complete the software compilation work, and the adaptability is good.

[0102] Example 2

[0103] An embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement a sensor data processing method based on a plug-in software architecture as provided in the above method embodiment.

[0104] Figure 6 A schematic diagram of the hardware structure of a device for implementing a sensor data processing method based on a plug-in software architecture provided in an embodiment of the present application is shown. The device may participate in or include the apparatus or system provided in an embodiment of the present application. Figure 6 As shown, the computer device 10 may include one or more processors 1002 (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1004 for storing data, and a transmission device 1006 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 6 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 6More or fewer components as shown, or with Figure 6 Different configurations are shown.

[0105] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer device 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0106] The memory 1004 can be used to store software programs and modules of application software, such as a program instruction / data storage device corresponding to a sensor data processing method based on a plug-in software architecture in an embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 1004, that is, implementing the above-mentioned method. The memory 1004 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1004 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0107] The transmission device 1006 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer device 10. In one example, the transmission device 1006 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 1006 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0108] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer device 10 (or mobile device).

[0109] Example 3

[0110] An embodiment of the present application also provides a computer-readable storage medium, which can be set in a server to store at least one instruction or at least one program related to a sensor data processing method based on a plug-in software architecture in an implementation method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement a sensor data processing method based on a plug-in software architecture provided in the above-mentioned method embodiment.

[0111] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0112] Example 4

[0113] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes a sensor data processing method based on a plug-in software architecture provided in the above various optional implementations.

[0114] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0115] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0116] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0117] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A sensor data processing method based on a plug-in software architecture, characterized in that: The software architecture framework includes, from top to bottom, a perception layer, a business layer, a management layer, and a network layer. The method includes the following steps: The perception layer drives the sensor device to obtain dynamic data of the sensor. The perception layer includes an image acquisition unit, which is connected to the image sensor device. The dynamic data at least includes an image. The business layer processes the dynamic data of the sensor, and the business layer includes at least one plug-in module, and the plug-in module is used to process the image; The plug-in module is written according to the GSteamer plug-in specification, and includes an image processing unit, a visual reasoning unit, a business processing unit, an image rendering unit, a video encoding unit, and a video push unit; The management layer performs parameter configuration management on each plug-in module of the perception layer and the business layer; The network layer is responsible for network communication and pushing the processed dynamic data of the sensor.

2. The sensor data processing method based on plug-in software architecture according to claim 1, characterized in that: After the method is executed, firstly, the configuration file is loaded in the management layer, and parameters of the configuration file are configured; The parameters in the configuration file include: (1) The model of the hardware platform on which the software runs; (2) Model and configuration parameters of image sensing equipment; (3) The name and path of the algorithm model and the hyperparameters of the algorithm; (4) Definition of the port number and data stream name of the RTSP server.

3. The sensor data processing method based on plug-in software architecture according to claim 2, characterized in that: The image acquisition unit acquires the model and configuration parameters of the image sensor device in the configuration file to collect the image; The image processing unit preprocesses the image transmitted by the image acquisition unit according to the requirements in the configuration file, and the preprocessing includes image shearing, adjusting width and height, image flipping, image filtering and morphological image processing; The visual reasoning unit loads the reasoning engine according to the model of the hardware platform on which the software in the configuration file runs, performs visual reasoning on the image after image processing, and performs accelerated calculation based on the hyperparameters of the algorithm in the configuration file to form a reasoning result; The business processing unit performs business logic processing on the inference result to obtain business layer data; The image rendering unit obtains the business layer data and renders it in the image; The video encoding unit loads an encoder according to the model of the hardware platform on which the software runs in the configuration file, and performs encoding processing on the rendered image; The video push unit pushes the encoded image according to the port number of the RTSP server and the address information of the video stream in the configuration file.

4. The sensor data processing method based on plug-in software architecture according to claim 3, characterized in that: When the business layer processes the image through a plug-in module and performs two-way reasoning, it includes at least three threads, namely a main thread, a first algorithm reasoning thread, and a second algorithm reasoning thread; The main thread is responsible for image acquisition, image processing, and business processing; The first algorithm reasoning thread and the second algorithm reasoning thread are responsible for visual reasoning.

5. The sensor data processing method based on plug-in software architecture according to claim 4, characterized in that: The execution of the three threads specifically includes the following steps: The main thread drives the image acquisition unit to acquire the image, and processes the image through the image processing unit to obtain a pre-processed image, and transmits the pre-processed image; The visual reasoning unit executes the first algorithm reasoning thread and the second algorithm reasoning thread, and during the execution, the main thread locks the first mutex lock and the second mutex lock, and the two mutex locks are locked, so that the main thread is blocked; When the first algorithm reasoning thread infers the preprocessed image and obtains a first reasoning result, an unlocking operation is performed on the first mutex lock; When the second algorithm reasoning thread infers the preprocessed image and obtains a second reasoning result, an unlocking operation is performed on the second mutex lock; When both mutexes are unlocked, the blocked main thread is unblocked; The business processing unit processes the acquired pre-processed image, the first reasoning result and the second reasoning result based on business requirements to obtain business layer data.

6. The sensor data processing method based on plug-in software architecture according to claim 4 or 5, characterized in that: Each of the main threads in the business processing unit corresponds to a business processing layer, which is responsible for user logic and business processing.

7. The sensor data processing method based on plug-in software architecture according to claim 1, characterized in that: The dynamic data also includes sensor data, and the business layer also includes a data filtering processing unit, which processes the sensor data and transmits the filtered sensor data to the business processing unit.

8. The sensor data processing method based on plug-in software architecture according to claim 1, characterized in that: The management layer also includes: log management and thread management, wherein the log management is used to save operation logs and abnormal logs, and the thread management is used to create and manage thread pools to optimize software structure.

9. A computer device, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the sensor data processing method based on the plug-in software architecture as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the sensor data processing method based on the plug-in software architecture as described in any one of claims 1 to 8.

Images