Self-processing method based on data symbolization and large model

Abstract

The invention relates to a self-processing method based on data symbolization and a large model, which comprises the following steps: in the steps of performing model training and responding to input by using the large model to realize self-processing, respectively acquiring various types of sensing data acquired in the two steps and processing result data corresponding to the acquired sensing data acquired in the model training step, the method comprises the following steps of: symbolizing a plurality of different modes of sensing data, namely multi-mode sensing data, to form training data and real-time data, and further converting the plurality of different modes of sensing data into data in a symbolic form, and the information amount contained in the symbolized data is approximately equal to the sum of the information amounts contained in the plurality of different modes of sensing data. On the basis, the multi-modal sensing data can be optimized to a great extent, so that the training complexity of a large model is reduced, the training efficiency and the response efficiency are improved, and the training cost and the implementation cost are reduced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence neural network technology, and more specifically, to a self-processing method based on data symbolization and large models. Background Art

[0002] The types, scales and applications of artificial intelligence neural networks in the prior art are mainly used to process single-modal inputs, or to process different-modal inputs in combination with models of different modalities. Regardless of whether single-modal inputs or different-modal inputs are processed, the input itself is processed, that is, the corresponding feature vectors are extracted, and then the model is trained. Therefore, in the prior art, the trained artificial intelligence model has a strong application scenario (such as text question and answer, voice question and answer, image recognition, etc. At the same time, an artificial intelligence model in different technical fields of an application scenario cannot be directly applied across technical fields), especially the input modality has serious limitations; and when it is necessary to process multiple different modal and interrelated inputs, it is necessary to set up multiple models for application scenarios corresponding to different modalities, and then associate the processing results output separately. It can be seen that the prior art has many shortcomings in achieving multi-modal processing, such as multiple model types, the need for separate training, the need for combined training, limitations in application scenarios, and limitations in input modalities.

[0003] In the prior art, the processing results output by the trained artificial intelligence model are limited by the training samples, that is, the training samples have a strong decisive effect on the processing results of the artificial intelligence model. Once the training samples are biased, it is easy to cause the trained model to have certain defects.

[0004] When the processing results of the artificial intelligence model based on the prior art are further applied, for example, the processing results are used as instructions to control other execution devices. Since the operation of the execution device is not directly controlled by the artificial intelligence model, but it is necessary to match the instructions of the execution device with various processing results, during implementation, the processing results are first output through the artificial intelligence model, and then the instructions that match the processing results are found, and the execution device then acts based on the instructions. The instructions used to control the execution device are fixed, which in turn limits the diversity of the processing results (that is, other situations in which the processing results have been matched are not allowed, and negation will not be used to control the execution device). At the same time, the output results of the artificial intelligence model are prone to security weaknesses and attacks when matching corresponding instructions.

[0005] With the "emergence" of large models, the limitations of artificial intelligence models in application scenarios have been overcome to a certain extent, especially the formation of general models under certain conditions, that is, under certain conditions, they can be applied to different technical fields. Furthermore, by combining large-scale language models with large-scale models of other modalities, a multimodal large-scale model can be obtained. A multimodal large-scale model can be used to process language and other modalities and generate multimodal outputs. For example, the PaLM-E model is a large-scale language model combined with a large-scale visual model. When deployed on a robot, requirements can be put forward through natural language, and then the robot automatically executes the requirements based on visual input.

[0006] However, when the large-scale multimodal model in the prior art processes multimodal input, each modal input is processed in its own modality. For example, after the PaLM-E model receives a demand in natural language and responds to the demand, when it needs to recognize an image for the next step of response, it still needs to process the acquired image in the form of an image (processed by the visual model ViT of the PaLM-E model). The specific process is as follows:

[0007] 1) Split the input image into fixed-size fragments (e.g. 16x16) and project each fragment into a vector as the input sequence of the Transformer;

[0008] 2) Add a special category tag (CLS) to the input sequence to indicate the category information of the image; at the same time, add a position code to each fragment to indicate the position information of the fragment in the image;

[0009] 3) The input sequence is fed into the Transformer encoder, and multi-layer self-attention and feed-forward network calculations are performed to obtain the output vector of each fragment.

[0010] 4) Take out the output vector corresponding to the category label, pass it through a linear layer and a Softmax layer, and get the category prediction probability of the image.

[0011] If more modal inputs need to be processed, the corresponding large-scale multimodal model needs to combine more large-scale models of different modalities, requires a larger data set for training, and requires greater computing power for the calculation of the large-scale multimodal model.

[0012] On the other hand, in the prior art, the collection of sensory data and the processing of the input sensory data using artificial intelligence models usually occur in a point-by-point manner, or in a repeated point-by-point manner at different locations. For example, identity confirmation or recognition usually uses a point-by-point manner, that is, recognition is performed only at one location, or repeated recognition is performed at multiple locations.

[0013] When confirming or identifying a person's identity, the method is: user password verification or biometric recognition is performed at the location where identification is required; the shortcomings are: on the one hand, there are major security risks. When the password is stolen or the face is imitated, there is a risk of identity impersonation regardless of whether the identification is performed at one location or repeated identification is performed at multiple locations; on the other hand, users are required to actively output information, and the user experience is poor.

[0014] When confirming or identifying the identity of a vehicle, the method is: feature recognition; the disadvantages are: the recognition result is easily broken, the data source is not completely reliable, the recognition technology is required to be high, but the recognition accuracy is low.

[0015] In addition, the single-point recognition method requires high-speed recognition operations at each point, which has high hardware requirements and will reduce execution efficiency as the amount of data increases.

[0016] Similarly, other application scenarios also have the shortcomings of single-point identification when they are applied in a point-by-point manner.

[0017] It can be seen that large-scale multimodal models are still in the laboratory stage. When they are applied in production, problems such as efficiency, cost, reliability, and safety need to be solved. Summary of the invention

[0018] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a self-processing method based on data symbolization and large models, which can simultaneously collect multimodal perception data and use large models for processing, so as to complete the processing of the currently input multimodal perception data; and also by symbolizing the multimodal perception data, model training is performed using training data in symbolic form, thereby achieving high efficiency, low cost, high reliability, high security and other effects of training and processing, which can be used for perception of the real world and accurately perform automatic response and processing.

[0019] The technical solution of the present invention is as follows:

[0020] A self-processing method based on data symbolization and large models includes the following steps:

[0021] 1) Model training step: using various types of data acquisition devices deployed in the working area to collect various types of corresponding types of perception data, the collected types of perception data cover objects within the monitoring range of the corresponding types of data acquisition devices; obtaining processing result data corresponding to the collected perception data;

[0022] Symbolize the various types of sensory data collected, including but not limited to the sensory data collected, and the processing result data corresponding to the sensory data collected, to form training data that describes the multimodal sensory data and the processing result data corresponding to the sensory data collected with symbols; perform model training using the training data to generate a large model;

[0023] 2) Self-processing step: Utilize various types of data acquisition devices deployed in the working area to collect various corresponding types of perception data, symbolize the collected perception data of various types, and form real-time data that describes multimodal perception data with symbols; input the real-time data into the large model to generate self-processing result data, and perform corresponding self-processing operations based on the self-processing result data.

[0024] Preferably, in step 1), taking functional scenarios as units, various types of perception data collected in different functional scenarios and training data formed by processing result data corresponding to the collected perception data are input into a neural network for generating a large model to train and generate the large model; the large model generates self-processing result data based on real-time data; wherein, there is no need to predefine the functional scenarios.

[0025] Preferably, in step 1), when there are multiple independent functional scenarios with the same functions, firstly, several of the multiple functional scenarios are automatically selected according to the selection rules, and then the model training is performed using the selected functional scenarios.

[0026] Preferably, after automatically selecting a number of functional scenarios, the symbolized training data is first output, or the symbolized training data is restored to corresponding perception data and output, and then the selected functional scenarios are prioritized manually based on the output symbolized training data or restored perception data;

[0027] And / or, manually set the starting and ending points of training data input for model training in functional scenarios, wherein the starting and ending points of training data input are respectively the state where training data input starts and ends.

[0028] Preferably, step 1) specifically comprises: utilizing various types of data acquisition devices deployed in the working area to continuously acquire various corresponding types of perception data, wherein the continuously acquired perception data of various types include perception data in chronological order acquired from objects in the working area covered by the monitoring range of the corresponding type of data acquisition device; converting the continuously acquired perception data of various types and the processing result data corresponding to the acquired perception data into training data described by symbols, inputting the training data into a neural network used to generate a large model, and training to generate the large model.

[0029] Preferably, a three-dimensional global model is obtained by three-dimensional modeling, wherein the three-dimensional global model includes a three-dimensional object model representing the object;

[0030] In step 1), various types of corresponding virtual acquisition devices are deployed in the three-dimensional global model; the virtual acquisition devices are used to perform in-model acquisition on the three-dimensional object model to obtain in-model perception data; in-model training data corresponding to the in-model perception data is input into the large model to generate in-model self-processing result data, and an automation script is generated corresponding to the in-model self-processing result data;

[0031] The self-processing result data in the model is the model-in-processing solution for the behavior, attribute, state or relationship between objects of the three-dimensional object model at the moment when the model-in-perception data is collected; the self-processing result data in the model is used as the model-in-processing result data corresponding to the model-in-perception data;

[0032] The 3D object model corresponding to the processing result data in the model is controlled by automated scripts or manually controlled to execute the model processing solution in the 3D global model, thereby optimizing the training of the large model.

[0033] Preferably, a modeling area is defined corresponding to the working area; three-dimensional modeling is performed on the real objects in the modeling area to construct a three-dimensional global model; the three-dimensional global model includes a three-dimensional object model corresponding to the real object;

[0034] Correspondingly, in step 1), corresponding to various types of data acquisition devices in the working area, corresponding various types of virtual acquisition devices are deployed in the three-dimensional global model; the virtual acquisition devices are used to perform in-model acquisition on the three-dimensional object model to obtain the perception data within the model.

[0035] Preferably, the perception data also includes but is not limited to laws and regulations, implementation standards, specifications, and custom constraints; the processing result data corresponding to the collected perception data includes but is not limited to control instructions and interactive information.

[0036] Preferably, the large model generates self-processing result data including but not limited to interaction data and / or task strategies;

[0037] When the self-processing result data is interactive data, the interactive device is controlled to output the interactive data through the matching task strategy;

[0038] When the self-processing result data is a task strategy, the task strategy is used to control the corresponding object to perform the corresponding operation.

[0039] Preferably, the task strategy includes control instructions and instruction sequences, and controls corresponding objects to perform corresponding operations based on the task strategy;

[0040] Preferably, the object includes or is provided with at least one object, and various working parameters of the object are obtained as perception data; the object responds to the various working parameters included in the processing scheme to control the object.

[0041] Preferably, the interaction data includes but is not limited to voice interaction data, image interaction data, and text interaction data, and data interaction is performed through corresponding types of interaction devices.

[0042] Preferably, a peer-to-peer computing system is set up, the peer-to-peer computing system includes multiple node devices, and there is no primary-secondary relationship between all node devices; the node devices of the peer-to-peer computing system are used to collect various types of perception data related to the target, and the target includes an object already existing in the working area and an object entering the working area;

[0043] The node device deploys an encoder for symbolizing the sensed data; for a certain node device, the encoder is used to symbolize the sensed data of various types collected and the processing result data corresponding to the sensed data collected, and the result data is processed as the original data to obtain the result data, and the result data is transmitted to other node devices; other node devices that receive the result data use the result data as one of the collected original data, and the result data of other node devices is influenced by the result data;

[0044] Based on this, multiple node devices in the peer computing system perform collaborative computing. When the result data of the collaborative computing can determine the target perception result, the target corresponding to the multimodal perception data is perceived to obtain the target perception result, wherein the target perception result is a comprehensive state of one or more combinations of the target's behavior, attributes, state, or the relationship between objects, and is represented in the result data.

[0045] Preferably, the node device is connected to and controls the corresponding execution device;

[0046] For a certain node device, a portion of the self-processing result data corresponding to the connected execution device is processed as original data to obtain result data, and the result data is transmitted to other node devices; other node devices that receive the result data use the result data as one of the collected original data, and influence the result data of other node devices through the result data;

[0047] The node devices in the peer computing system perform collaborative computing as the original data is acquired and the result data is calculated; based on the collaborative computing, the node devices respond to the self-processing result data until the result data calculated by one or more node devices matches the request content corresponding to the self-processing result data;

[0048] The self-processing result data represented by the result data is symbolized by an encoder deployed on the node device, and converted into a control symbol of the corresponding execution device, and the execution device is controlled by the control symbol; or, the output control symbol is used as a request, the result data obtained by calculation includes the corresponding control symbol, and the execution device is directly controlled by the control symbol.

[0049] Preferably, in step 1), various types of perception data related to the target are collected by using node devices of the peer computing system; for a certain node device, the collected perception data of various types and training data corresponding to the processing result data corresponding to the collected perception data are obtained;

[0050] When using training data to perform model training, the node device receives the assigned training tasks and training data, and performs corresponding model training; the training results are processed as raw data to obtain result data, and the result data is transmitted to other node devices; other node devices that receive the result data use the result data as one of the collected raw data, and influence the result data of other node devices through the result data;

[0051] The node devices in the peer computing system perform collaborative computing as the original data is acquired and the result data is calculated; based on the collaborative computing, they respond to the training results until the result data calculated by one or more node devices meets the goal of model training.

[0052] Preferably, in step 2), various types of perception data related to the target are collected by using node devices of the peer computing system; for a certain node device, real-time data formed by the collected various types of perception data is obtained;

[0053] When the large model is used to generate self-processing result data, the node device receives the assigned generation task and generates the corresponding part of the self-processing result data; the generated result is processed as the original data to obtain the result data, and the result data is transmitted to other node devices; other node devices that receive the result data use the result data as one of the collected original data, and influence the result data of other node devices through the result data;

[0054] The node devices in the peer computing system perform collaborative computing as the original data is acquired and the result data is calculated; based on the collaborative computing, the generated results are responded to until the result data calculated by one or more node devices matches the self-processing result data.

[0055] Preferably, for any node device that collects various types of perception data related to the target, the current node device receives the previous result data output by the previous node device, and in combination with the current perception data collected by the current node device, distributes and transmits it to the node device that meets the calculation requirements through the peer computing system to obtain the current result data; and transmits it to the subsequent node device in a collaborative computing manner; and so on, thereby completing the transmission of the target perception result between the node devices;

[0056] The target perception result is a comprehensive state of one or more combinations of the target's behavior, attributes, state, or relationship between objects, and is represented in the result data;

[0057] The current perception data represents the current behavior, attributes, state, or current state of the relationship between objects;

[0058] The predecessor result data represents the predecessor behavior, attribute, state or the relationship between objects of the target predecessor.

[0059] The current result data represents the current comprehensive state of the target's current behavior, attribute, state or the relationship between objects. The current comprehensive state is the combination of the current state and the previous comprehensive state.

[0060] Preferably, the self-processing result data generated by the large model is a processing solution for the behavior, attribute, state or relationship between objects within the monitoring range at the moment when the perception data is collected by various types of data acquisition devices; in step 2), the corresponding self-processing operation is performed based on the processing solution.

[0061] Preferably, when the result data calculated by one or more node devices in the peer computing system matches the request content corresponding to the self-processing result data, the result corresponding to the request content is represented in the result data output by one or more node devices according to preset conditions or the output of a pre-deployed program or a data processing model deployed on the node device; if it is determined that the current node device needs to respond to the request command, the current node device will send a control instruction to the execution device connected to the current node device according to the calculated result data; when all the execution devices connected to all the node devices that need to respond have completed the response action, the self-processing operation is completed.

[0062] Preferably, the data acquisition device includes one or a combination of an image acquisition device, an electromagnetic induction device, a temperature measuring device, a vibration frequency sensing device and a laser radar. The data collected by the above devices and the three-dimensional point cloud collected by the laser radar or the point cloud generated according to the images collected by multiple image acquisition devices are jointly calculated to obtain three-dimensional points with data; the image color, contour, line, reflectivity, motion trend, electromagnetic characteristics, temperature, temperature change trend, vibration frequency, and vibration frequency change trend based on two-dimensional perception are used as additional attributes of the corresponding three-dimensional points to form a three-dimensional point cloud with attributes; combined with electromagnetic induction, temperature laws, vibration frequency change characteristics, motion correlation, and reflectivity, the correspondence between each area of ​​the three-dimensional point cloud with attributes and each part or associated parts of the consumer's 3D appearance is determined.

[0063] Preferably, if in step 1), new training data is obtained using perception data of different working areas, different objects, and different modalities, and the large model is trained, the large model is fine-tuned to enable the large model to adapt to the perception data of different objects and different modalities.

[0064] Preferably, known encoding rules are used to encode the multimodal perception data and the processing result data corresponding to the collected perception data to obtain symbol encoding corresponding to the encoding rules, thereby realizing symbolization of the multimodal perception data and the processing result data corresponding to the collected perception data; the known encoding rules include but are not limited to one or more combinations of texts, characters, very large base sequences, known symbols of non-natural languages, and custom symbols of various natural languages.

[0065] The beneficial effects of the present invention are as follows:

[0066] The self-processing method based on data symbolization and large models described in the present invention, in the steps of model training and using the large model to respond to inputs to achieve self-processing, symbolizes the various types of perception data collected in the two steps and the processing result data corresponding to the collected perception data obtained in the model training step, that is, multi-modal perception data, to form training data and real-time data, and then transforms the perception data of multiple different modes into data in symbolic form, and the amount of information contained in the symbolized data is approximately equal to the sum of the amount of information contained in the perception data of multiple different modes. Based on this, the present invention can optimize the multi-modal perception data to a great extent, thereby reducing the training complexity of the large model, improving the training efficiency and response efficiency, and reducing the training cost and implementation cost.

[0067] In the present invention, after the technical solution is implemented, the type of data acquisition device can be known, that is, the type of perception data can be known, that is, the modality of the perception data can be known, and after determining the processing result data corresponding to the collected perception data, the inherent rules between each type of perception data, multi-modal perception data, and corresponding processing result data can be covered. Therefore, the symbolically acquired training data and real-time data have specific inherent rules, and the context logic is clear, which improves the accuracy of the large model output while ensuring the training efficiency. The training data and real-time data have a definite correspondence with the perception data they represent, and the training data and real-time data are not necessarily related to natural language vocabulary. For large models, training data and real-time data can be regarded as equivalent to the "language" used between sensors.

[0068] Based on this, when the present invention performs model training, the multimodal perception data is converted into symbolic form, and then the training data including the symbolic form is used for model training; when the model is used to respond to the input real-time data including the symbolic form to achieve self-processing, the processing of more complex multimodal perception data is achieved in a relatively simple manner. That is, in the present invention, what is input into the large model is not the perception data itself, and the large model does not need to perform corresponding processing on the perception data of different modalities. For example, when it is necessary to respond to the collected image, the large model does not directly receive the image data, but receives the symbolized image data, and then uses the large model to process the data in symbolic form to generate self-processing result data, and then performs corresponding self-processing operations based on the self-processing result data.

[0069] After the large model is deployed, the present invention continuously collects different types of perception data of all objects through the data collection device deployed in the working area, and after symbolization, the large model is used to respond to the input real-time data to complete self-processing, thereby realizing the perception of the real world. Based on the technical solution of the present invention, the deficiency that the large-scale language model of the prior art and the real world cannot form a corresponding relationship can be perfectly avoided. Without the need to combine large-scale models of other modalities, it is still possible to realize the perception of the multi-modal real world (the perception recorded in the present invention is a different definition from the perception data collection. The perception means completing the perception of the target or object corresponding to the multi-modal perception data to obtain the target perception result).

[0070] At the same time, including but not limited to laws and regulations, implementation standards, specifications, and custom constraints are also used as perception data, and thus, as a type of training data. When other training data are real data for perceiving the real world, it is ensured that the behaviors and events corresponding to the training data input into the big model are carried out under the constraints of the laws, regulations, and scientific laws of the real world, thereby avoiding the provision of false and one-sided training data and preventing the accuracy of the big model's self-processing and functional realization from being affected.

[0071] The present invention trains the model based on functional scenarios, and can use unsupervised learning such as the transformer model, so that training for different technical fields can be completed without pre-defining functional scenarios. Based on large-scale data sets, a general model under certain conditions can be obtained.

[0072] The present invention can also utilize a three-dimensional model to train or optimize the model in the three-dimensional model. The present invention can build a three-dimensional global model that simulates the real world, or use the real world as a prototype to build a three-dimensional global model that is basically the same as the real world. The three-dimensional global model contains three-dimensional object models of all corresponding virtual objects or real objects, and they move and interact with each other according to certain rules. Then, the model is trained or optimized by collecting different types of perception data from the three-dimensional object model and providing processing result data corresponding to the collected perception data. By training or optimizing the model through a three-dimensional global model, with the assistance of an automated script, it is possible to reduce manpower input, maximize time utilization, improve the efficiency of training or optimizing large models, and reduce costs.

[0073] The present invention utilizes a peer-to-peer computing system to calculate and transmit result data through collaborative computing (the result data calculated and transmitted does not include training data and real-time data in symbolic form, but is the calculation result obtained by calculating the training data and real-time data in symbolic form based on the corresponding data processing mode, that is, when transmitting the result data, the training data and real-time data in symbolic form are not directly transmitted), and then conduct model training and respond to input. In the described peer-to-peer computing system, there is no master-slave relationship between all node devices, and there is no fixed connection path between node devices. Node devices only receive the calculation results of other node devices (that is, the result data calculated by other node devices), and send out the result data of their own calculations. Model training and / or responding to inputs do not rely on a single node device for processing and control, but are completed through collaborative computing and result data transmission by multiple node devices in the peer-to-peer computing system.

[0074] Furthermore, the model training and response to input of the present invention do not rely on point-to-point occurrence, and the computing functions are distributed throughout the entire peer-to-peer computing system, which not only reduces the hardware and software requirements of single-point computing, has high execution efficiency, and greatly improves the ability to resist attacks; but also presents relatively symmetrical information between node devices, and is immune to the problem of illegal data tampering. Even if a single node device is physically cracked and the data it sends is tampered with, because the calculation of the peer-to-peer computing system is a complex calculation with ultra-high redundancy and ultra-multi-dimensional verification, the tampering of the data sent by a single node device does not affect the calculation results of the peer-to-peer computing system, and the fault and the tampered node device can be quickly located to ensure the credibility of the calculation results of the peer-to-peer computing system. DETAILED DESCRIPTION

[0075] The present invention is further described in detail below with reference to the embodiments.

[0076] In order to solve the shortcomings of the prior art such as low efficiency, high cost, poor reliability and low security, the present invention provides a self-processing method based on data symbolization and a large model, which symbolizes different types of perception data used as training samples and as input, that is, perception data of multiple different modalities collected separately, trains to generate a large model, and uses the large model to process the symbolized perception data of multiple different modalities, so as to generate self-processing result data corresponding to the perception data of multiple different modalities.

[0077] The self-processing method based on data symbolization and large model described in the present invention comprises the following steps:

[0078] 1) Model training step: using various types of data acquisition devices deployed in the working area to collect (preferably, continuously collect) various corresponding types of perception data, and the collected perception data of various types covers objects within the monitoring range of the corresponding type of data acquisition device; obtaining processing result data corresponding to the collected perception data (i.e., different types of perception data collected for objects in the working area, corresponding data outputs, such as control instructions that need to be issued to the execution device, interactive information that needs to be output for specific personnel, and other processing solutions, and the processing result data corresponding to the collected perception data includes but is not limited to control instructions, interactive information, etc.);

[0079] Symbolize the various types of sensory data collected as described above (since the present invention is implemented in the real world, and the real world operates based on various legal and human constraints, in specific implementation, the sensory data also includes but is not limited to laws and regulations, implementation standards, specifications, and user-defined constraints, and thus, the self-processing result data output by the big model can comply with the operating rules of the real world) and the processing result data corresponding to the collected sensory data to form training data that describes multimodal sensory data and the processing result data corresponding to the collected sensory data with symbols; use the training data to train the model and generate a big model;

[0080] 2) Self-processing step: using various types of data acquisition devices deployed in the work area to collect various corresponding types of perception data, symbolizing the collected perception data of various types to form real-time data that describes multi-modal perception data with symbols; inputting the real-time data into the big model to generate self-processing result data, and performing corresponding self-processing operations based on the self-processing result data. The self-processing result data generated by the big model includes but is not limited to interaction data and / or task strategies, and correspondingly, the self-processing operations include but are not limited to outputting interaction data for human-machine interaction or machine-machine interaction and / or controlling objects corresponding to task strategy control.

[0081] Specifically, when the self-processing result data is interactive data, the interactive device is controlled to output the interactive data through the matching task strategy; when the self-processing result data is a task strategy, the corresponding object is controlled by the task strategy to perform the corresponding operation. Wherein, the control instructions and instruction sequence contained in the task strategy are used to control the corresponding object to perform the corresponding operation based on the task strategy; wherein, the corresponding object controlled based on the task strategy is at least one object whose sensed data is collected and controlled, or at least one object is set (such as at least one controlled object set on an object whose sensed data is collected but not controlled), and various working parameters of the object are obtained as sensed data; the object responds to the various working parameters contained in the processing scheme to control the object. The interactive data includes but is not limited to voice interactive data, image interactive data, and text interactive data, and data interaction is performed through interactive devices of corresponding types.

[0082] The present invention is based on perception data, and the perception data collected from the real world (the perception of the real world described in the present invention is different from the definition of perception data collection, and the perception means completing the perception of the target or object corresponding to the multimodal perception data to obtain the target perception result) are all objective parameters, and different types of perception data and their relationships are subject to the laws of the real world; furthermore, the collected perception data is used as input, and with sufficient data training, the large model generated by training is used to make predictions or calculate outputs, which can achieve more accurate adaptability.

[0083] In the present invention, after the technical solution is implemented, the type of data acquisition device can be known, that is, the type of perception data can be known, that is, the modality of the perception data can be known, and after determining the processing result data corresponding to the collected perception data, the inherent rules between each type of perception data, multi-modal perception data, and corresponding processing result data can be included. Therefore, the training data and real-time data obtained by symbolization have specific inherent rules, and the context logic is clear. The symbolization can be carried out by simple coding or complex multiple dimensional coding. Under the constraints of the rules of the real world, no matter which symbol is used for expression, it can cover specific inherent rules and the context logic is clear. Although in almost all cases, it is not suitable for humans to read in the habit of natural language, it is particularly suitable for machine learning.

[0084] The symbolization described in the present invention is different from the technical means of extracting features used in machine learning or deep learning in the prior art in that the feature extraction performed by machine learning or deep learning in the prior art is characterized, and the characterization of deep learning can learn some abstract and high-level representations from the data to describe the structure and regularity of the data; for example, in image recognition, deep learning can extract features such as edges, textures, shapes, objects, etc. from pixel-level inputs, and these features rely on numerical matrices or tensors to represent data; for another example, in natural language processing, deep learning usually uses word embedding or sentence embedding to represent text, and these embeddings are real vectors. The symbolization described in the present invention will include but is not limited to the transcoding of the various types of sensory data collected, the processing result data corresponding to the collected sensory data to form sensory data that describes multimodal information in symbols, and the training data of the processing result data corresponding to the collected sensory data; and the symbolization occurs before the input of the neural network used to generate the large model, and compared with the characterization performed after the training sample is input into the neural network, due to the different timing, location, and role played in the technical solution, it should be clearly distinguished.

[0085] It can be seen that the features extracted by traditional algorithms or deep learning in the prior art can be regarded as the formal transformation of image data and text data, but they do not clearly define and use symbols. Instead, they rely on numerical matrices, tensors or real vectors to represent data, rather than symbols or logical expressions. Moreover, the numerical representation obtained by characterization is usually distributed, continuous, and high-dimensional, rather than discrete, sparse, and low-dimensional, and therefore is very different from the traditional cognitive symbol system.

[0086] Therefore, although the characterization of deep learning can help understand the intrinsic structure and meaning of data, it does not explicitly use and operate symbols, but relies on numerical calculations and optimization, which is completely different from the symbolization described in the present invention. The symbolization described in the present invention is not to extract features. The training data and real-time data formed by the symbolization have a certain correspondence with the perception data they represent. The training data and real-time data do not necessarily have to be related to natural language vocabulary. For large models, training data and real-time data can be regarded as equivalent to the "language" used between sensors, which is a way to interact across modalities for perception data of different modalities.

[0087] In contrast, the symbolization described in the present invention can be implemented as the use of clearly defined symbol types to represent the object at the moment when the perception data is collected, based on one or more combinations of the object's behavior, attributes, state or relationship between objects reflected in the perception data. That is, the symbolization described in the present invention is different from representing data in the form of numerical matrices, tensors or real vectors.

[0088] The difference between the symbolization described in the present invention and the characterization of the prior art is that the timing of the implementation is different, and correspondingly, the roles played in the respective technical solutions are different. The characterization of the prior art, for example, the characterization of deep learning generally occurs in the middle layer or the last layer of the convolutional neural network (CNN). These layers can extract high-dimensional feature vectors from the input image or text for subsequent classification, regression, matching and other tasks. The features extracted by different layers have different levels of abstraction and semantic information. Generally speaking, the features closer to the output layer are more abstract, and the features closer to the input layer are more specific.

[0089] The symbolization of the perception data by the present invention occurs before it is input into the neural network used to generate the large model, that is, the symbolization does not necessarily require the use of a neural network. Furthermore, there is no difference in the abstractness or specificity of the features caused by the different locations where the characterization occurs as in the above-mentioned prior art.

[0090] Regarding traditional algorithms, the main difference between deep learning feature extraction and traditional methods is that deep learning is a data-driven approach that uses neural networks to automatically learn feature representation and abstraction, while traditional methods are manually designed to extract features using some fixed operators or patterns. The disadvantage of deep learning is that it requires a large amount of data and computing power, and its interpretability is weak. The disadvantage of traditional algorithms is that they have poor generalization and adaptability, and need to be specially designed for different tasks.

[0091] It can be seen that, similarly, the characterization of the traditional algorithm is different from the symbolization described in the present invention, and the timing of the symbolization described in the present invention is different, and the role played in each technical solution is different.

[0092] Based on this, when the present invention performs model training, the multimodal perception data is converted into symbolic form, and then the training data including the symbolic form is used for model training; when the model is used to respond to the input real-time data including the symbolic form to achieve self-processing, the processing of more complex multimodal perception data is achieved in a relatively simple manner. That is, in the present invention, the input to the large model may not be the perception data itself, and the large model does not need to perform corresponding processing on perception data of different modalities. For example, when it is necessary to respond to the collected image, the model is not trained with image data as training data, but with symbolized image data as part of the training data, and then the large model is used to process the data in symbolic form to generate self-processing result data, and then the corresponding self-processing operation is performed based on the self-processing result data.

[0093] After the large model is deployed, the present invention continuously collects different types of perception data of all objects through the data collection device deployed in the working area, and after symbolization, the large model is used to respond to the continuously input real-time data to complete self-processing, thereby realizing the perception of the real world. Based on the technical solution of the present invention, the deficiency that the large-scale language model of the prior art and the real world cannot form a corresponding relationship can be perfectly avoided, and the perception of the multi-modal real world can still be realized without the need to combine large-scale models of other modalities.

[0094] In step 1) of the present invention, the training of the large model is carried out in units of functional scenarios; wherein the functional scenarios are application scenarios for realizing certain functions, such as robot automatic vehicle moving, automatic garbage processing, automatic fire fighting, etc., which can realize a specific event. Since the present invention uses symbolized training data for model training, when the training data contains perception data and corresponding processing result data, it is not necessary to predefine the functional scenarios, that is, the model training can be realized by unsupervised learning. In specific implementation, the occurrence and end of a specific event are used as the starting and ending points of the scenario, and the training data formed by the various types of perception data collected in different functional scenarios and the processing result data corresponding to the collected perception data are input into the neural network for generating the large model, and the large model is trained and generated. Furthermore, in step 2), the large model can generate self-processing result data based on real-time data; that is, the large model has universality, and after the training is completed, it is not necessary to carry out special training for a specific functional scenario. It only needs to be optimized and trained when necessary, such as when the self-processing result data output for the uninvolved functional scenario cannot meet the demand, so as to realize the optimization of the large model. Specifically, if in step 1), new training data is obtained using perception data of different working areas, different objects, and different modalities, and the large model is trained, the large model is fine-tuned to enable the large model to adapt to the perception data of different objects and different modalities, which is the model optimization (i.e., the fine-tuning process).

[0095] In order to take into account both the convenience and efficiency of model training, in step 1), when there are multiple independent functional scenarios with the same functions, for example, there are N functional scenarios of automatic handling and parking, all of them are used for model training. First, according to the selection rules, several of the multiple (such as N) functional scenarios are automatically selected, and then the selected several (such as M of the N functional scenarios) functional scenarios are used for model training. Then, more suitable scenarios can be screened out for model training to achieve the training expectations. Among them, the multiple independent functional scenarios with the same functions are not a functional scenario copied into multiple ones, but a plurality of independent, similar functional scenarios with the same functions are set. For example, there are multiple functional scenarios where robots automatically move vehicles. The multiple functional scenarios have the same function, that is, "there are vehicles parked in a specific area, and the robot automatically moves the vehicle to other specific areas", and the multiple functional scenarios are independent and not completely the same.

[0096] In order to facilitate manual or automatic identification and sorting of multiple functional scenarios to improve the efficiency of model training, in specific implementation, further, after automatically selecting several functional scenarios, the symbolized training data is first output, or the symbolized training data is restored to corresponding perception data and output, and then the selected functional scenarios are prioritized based on the output symbolized training data or restored perception data through manual or automatic identification; then, the functional scenarios with the best training effect are given priority for model training, so that a large model with a certain degree of usability can be obtained in the shortest possible training time.

[0097] Furthermore, in order to make the degree of model training controllable, in specific implementation, the start and end points of training data input for model training can be set manually, where the start and end points of training data input are the state of training data input start and end, that is, for example, the time when training data input starts and ends can be set. In addition, when the output accuracy of the large model is better than the threshold, optimization training can be performed to ensure output accuracy.

[0098] In the present invention, step 1) is more specifically to continuously collect various types of corresponding types of sensory data using various types of data collection devices deployed in the working area, and the continuously collected sensory data of various types include sensory data collected from objects in the working area covered by the monitoring range of the corresponding type of data collection device in a time sequence; the continuously collected sensory data of various types and the processing result data corresponding to the collected sensory data are converted into training data described by symbols, and the training data are input into the neural network used to generate the large model, and the large model is trained and generated. Among them, when symbolizing the continuously collected sensory data and the processing result data corresponding to the collected sensory data, it can be implemented as a single collection, a single symbolization, or multiple connection collections and collective symbolization, wherein the sensory data and processing result data for collective symbolization can be selected in random quantity, fixed quantity cycle selection, or other set rules (such as selecting nodes with value changes, or selecting cycles with cyclic changes). The implementation methods of different quantity selections are all based on the selection criteria of whether the corresponding type of sensory data can achieve improved symbolization and improved model training efficiency.

[0099] In the present invention, when the trained large model is used to output the self-processing result data, it can theoretically cover all prominent conditions in the working area, including the target effect corresponding to the self-processing result data; but in order to provide a redundant guarantee scheme and ensure the accuracy and safety of the self-processing of the present invention, in step 2), the collected perception data is symbolized, and the real-time data at the starting moment is obtained. Based on the large model, the corresponding complete task self-processing result data is obtained; following the execution of the previous self-processing operation, the perception data is collected in real time and symbolized, and the real-time data at the current moment is obtained. Based on the large model, the corresponding remaining task self-processing result data is generated. Wherein, the complete task self-processing result data includes the target interaction data and / or target task strategy corresponding to the moment when the real-time data at the starting moment is collected, and the remaining task self-processing result data includes the verification interaction data and / or verification task strategy corresponding to each moment when the real-time data at the current moment is collected.

[0100] When the verification interaction data and / or verification task strategy contained in the remaining task self-processing result data are different from or contradictory to the target interaction data and / or target task strategy contained in the complete task self-processing result data, the complete task self-processing result data is judged to be invalid, and then, the remaining task self-processing result data is used as the replacement self-processing result data, and the corresponding self-processing operation is performed based on the replacement self-processing result data. Based on this, the self-processing result data is automatically optimized or automatically adjusted. Among them, after the remaining task self-processing result data is generated, it does not affect the self-processing based on the complete task self-processing result data. Before the remaining task self-processing result data replaces the complete task self-processing result data, it is only used to verify whether the complete task self-processing result data is valid for the perception data at the current moment.

[0101] Furthermore, the self-processing result data is used as training data to optimize the large model.

[0102] In order to improve the efficiency of model training and model optimization, the present invention also uses software to assist in model training and model optimization. Specifically, a three-dimensional global model is obtained through three-dimensional modeling, and the three-dimensional global model includes three-dimensional object models representing objects (three-dimensional object models move and interact with each other according to certain rules); wherein the virtual area corresponding to the three-dimensional global model corresponds to the working area of ​​the real world, and the three-dimensional object model, as a virtual object, corresponds to the object in the real world.

[0103] Correspondingly, in step 1), various types of corresponding virtual acquisition devices (corresponding to data acquisition devices in the real world) are deployed in the three-dimensional global model; the virtual acquisition devices are used to perform in-model acquisition on the three-dimensional object model to obtain in-model perception data; in-model training data corresponding to the in-model perception data is input into the large model to generate in-model self-processing result data, and an automation script is generated corresponding to the in-model self-processing result data;

[0104] The self-processing result data in the model is the model-in-processing solution for the behavior, attribute, state or relationship between objects of the three-dimensional object model at the moment when the model-in-perception data is collected; the self-processing result data in the model is used as the model-in-processing result data corresponding to the model-in-perception data;

[0105] The 3D object model corresponding to the processing result data in the model is controlled by automated scripts or manually controlled to execute the model processing solution in the 3D global model, thereby optimizing the training of the large model.

[0106] Since the three-dimensional global model can be separated from the real world and accurately set based on expectations, compared with the real world, the training data and functional scenarios are completely controllable, and the interference of emergencies on the training process can be almost avoided, including the obstruction of buildings and trees that may exist in the real world, or the selection of the location of the data acquisition device, etc., which have to be compromised or avoided according to the actual situation in the real world; and the perception data and processing result data collected within the model are the same as the perception data and processing result data collected in the real world, are the same type of data, and play the same role; and it is capable of long-term, continuous, and massive data training, greatly shortening the time span of model training.

[0107] In the present invention, in order to ensure that the real world can be completely replicated when the three-dimensional global model is used for model training and model optimization, in specific implementation, a modeling area is demarcated corresponding to the working area; three-dimensional modeling is performed on the real objects in the modeling area to construct a three-dimensional global model; the three-dimensional global model includes a three-dimensional object model corresponding to the real object;

[0108] Correspondingly, in step 1), corresponding to various types of data acquisition devices in the working area, corresponding various types of virtual acquisition devices are deployed in the three-dimensional global model; the virtual acquisition devices are used to perform in-model acquisition on the three-dimensional object model to obtain the perception data within the model.

[0109] Furthermore, when implementing long-term, continuous, and massive data training, it can also ensure that the model training and model optimization performed using the three-dimensional global model are nearly consistent with the training results performed in the real world.

[0110] In the present invention, the three-dimensional modeling, three-dimensional global model, and three-dimensional object model are three-dimensional models formed by image data through modeling, rendering, and other processes using a three-dimensional engine. When the three-dimensional global model is used for model training, the three-dimensional global model can be automatically operated through a script. As the three-dimensional global model runs, the model is collected, self-processing result data in the model is generated, and an automated script is generated. The model training process may not be shown throughout the process. When the implementation requirements require the model training process to be shown, especially the operation process of the three-dimensional global model, the three-dimensional global model can be visualized for users to view or check.

[0111] In this embodiment, the symbolization can be implemented as a simplified transcoding format result, specifically: simplifying and transcoding the collected multimodal perception data based on the transcoding rules to obtain a matching perception simple code; constructing a perception multidimensional matrix for the perception simple codes corresponding to all the current perception data collected, and then simplifying and transcoding the perception multidimensional matrix again, that is, completing the symbolization. Correspondingly, in order to further effectively simplify the complex original data and facilitate the transcoding of the multidimensional matrix, in this embodiment, all perception simple codes and the perception multidimensional matrices they constitute are exhaustively enumerated, and a certain simple code value is assigned to each perception multidimensional matrix; an M-bit N-ary sequence is constructed, and each bit of the N-ary sequence is assigned one of K different symbols; each specific value of the M-bit N-ary sequence is matched one by one with each simple code value; wherein the value of M×N is greater than the number of simple code values. In the specific implementation, the setting position, time, type and quantity of the data acquisition device and other parameters can be clearly defined, and then it is very simple to exhaustively list all the perception simple codes and the perception multidimensional matrix they constitute, and the amount of data is not large. For example, if there are 100 million possibilities, 10,000 different symbols can be used to form a ten thousand base sequence, that is, a 2-bit ten thousand base sequence, such as X 1 X 0 , where X 1 , X 0 can be assigned one of 10,000 different symbols, and thus, X 1 X 0 There are 100 million values ​​in total, each corresponding to one of the 100 million possibilities. If a larger amount of information is required during implementation, the number of digits in the permafrolic sequence can be increased according to the implementation requirements, such as X 3 X 2 X 1 X 0 , there are 10 in total 16 Each value corresponds to one of 100 trillion possibilities.

[0112] Furthermore, in order to improve the efficiency of simplified transcoding, in this embodiment, the original data such as the perception data that needs to be simplified transcoded and the corresponding result processing data can be divided into multiple data blocks, and simplified transcoding is performed separately to obtain matching symbol codes, and then multiple data block symbols are obtained; each data block symbol is then used as the original data and simplified transcoded again to achieve nested transcoding, and finally obtain training data and real-time data.

[0113] In this embodiment, the transcoding rules are as follows: different types of perception data belonging to different objects collected by different types of data acquisition devices are predefined into several levels, and all levels are divided into several level segments; a corresponding level segment identifier is assigned to each level segment, and a predefined type identifier is assigned to each type of perception data, and the type identifier + level segment identifier constitutes the original code parameter; the difference between the level of the perception data of the type being sent and the first value of the level segment to which it belongs is set as a correction value.

[0114] Arrange multiple groups of original code parameters that need to be sent simultaneously into an original code parameter string, arrange the correction values ​​into a correction value string, convert the original code parameter string into a matching simple code value, convert the correction value string into a matching correction code, and send the simple code value + correction code to subsequent node devices.

[0115] After the subsequent node device receives the simple code value + correction code, it translates the simple code value back to the original code parameter string, and disassembles the original code parameter string into several original code parameters corresponding to the current node device; translates the correction code back to the correction value string, and disassembles the correction value string into several correction values ​​corresponding to the current node device; adds the correction value to the first value of the level segment corresponding to the original code parameter, and restores the level of the perception data of the type sent by the current node device.

[0116] When it is necessary to restore the symbolic form (including training data or self-processing result data in symbolic form) that satisfies the transcoding rules, that is, to de-symbolize, the inverse operation can be performed based on the transcoding rules.

[0117] In order to achieve the purpose of simplifying the transmission content, the word length of the simplified code value is smaller than the word length of the original code parameter string, and the word length of the correction code is smaller than the word length of the correction value string. In the present invention, the number of bits of the simplified code value and the correction code is determined according to the number of exhaustively enumerated original code parameter strings and correction value strings, so that the number of original code parameter strings and correction value strings matched by the simplified code value and the correction code is not less than the number of exhaustively enumerated original code parameter strings and correction value strings.

[0118] As an implementation method that is easy to implement, in the present invention, if the number of exhaustive original code parameter strings and correction value strings is less than 256, the simplified code value and the correction code are both represented by ASCII code, and each original code parameter string and correction value string are matched with one ASCII code. One ASCII code includes an 8-bit binary number combination, representing 256 possible characters. Each character can be used to represent an original code parameter string or a correction value string, and an original code parameter string + correction value code represents a combination of a determined number of multiple sets of parameters transmitted in parallel each time.

[0119] Furthermore, if the number of exhaustively enumerated original code parameter strings and correction value strings is greater than the number of ASCII codes, each original code parameter string and correction value string matches at least two ASCII codes, and multiple ASCII codes are arranged to form an ASCII code string. Specifically, the ASCII code string includes several ASCII codes, and the number of original code parameter strings and correction value strings matched by the ASCII code string is P = 256 n , where n = 1, 2, 3, ... That is, in order to ensure that the number of possible characters included in the ASCII code string is greater than the number of exhaustively enumerated original code parameter strings and modified value strings, in implementation, the number of digits of the ASCII code string is determined according to the number of exhaustively enumerated original code parameter strings and modified value strings, so that the number of original code parameter strings and modified value strings matched by the ASCII code string is not less than the number of exhaustively enumerated original code parameter strings and modified value strings.

[0120] In the above embodiment, the ASCII code is used only as an embodiment for convenient explanation of the present invention. The simple code value and the correction code described in the present invention can also be other binary codes with a preset number of bits (such as ASCII code is 8 bits, 1 byte), and 2 bits are formed according to the number of bits of the binary code. n Number of characters.

[0121] For example, the performance status of three devices a, b, and c is sent as an example. The performance status of devices a, b, and c are collected. The parameters of devices a, b, and c have 30 levels from 0 to 29. Exhaustive original code parameter strings can obtain 27,000 combinations. The data of the performance status of the three devices obtained is 9 characters, such as a15b27c18. In computer storage and transmission, these 9 characters need to occupy 9 8-bit binary codes, that is, a total of 9 bytes (72 bits) of information need to be transmitted.

[0122] In this embodiment, the parameters of each device are graded, such as being divided into six sections: a0-4, a5-9, a10-14, a15-19, a20-24, and a24-29. The same is true for devices B and C. A parameter section identifier is generated according to which level section the parameter level to be sent falls into. In this way, there are 216 combinations of parameter sections for the three devices, which can be represented by only one ASCII code. A parameter correction table is predefined in the current node device. The correction value represents the value of the parameter level floating on the level section to which it belongs, that is, the difference with the first value of the level section to which it belongs. For example, the correction values ​​of a15b27c18 are 0, 3, and 3 respectively, and the correction value string is 033. The parameter correction table is an exhaustive combination of correction value strings. In this embodiment, there are 125 correction value strings, which can also be represented by one ASCII code. It can be seen that the present invention can use two bytes to transmit 9 bytes of information. For example, the combination a15b27c18 corresponds to the character "p" in the transcoding mapping table and the character "k" in the parameter correction table. "pk" is used as the original data to participate in the peer calculation, and the result data obtained through the calculation is transmitted to the subsequent node device. The subsequent node device presets a transcoding mapping table and parameter correction table or decoder with the content completely consistent with the current node device. It is found in the transcoding mapping table or determined through the decoder that "p" corresponds to the level segments of a15-19, b24-29, and c15-19, and it is found in the parameter correction table or determined through the decoder that "k" corresponds to 0, 3, and 3. The correction value is added to the first value of the level segment to restore a15b27c18.

[0123] Through the above process, 9 bytes of information can be transmitted using 2 bytes, thereby achieving the purpose of the present invention.

[0124] The symbolization described in the present invention can also be combined with multi-dimensional matrix coding, then the above example can be evolved into: the xyz coordinates of object A observed by camera 1 can be represented by abc (equivalent to recording data of the object in only three dimensions, and the same applies to multi-dimensional data), and then after simplified transcoding, the data of A in the multi-dimensional matrix is ​​pk. In the multi-dimensional matrix output by the node device, the observation results of camera 1 are A pk B mw, and the observation results of camera 2 are C ab D ef. The data of other sensors are similar, and finally a multi-dimensional matrix is ​​formed.

[0125] Based on the simplified transcoding, the result data obtained by calculation by the node device of the peer computing system provided by the present invention can be represented in a multidimensional matrix as the data of the multidimensional matrix, the state value of the multidimensional matrix data or the result of simplified transcoding of the multidimensional nested matrix data according to implementation requirements.

[0126] The simplified transcoding is one implementation method of symbolization. During specific implementation, a suitable encoding method can be selected for symbolization according to the implementation situation.

[0127] In specific implementation, the traditional single-point recognition method can be used to collect perception data of the targets in the working area (including objects already existing in the working area and objects entering the working area) at the set position, and identify based on the perception data. For example, based on the collected images, image recognition can be used to determine what objects are contained in the image, the behavior, attributes, status of the objects, or the relationship between the objects. For example, based on the collected radio frequency signals, the identity of the corresponding associated objects can be determined, and so on. The collaborative computing based on the peer computing system provided by the present invention can also be used to complete the perception of the target. The present invention sets up a peer computing system, which is based on collaborative computing, does not rely on single-point identification, and distributes all computing functions in the peer computing system, thereby reducing the software and hardware requirements of single-point computing, having high execution efficiency, and greatly improving the ability to resist attacks. The information between node devices is relatively symmetrical, and can be immune to the problem of illegal data tampering. Even if a single node device is physically cracked and the data it sends is tampered with, because the calculation of the peer computing system is a complex calculation with ultra-high redundancy and ultra-multi-dimensional verification, the tampering of the data sent by a single node device does not affect the calculation results of the peer computing system, and the fault and the tampered node device can be quickly located, thereby ensuring the credibility of the calculation results of the peer computing system. Furthermore, the contradiction between data sharing and information security between departments can be resolved.

[0128] The result data transmitted between node devices can be the processing result of information rather than the information itself, so the collected original data (i.e., perception data) does not need to be stored. The node device only receives the calculation results output by other node devices and sends out its own calculation results. The amount of information contained in a single calculation result is not enough to restore any event and target information. The calculation results on the entire peer-to-peer computing system, the elements of the multi-dimensional data matrix, and the correspondence between the physical space and facilities must be jointly calculated to obtain a definite result. Collaborative computing is less dependent on the information transmitted by a few node devices, and can fundamentally change the nature of traditional information technology that is sensitive to single-point security.

[0129] The peer-to-peer computing system includes multiple node devices, and there is no master-slave relationship between all node devices, forming a decentralized network and computing architecture. Different from the traditional information-based single-point convergence computing mode, there is no fixed, preset path relationship in the direction of data transmission between the node devices of the present invention. In the peer-to-peer computing system described in the present invention, for a certain node device, the collected original data is processed to obtain result data, and the result data is propagated to other node devices; other node devices that receive the result data use the result data as one of the collected original data, and the result data of other node devices is affected by the result data. For the convenience of expression, the aforementioned "a certain node device" is referred to as the "current node device" and the "other node devices" are referred to as "subsequent node devices". Then one of the effects is that the result data calculated by the subsequent node device is not completely determined by the original data collected by itself, but is jointly determined with the result data output by the current node device; wherein, the result data output by the current node device may change the data processing model and parameters used by the subsequent node device to calculate the result data, thereby affecting the result data of the subsequent node device. For example, if the result data output by the current node device is correlated with the original data collected by the subsequent node device, it is necessary to consider the impact of the result data output by the current node device on the accuracy of the result data of the subsequent node device; specifically, for the perception of a specific target, if the result data obtained by calculation is only based on the original data collected by the subsequent node device, it can only reflect the real-time (including real-time location and time) single-point result judgment of the target within the perception range of the subsequent node device; and the result data output by the current node device reflects the direct perception data and result judgment about the target at other locations and other times, or other indirectly related perception data and result judgment, which helps to improve the accuracy and comprehensiveness of the result data of the subsequent node devices, including superposition calculations of the same dimension and correlation references of different dimensions.

[0130] Since there is no master-slave relationship between the node devices in the peer-to-peer computing system, point-to-point transmission can be performed between the node devices. Therefore, for the calculation result corresponding to a certain perception data of a certain target reflected in the result data output by a certain node device, the information is relatively symmetrical in other node devices that receive the result data; other node devices use the received result data as input, combine the perception data of their own sensors, and calculate their own result data. Their own result data naturally covers the received result data and the information reflected by their own sensors, and transmits it to other node devices in the next layer. Therefore, for a certain perception data of a certain target, the information is relatively symmetrical in all node devices, which can be immune to the influence of the calculation process and calculation results of a single node device being tampered with and forged, which leads to the result data. At the same time, it also becomes a means to discover faulty or tampered node devices and node devices with non-compliant performance, fundamentally solving the fundamental hidden dangers of traditional information technology, that is, false information, forged information, and erroneous information caused by information asymmetry, which then become the entry point for fraud and network attacks, and complex comprehensive applications have poor accuracy, too long time consumption, poor credibility, and poor adaptability. Therefore, it can truly become the information infrastructure for comprehensive management of larger areas and the infrastructure of the digital economy. The present invention is different from the blockchain technology that still uses each node to independently calculate, determine the results, and pay attention to the technical solution of original data storage. The present invention focuses on peer-to-peer collaborative computing between node devices. Through peer-to-peer collaborative computing, each node device can adjust its own data processing model (i.e., the algorithm for calculating the result data) and parameters when processing data. The adjustment is the feedback of all node devices on their own adjustments, thereby turning the calculations of all node devices into a whole. Each node device no longer completes the calculation independently, but all node devices complete the calculation together. After the data processing model of the node device is adjusted, it is an objective adjustment that will affect the next data processing.

[0131] The node device is provided with a data acquisition device (in specific implementation, it may include but is not limited to one or more of an image acquisition device, an audio acquisition device, a temperature measurement device, a vibration frequency sensing device, a laser radar, a chemical sensor, and an electromagnetic induction device) and a computing module, and uses the node device of the peer computing system to collect various types of perception data related to the target. Among them, the data acquisition device includes at least one type of sensor for collecting perception data of different corresponding types. The computing module calculates and obtains the result data based on the data processing model. Node devices set at different collection positions (i.e., located at different physical installation positions) collect at least one point sample of the target, and the point sample is the perception data of the corresponding sensor type. The targets include objects already existing in the working area and objects entering the working area.

[0132] In the present invention, the node device deploys an encoder for symbolizing the perception data, which is used to symbolize the various types of perception data collected and the processing result data corresponding to the collected perception data. Alternatively, the multimodal perception data and the processing result data corresponding to the collected perception data are encoded using known encoding rules to obtain symbol encoding corresponding to the encoding rules, thereby realizing the symbolization of the multimodal perception data and the processing result data corresponding to the collected perception data; then the node device obtains the symbolized training data. The known encoding rules include, but are not limited to, one or more combinations of texts, characters, super-large base sequences, known symbols of non-natural languages, and custom symbols of various natural languages.

[0133] In this embodiment, for a certain node device, an encoder is used to symbolize the various types of sensory data collected and the processing result data corresponding to the sensory data collected, and the result data is processed as raw data to obtain result data, and the result data is propagated to other node devices; other node devices that receive the result data use the result data as one of the collected raw data, and influence the result data of other node devices through the result data. Based on this, multiple node devices in the peer computing system perform collaborative computing, and when the result data of the collaborative computing can determine the target perception result, the target corresponding to the multimodal sensory data is perceived to obtain the target perception result, wherein the target perception result is a comprehensive state of one or more combinations of the target's behavior, attributes, state, or relationship between objects, and is represented in the result data.

[0134] The present invention uses symbols to represent various types of perception data about the target, simplifying the amount of information corresponding to the perception data. It is no longer necessary to rely on a large and slow database to record massive targets and complex spatiotemporal correlations. The result data output by any node device each time contains the event sequence and details of each dimension over a fairly long period of time covered by the peer computing system.

[0135] The present invention realizes the transmission of target perception results between node devices by transmitting result data between node devices (in the present invention, the result data calculated and transmitted does not include training data and real-time data in symbolic form, but is the calculation result obtained by calculating the training data and real-time data in symbolic form based on the corresponding data processing mode, that is, when transmitting the result data, the training data and real-time data in symbolic form are not directly transmitted); wherein, the target perception result is a comprehensive state of one or more combinations of the behavior, attribute, state or relationship between objects of the target, and is characterized by the result data. Specifically, for any node device that collects various types of perception data related to the target, the current node device receives the previous result data output by the previous node device, and combines the current perception data collected by the current node device, distributes and transmits it to the node device that meets the calculation requirements through the peer computing system for calculation, and obtains the current result data; and transmits it to the subsequent node device in a collaborative computing manner; and so on, thereby completing the transmission of the target perception result between node devices; and so on, thereby completing the transmission of the target perception result between node devices. That is, in each transmission of result data by a node device, it combines the received data of various types with the current perception data collected by itself, performs calculations based on the data processing model, obtains result data, and then sends the result data to other node devices in the next layer (in most cases, including multiple subsequent node devices, each subsequent node device corresponds to multiple current node devices, and each node device corresponds to multiple preceding node devices; the logical principle of the entire peer-to-peer computing system is the same), that is, the node device completes the transmission of a target perception result.

[0136] In the present invention, the current perception data represents the current state of the target's current behavior, attribute, state or the relationship between objects; the preceding result data represents the preceding through-comprehensive state of the relationship between the target's preceding behavior, attribute, state or objects; the current result data represents the current comprehensive state of the target's current behavior, attribute, state or the relationship between objects, and the current comprehensive state is the combination of the current state and the preceding through-comprehensive state. For example, in the case of a simple single-dimensional single-target, the preceding result data output by the preceding node device is ab (i.e., the result data output by the upper-layer node device of the preceding node device is a, the perception data of the preceding node device is b, and the current result data obtained by the combination is ab), and the current perception data of the current node device is c, then the current result data is abc. The same is true for other complex multi-dimensional and multi-target situations.

[0137] In the present invention, the preceding node device, current node device, and subsequent node device are not necessarily in a fixed order. For example, among the multiple node devices of a peer computing system, three node devices x, y, and z, in a certain transmission of result data, happen to be node device x that transmits result data x to node device y, and node device y that transmits result data y to node device z; and in another transmission of result data, the three node devices x, y, and z do not necessarily have a data transmission relationship; the same is true for the node devices of the entire peer computing system. Correspondingly, in the example given in the present invention, the corresponding transmission path in a single transmission of result data does not mean that the node devices constituting the transmission path are on the same transmission path in each transmission of result data. The transmission path is obtained by induction based on the result data transmission that has occurred, rather than a pre-planned path (including a fixed path and a dynamic path) or a path obtained by calculation. The transmission path of the present invention only represents an inductive result, not its existence.

[0138] Based on the peer-to-peer computing system, there is no master-slave relationship between all node devices. Therefore, the present invention can be implemented without a transmission path, which is different from some existing technologies that must rely on paths to complete event judgment (paths are typical descriptions of a single target, usually what the existing technologies of single-point identification need to rely on to judge events), and achieve event judgment or response from different technical ideas. The node device connects and controls the corresponding execution device, and correspondingly, the response includes the response made by the execution device to which the node device is connected.

[0139] For a certain node device, part of the self-processing result data corresponding to the connected execution device (that is, part of the content in the self-processing result data corresponds to the execution device connected to the control node device) is processed as raw data to obtain result data, and the result data is propagated to other node devices; other node devices that receive the result data use the result data as one of the collected raw data, and affect the result data of other node devices through the result data. Based on this, the node devices in the peer computing system perform collaborative computing as the raw data is acquired and the result data is calculated; the self-processing result data is responded to based on the collaborative computing until the result data calculated by one or more node devices matches the request content corresponding to the self-processing result data (that is, the self-processing result data is regarded as a request, and correspondingly, the request content is formed, such as the need to perform a certain control on a certain execution device, including specific control parameters, etc.).

[0140] In the present invention, the execution device is controlled by symbolic means. Specifically, the self-processing result data represented by the result data is symbolized by an encoder deployed on the node device, and converted into a control symbol corresponding to the execution device, and the execution device is controlled by the control symbol; or, the output control symbol is used as a request, and the result data obtained is calculated, and the result data includes the corresponding control symbol. After the result data is obtained by calculation, the execution device can be directly controlled by the control symbol.

[0141] In step 1) of the present invention, the node devices of the peer computing system are used to collect various types of perception data related to the target; for a certain node device, the various types of perception data collected and the training data corresponding to the processing result data corresponding to the collected perception data are obtained. In order to reduce the computing power overhead of training a large model and improve the training efficiency, in the present invention, the large model is segmented and trained by different node devices respectively, and the segmented large model is used to generate self-processing result data. In step 1) of the present invention, when the training data is used for model training, the node device receives the assigned training tasks and training data and performs corresponding model training; the training results are processed as raw data to obtain result data, and the result data is propagated to other node devices; other node devices that receive the result data use the result data as one of the collected raw data, and the result data of other node devices is affected by the result data. Based on this, the node devices in the peer computing system perform collaborative calculations as the raw data is obtained and the result data is calculated; the training results are responded to based on the collaborative calculation until the result data calculated by one or more node devices meets the goal of model training.

[0142] Correspondingly, in step 2), the node devices of the peer computing system are used to collect various types of perception data related to the target; for a certain node device, real-time data formed by the collected various types of perception data is obtained;

[0143] When using a large model to generate self-processing result data, the node device receives the assigned generation task and generates the corresponding part of the self-processing result data; the generated result is processed as the original data to obtain the result data, and the result data is propagated to other node devices; other node devices that receive the result data use the result data as one of the collected original data, and influence the result data of other node devices through the result data. Based on this, the node devices in the peer computing system perform collaborative computing as the original data is acquired and the result data is calculated; based on the collaborative computing, the generated result is responded to until the result data calculated by one or more node devices matches the self-processing result data. Among them, the self-processing result data generated by the large model is a processing solution for the behavior, attribute, state or relationship between objects at the moment when the objects within the monitoring range are collected by various types of data acquisition devices for perception data; further, in step 2), the corresponding self-processing operation is performed based on the processing solution.

[0144] It can be seen that the present invention utilizes a peer computing system, which can be used to realize the perception of targets, segmentation training of large models, and the generation of self-processing result data of large models using segmentation, and all are based on the same principle, namely collaborative computing and the transmission of result data.

[0145] In the present invention, especially in small-scale scenarios and short-time application scenarios, and using symbolization such as simplified transcoding, training data and real-time data are obtained, and based on a peer computing system, the information volume of the original data is integrated into the result data through collaborative computing. When performing model training and self-processing, the collected original data (i.e., step 1) and step 2) collected perception data and corresponding processing result data) do not need to be stored and can be restored, thereby saving storage costs and being immune to attacks on the model by providing false or defective training data.

[0146] In this embodiment, simplified transcoding and corresponding decoding and restoration are also performed based on a peer-to-peer computing system. Specifically, the node device deploys an encoder for generating training data and real-time data, and a decoder matching the encoder; wherein the encoder and decoder can generate a coding model for generating training data and real-time data through machine learning, or can be implemented as a hardware encoder and decoder and connected to the node device. Furthermore, when encoding and decoding are required, simplified transcoding and restoration can be performed through the encoder and decoder deployed in the node device, or through the encoder and decoder connected to the corresponding node device.

[0147] Correspondingly, the various types of perception data collected as described in step 1), including but not limited to, the processing result data corresponding to the collected perception data, or the various types of perception data collected as described in step 2), are input into the encoder to generate training data or real-time data in a format that simplifies transcoding.

[0148] The training data or real-time data of the format result of the simple code value + correction code is used as one of the original data of the calculation result data, and the result data obtained by calculation is transmitted to the subsequent node device. The subsequent node device receives the result data, uses a decoder to restore the simple code value + correction code as the training data or real-time data from the result data, and then translates the simple code value into an original code parameter string, and disassembles the original code parameter string into several original code parameters corresponding to the current node device; translates the correction code into a correction value string, and disassembles the correction value string into several correction values ​​corresponding to the current node device; adds the correction value to the first value of the level section corresponding to the original code parameter, and restores the level of the perception data of the type sent by the current node device.

[0149] Based on this, when there is a data restoration request, the result data and the decoder are combined to restore the various types of sensory data collected in step 1), including but not limited to the processing result data corresponding to the collected sensory data, and the various types of sensory data collected in step 2). In specific implementation, data restoration can be used as a request, and the result data obtained by calculation can be directly restored through the encoder and decoder to obtain the various types of sensory data collected in step 1), including but not limited to the processing result data corresponding to the collected sensory data, and the various types of sensory data collected in step 2).

[0150] Specifically, a certain node device is taken as the current node device, and combined with the data transmission of its preceding node device and subsequent node device (the preceding node device and subsequent node device in the present invention are only used to describe the relationship between the current node device and the current node device in the current calculation and data transmission process, and do not mean that there is a necessary relationship between them and the priority relationship), correspondingly, the current node device receives the result data output by other node devices (including the preceding node device), and the subsequent node device receives the result data output by other node devices (including the current node device). For the current node device, the collected perception data is combined with the result data from other node devices (including the preceding node device), the result data of the current node device is calculated, and sent to other node devices (including the subsequent node device). Similarly, the working process of the subsequent node device is the same as that of the current node device, and the preceding node device also receives the result data of the preceding node device of the preceding node device, and performs the same working process as the current node device; that is, the node devices in the peer computing system perform the same working process. Furthermore, the node devices in the peer computing system perform collaborative computing with the collection of perception data and the calculation of result data. Among them, the result data output by a certain node device is only received by the subsequent layer of node devices and used as input. The result data of the subsequent layer of node devices will cover the result data of the previous layer of node devices (including the aforementioned certain node device).

[0151] In a peer-to-peer computing system, all events are processed synchronously, and it is not necessary to explicitly generate staged result outputs such as what events are discovered and what the specific content of the events is; in a peer-to-peer computing system, only the perception of the sensor and the response of the corresponding execution device are clear, and the other intermediate processes are processed by collaborative computing at the same time, that is, during the operation of the present invention, the intermediate process of event discovery is imperceptible, and as the collaborative computing proceeds, the result data of the node device is obtained, and the corresponding execution device automatically responds and executes. Furthermore, the present invention is different from the single-target calculation of the prior art, and is a multi-target state simultaneous calculation, such as tolls. If the single-target calculation of the prior art is used, the specific fee is calculated for a certain vehicle path. The present invention can calculate the fees of all 35,000 vehicles that need to pay for 50,000 vehicles through 5,000 node devices, and simultaneously calculate the paths of 50,000 vehicles, the matching of charging conditions and the fee calculation, and is reflected in the direct driving of the corresponding terminal (that is, the execution device is a payment terminal) to implement non-sensing payment deduction or display the charging QR code according to the corresponding vehicle position (that is, the execution device is a display terminal). During this period, the above-mentioned goals and logics are calculated simultaneously, and there is no calculation mode in which a certain node has a certain vehicle's path information, charging standards, location judgment, and terminal interactive control.

[0152] The present invention is different from the traditional mode. What is transmitted between node devices (including between node devices in a peer computing system) is actually a state representation (such as a representation through a state value), not the state data itself. In the present invention, the state representation can be simplified and transcoded, and some original information can be discarded and compressed under the premise of achieving the event target accuracy. It can also be simplified and transcoded in the original accuracy, so that the node devices can communicate with very small bytes, but ensure that a large number of node devices can accurately calculate and obtain the behavior, attributes, state or relationship between objects of a large number of targets when needed (wherein the acquisition is often unnecessary).

[0153] For example, in a simplified embodiment, target a is sensed by node devices 1, 2, and 3 in sequence, all the way to node device 9; then the target perception data output by node device 2 to node device 3 is target a from node device 1 to node device 2, that is, 1-2, which can be described by a state value in the matrix. Subsequently, node device 3 also senses target a, and the perception result of node device 3 is added in a matrix manner, that is, 1-2-3, and sent to the target node device. 1-2-3 is still a state value in the matrix. Similarly, the target perception data transmitted by node device 9 to the next group of node devices is target a from node device 1 to node device 9, that is, 1-2-3-4-5-6-7-8-9, which is still just a state value in the matrix.

[0154] In the case of full coverage, the path information involved in the node devices is often multi-dimensional. Correspondingly, the node devices will receive multiple result data. For example, node device x may receive the tracking sequences of target a corresponding to paths such as path 123, path 825, path abc, path dbm, and path lmn. Node device x aggregates these result data into a matrix for expression. Therefore, although these node devices discover target a from their respective different paths, at the node device x which is the collection point of all paths, the target perception results of all node devices for target a will be expressed in the form of a matrix at the position of node device x, and the expression result is still a state value.

[0155] During specific implementation, different node devices may collect data of different types due to different sensor types. Correspondingly, the types of perception data collected by different node devices are also not exactly the same. Furthermore, if the current node device does not have one or more types of sensors set by the previous node device, the current node device will still take the previous result data containing the corresponding type of perception data sent by the previous node device as input, and synchronously process it with the perception data collected by the sensor of the current node device to avoid the loss of some types of perception data during the result data transmission process, thereby affecting the integrity of the result data.

[0156] In the present invention, the automatic control that the execution device needs to perform can be regarded as a request command. In the present invention, the response to the request command includes a variety of different situations such as "demand-execution", "request-response" or other situations. In the present invention, when the result data calculated by one or more node devices in the peer computing system matches the request content corresponding to the self-processing result data, the result corresponding to the request content (for example, the result after self-processing based on the self-processing result data) is characterized in the result data output by one or more node devices according to the preset conditions or the output of the data processing model deployed in advance or the program deployed on the node device; if it is determined that the current node device needs to respond to the request command, the current node device will send a control instruction to the execution device connected to the current node device according to the calculated result data; that is, the "demand-execution" situation. When all the execution devices connected to all the node devices that need to respond have completed the response action, the self-processing operation is completed. When the response to the request command is "request-response", that is, the request content corresponding to the request command issued by the access terminal requires the peer computing system to feedback the response content, then the obtained response content is transcribed through multiple semantic modules to obtain one or several data information that matches the requester's needs, and is responded by one or more output terminals (execution devices).

[0157] In order to ensure that the data source and the calculation process are further credible, in the present invention, all node devices encrypt the result data obtained by their calculations based on the encryption consensus mechanism to obtain the encryption result, and then send the encryption result to other node devices. The encryption consensus mechanism includes one or more consensus mechanisms, and different consensus mechanisms correspond to changes in the encryption algorithm structure and parameters of the node device.

[0158] Node devices communicate with each other using standard-sized data packets (i.e., result data or calculation results). In the present invention, the node devices of the peer-to-peer computing system are similar to human neurons. Each neuron does not transmit specific data that directly describes external events. Similarly, the node devices do not output raw data, but process the raw data obtained by the connected sensors and data acquisition devices into standard-sized data packets (i.e., result data or calculation results, similar to the nerve impulses of neurons) according to their own data processing models (similar to the biological characteristics of nerve cells). The amount of information contained in a single data packet is not enough to restore any event and target information. The calculation results, multi-dimensional data matrix elements, and physical space and facility correspondences on the entire peer-to-peer computing system must be jointly calculated to obtain a definite result. The collaborative calculation is not very dependent on the data output by a few node devices, and the collaborative calculation simultaneously processes the needs received or initiated by all node devices. It is a collaborative verification calculation of super-multi-dimensional related information, which can fundamentally change the nature of traditional information technology that is sensitive to single-point security.

[0159] In order to ensure the integrity of data and the effective execution of collaborative computing, in the present invention, the peer computing system deploys a QoS mechanism, and the QoS mechanism is to give priority to ensuring the transmission quality of the result data between node devices.

[0160] In the specific implementation, the networking mode of the peer computing system is one or a combination of 4G mode, 5G mode or MESH mode, so as to adapt to different application scenarios, comprehensively implement feasibility, cost considerations and other factors, and realize the optimal solution. Among them, the MESH mode is based on the LTE standard, and communicates at the physical layer of the LTE standard; it carries data with a customized frame structure and interacts with a dedicated wireless communication protocol. The frame structure is customized to be suitable for the calculation of the peer computing system, and the proprietary wireless communication protocol developed for the calculation of the urban agglomeration peer computing system is adopted, which can further improve its security and reliability. In addition, the wireless algorithm fully adapts to the multipath channel environment based on the consensus mechanism control required for the calculation of the peer computing system. The communication distance in the city is 100 meters-10 kilometers, and the omnidirectional antenna is used in the field. Under the condition of efficient transmission of 120 kilometers, it can also achieve efficient transmission at a distance of 120 kilometers. In this embodiment, the communication distance of the Mesh network is: the distance between node devices indoors is 50-150 meters, the distance between node devices outdoors is 50 meters-120 kilometers, and the number of node devices that can be accessed by each node is 65535. In addition, when networking in 4G mode or 5G mode, there is no limit on the communication distance, and the number of accessible node devices depends on the computing power of the computing chip and the communication delay.

[0161] In a peer-to-peer computing system, for a certain point sample of a certain object, the result data transmitted from the node device that collects the point sample to other node devices can enable subsequent node devices to adjust their perceptual attention according to the characteristics of the point sample (it is not necessary to include the characteristics of the point sample in the result data, but the characteristics of the point sample participate in the calculation of the previous node device, so that when the result data of the previous node device is used as the input of the data processing model of the subsequent node device, the data processing model of the subsequent node device can achieve the effect of adjusting perceptual attention in the calculation); or, report the characteristics of the point sample for subsequent node devices to adjust their perceptual attention (directly express the characteristics of the point sample in the result data). If other subsequent node devices do not detect the characteristics of the point sample, but it can be determined from the characteristics of other point samples that the characteristics of the point sample that cannot be detected still belong to the object, then the characteristics of the point sample that cannot be detected will continue to be expressed in the result data of the current node device and transmitted to other node devices. For example, the previous node device perceives the color of object A, and the current node device does not perceive the color of object A. However, it can be determined from the perception data of other node devices that object A exists in addition to other objects. Then the color of object A that has not been perceived is still expressed in the result data of the current node device.

[0162] In this embodiment, the method of reporting the characteristics of the point sample for subsequent node devices to adjust the perception attention is: adjusting the parameters of the data processing model of the subsequent node device according to the characteristics of the point sample provided by the previous node device, so that the subsequent node device improves the computing power of identifying the characteristics of the point sample; or, the subsequent node device uses the perception attention model to match the characteristics of the received point sample to adjust the computing power. Among them, the above-mentioned "feature" is different from the meaning of "feature recognition" in the prior art. The "feature recognition" in the prior art usually refers to information that can determine the identity of the target, while the "feature" of the present invention represents a kind of perceived perception data belonging to the object, such as coordinates, the color of the object, etc., and the "non-specific feature recognition" of the object cannot be directly completed by the "feature" of single-point perception. The "non-specific feature recognition" described is different from the "recognition" in the usual sense in terms of strict conceptual definition. The "recognition" in the usual sense refers to the concreteness of the target or the specific identity information of the target, such as who it is (including name, specific information indicating the identity of the target), what it is (such as a car, a person, etc.). The "identification" in the "non-specific feature identification" described in the present invention means that each unique target (i.e., an object that already exists in the working area of ​​the vehicle, an object that enters the working area) is identified as itself; that is, for a certain object to be identified, its existence is unique. After the present invention implements "non-specific feature identification", it is determined that the object to be identified (i.e., a target that has not been identified or confirmed) is itself, rather than other objects to be identified. The result of "non-specific feature identification" does not need to determine the specific features of the object to be identified, and does not need to determine the identity information or concrete image of the object to be identified. For example, for a person, it is regarded as the object to be identified A, and for an object, it is regarded as the object to be identified B. Then, after implementing "non-specific feature identification", it is not necessary to identify the object to be identified A as a person and who is the specific identity, and it is not necessary to identify the object to be identified B as an object and what the specific object is; instead, it is necessary to determine that the object to be identified A is the object to be identified A itself, and the object to be identified B is the object to be identified B itself. Then, corresponding services or controls can be performed for the object to be identified A or the object to be identified B.

[0163] In this embodiment, the method for reporting the characteristics of the point sample for subsequent node devices to adjust perceptual attention is: adjusting the parameters of the data processing model of the subsequent node device according to the result data expressing the characteristics of the point sample provided by the previous node device (in the present invention, the characteristics of the point sample itself are usually not provided, but the characteristics of the point sample are expressed in the result data), or the characteristics of the point sample (that is, the characteristics of the point sample itself), so that the subsequent node device improves the computing power for identifying the characteristics of the point sample; or, the subsequent node device uses the perceptual attention model to match the characteristics of the received point sample or the result data expressing the characteristics of the point sample to adjust the computing power.

[0164] When a node device processes the result data output by several previous node devices, based on the data processing model, when the objects described by several previous node devices can be determined as the same target through some common point sample features, the point sample features and other information described by each node device are merged into the same target. For example, point sample features in physical spaces that are almost completely overlapped at the same time can be determined to be the same target.

[0165] In the present invention, the data acquisition device of the node equipment includes one or several combinations of an image acquisition device, an electromagnetic induction device, a temperature measurement device, a vibration frequency sensing device and a laser radar. The data collected by the above devices (i.e., one or several combinations of the image acquisition device, the electromagnetic induction device, the temperature measurement device, and the vibration frequency sensing device) and the three-dimensional point cloud collected by the laser radar or the point cloud generated according to the images collected by multiple image acquisition devices are jointly calculated to obtain three-dimensional points with data; the image color, contour, line, reflectivity, motion trend, electromagnetic characteristics, temperature, temperature change trend, vibration frequency, and vibration frequency change trend based on two-dimensional perception are used as additional attributes of the corresponding three-dimensional points to form a three-dimensional point cloud with attributes; combined with electromagnetic induction, temperature law, vibration frequency change characteristics, motion correlation (different motion correlations presented by different materials such as ropes and cloth), and reflectivity, the correspondence between each area of ​​the three-dimensional point cloud with attributes and each part or associated parts of the 3D appearance of the object to be identified is determined. This embodiment uses the attributes of the attributed three-dimensional point cloud and their correlation to determine the relationship between each point, the correspondence between each area to which each related point belongs and each part or associated part of the 3D appearance of the object to be identified, so as to more accurately determine the point sample features belonging to the object to be identified.

[0166] When the result data received by the node device indicates that the flag used by the current node device to identify the object before the current node device receives the result data is different from the flag used by other node devices to identify the object, and the flags assigned to the object by other node devices are updated, the flag used by the current node device to identify the object before the current node device receives the result data is converted. Specifically, the method for converting the flag used by the current node device to identify the object before the current node device receives the result data is:

[0167] The mark used by the current node device to identify the object before the current reception of result data is replaced with the latest mark allocated to the object by other node devices; this is a relatively simple implementation method provided by the present invention.

[0168] Alternatively, the conversion relationship between the flag used by the current node device to identify the object before the current reception of the result data and the flag assigned to the object by other node devices after the update is recorded, and the conversion is performed when it is necessary to reference the current node device's current reception of the result data; this is a relatively complex implementation method provided by the present invention.

[0169] Alternatively, the node device deploys a conversion model to perform corresponding conversions on the flags of multiple objects according to the input original data or result data; this is a more complex implementation method provided by the present invention.

[0170] The above embodiments are only used to illustrate the present invention, and are not used to limit the present invention. As long as they are based on the technical essence of the present invention, changes and modifications to the above embodiments will fall within the scope of the claims of the present invention.

Claims

1. A self-processing method based on data symbolization and large models, characterized in that: The steps include: 1) Model training step: using various types of data acquisition devices deployed in the working area to collect various types of corresponding types of perception data, the collected types of perception data cover objects within the monitoring range of the corresponding types of data acquisition devices; obtaining processing result data corresponding to the collected perception data; Symbolize the various types of sensory data collected, including but not limited to the sensory data collected, and the processing result data corresponding to the sensory data collected, to form training data that describes the multimodal sensory data and the processing result data corresponding to the sensory data collected with symbols; perform model training using the training data to generate a large model; 2) Self-processing step: Utilize various types of data acquisition devices deployed in the working area to collect various corresponding types of perception data, symbolize the collected perception data of various types, and form real-time data that describes multimodal perception data with symbols; input the real-time data into the large model to generate self-processing result data, and perform corresponding self-processing operations based on the self-processing result data.

2. The self-processing method based on data symbolization and large model according to claim 1, characterized in that: In step 1), taking functional scenarios as units, various types of perception data collected in different functional scenarios and training data formed by processing result data corresponding to the collected perception data are input into a neural network for generating a large model to train and generate the large model; the large model generates self-processing result data based on real-time data; wherein, there is no need to predefine functional scenarios.

3. The self-processing method based on data symbolization and large model according to claim 2 is characterized in that: In step 1), when there are multiple independent functional scenarios with the same functions, firstly, several of the multiple functional scenarios are automatically selected according to the selection rules, and then the selected functional scenarios are used for model training.

4. The self-processing method based on data symbolization and large model according to claim 3 is characterized in that: Furthermore, after automatically selecting a number of functional scenarios, the symbolized training data is first output, or the symbolized training data is restored to corresponding perception data and output, and then the selected functional scenarios are prioritized manually based on the output symbolized training data or restored perception data; And / or, manually set the starting and ending points of training data input for model training in functional scenarios, wherein the starting and ending points of training data input are respectively the state where training data input starts and ends.

5. The self-processing method based on data symbolization and large model according to claim 2 is characterized in that: Step 1) specifically comprises: utilizing various types of data acquisition devices deployed in the working area to continuously acquire various corresponding types of perception data; the continuously acquired perception data of various types include perception data in chronological order acquired from objects in the working area covered by the monitoring range of the corresponding type of data acquisition device; converting the continuously acquired perception data of various types and the processing result data corresponding to the acquired perception data into training data described by symbols, inputting the training data into a neural network used to generate a large model, and training and generating the large model.

6. The self-processing method based on data symbolization and large model according to claim 1 is characterized in that: A three-dimensional global model is obtained by three-dimensional modeling, wherein the three-dimensional global model includes a three-dimensional object model representing the object; In step 1), various types of corresponding virtual acquisition devices are deployed in the three-dimensional global model; Using a virtual acquisition device to perform in-model acquisition on a three-dimensional object model to obtain in-model perception data; inputting in-model training data corresponding to the in-model perception data into a large model to generate in-model self-processing result data, and generating an automated script corresponding to the in-model self-processing result data; The self-processing result data in the model is the model-in-processing solution for the behavior, attribute, state or relationship between objects of the three-dimensional object model at the moment when the model-in-perception data is collected; the self-processing result data in the model is used as the model-in-processing result data corresponding to the model-in-perception data; The 3D object model corresponding to the processing result data in the model is controlled by automated scripts or manually controlled to execute the model processing solution in the 3D global model, thereby optimizing the training of the large model.

7. The self-processing method based on data symbolization and large model according to claim 6 is characterized in that: Delimiting a modeling area corresponding to the working area; performing three-dimensional modeling on real objects in the modeling area to construct a three-dimensional global model; the three-dimensional global model includes a three-dimensional object model corresponding to the real object; Correspondingly, in step 1), corresponding to various types of data acquisition devices in the working area, corresponding various types of virtual acquisition devices are deployed in the three-dimensional global model; the virtual acquisition devices are used to perform in-model acquisition on the three-dimensional object model to obtain the perception data within the model.

8. The self-processing method based on data symbolization and large model according to claim 1, characterized in that: Perception data also includes but is not limited to laws and regulations, implementation standards, specifications, and custom constraints; the processing result data corresponding to the collected perception data includes but is not limited to control instructions and interactive information.

9. The self-processing method based on data symbolization and large model according to claim 8, characterized in that: The large model generates self-processing result data including but not limited to interaction data and / or task strategies; When the self-processing result data is interactive data, the interactive device is controlled to output the interactive data through the matching task strategy; When the self-processing result data is a task strategy, the task strategy is used to control the corresponding object to perform the corresponding operation.

10. The self-processing method based on data symbolization and large model according to claim 9, characterized in that: The task strategy includes control instructions and instruction sequences, and controls corresponding objects to perform corresponding operations based on the task strategy.

11. The self-processing method based on data symbolization and large model according to claim 10, characterized in that: The object includes or is provided with at least one object, and various working parameters of the object are obtained as perception data; the object responds to the various working parameters included in the processing scheme to control the object.

12. The self-processing method based on data symbolization and large model according to claim 9, characterized in that: The interactive data includes but is not limited to voice interactive data, image interactive data, and text interactive data, and data interaction is performed through corresponding types of interactive devices.

13. The self-processing method based on data symbolization and large models according to any one of claims 1 to 12, characterized in that: Setting up a peer-to-peer computing system, the peer-to-peer computing system includes a plurality of node devices, and there is no primary-secondary relationship between all the node devices; using the node devices of the peer-to-peer computing system to collect various types of perception data related to the target, the target including an object already existing in the working area and an object entering the working area; The node device deploys an encoder for symbolizing the sensed data; for a certain node device, the encoder is used to symbolize the sensed data of various types collected and the processing result data corresponding to the sensed data collected, and the result data is processed as the original data to obtain the result data, and the result data is transmitted to other node devices; other node devices that receive the result data use the result data as one of the collected original data, and the result data of other node devices is influenced by the result data; Based on this, multiple node devices in the peer computing system perform collaborative computing. When the result data of the collaborative computing can determine the target perception result, the target corresponding to the multimodal perception data is perceived to obtain the target perception result, wherein the target perception result is a comprehensive state of one or more combinations of the target's behavior, attributes, state, or the relationship between objects, and is represented in the result data.

14. The self-processing method based on data symbolization and large model according to claim 13, characterized in that: The node device connects to and controls the corresponding execution device; For a certain node device, a portion of the self-processing result data corresponding to the connected execution device is processed as original data to obtain result data, and the result data is transmitted to other node devices; other node devices that receive the result data use the result data as one of the collected original data, and influence the result data of other node devices through the result data; The node devices in the peer-to-peer computing system perform collaborative computing as the original data is acquired and the result data is calculated; Respond to the self-processing result data based on collaborative computing until the result data calculated by one or more node devices matches the request content corresponding to the self-processing result data; The self-processing result data represented by the result data is symbolized by an encoder deployed on the node device, and converted into a control symbol of the corresponding execution device, and the execution device is controlled by the control symbol; or, the output control symbol is used as a request, the result data obtained by calculation includes the corresponding control symbol, and the execution device is directly controlled by the control symbol.

15. The self-processing method based on data symbolization and large model according to claim 13, characterized in that: In step 1), various types of perception data related to the target are collected using node devices of the peer computing system; for a certain node device, the collected perception data of various types and training data corresponding to the processing result data corresponding to the collected perception data are obtained; When using training data to perform model training, the node device receives the assigned training tasks and training data, and performs corresponding model training; the training results are processed as raw data to obtain result data, and the result data is transmitted to other node devices; other node devices that receive the result data use the result data as one of the collected raw data, and influence the result data of other node devices through the result data; The node devices in the peer-to-peer computing system perform collaborative computing as the original data is acquired and the result data is calculated; Respond to the training results based on collaborative computing until the result data calculated by one or more node devices meets the objectives of model training.

16. The self-processing method based on data symbolization and large model according to claim 15, characterized in that: In step 2), various types of perception data related to the target are collected by using node devices of the peer computing system; for a certain node device, real-time data formed by the collected various types of perception data is obtained; When the large model is used to generate self-processing result data, the node device receives the assigned generation task and generates the corresponding part of the self-processing result data; the generated result is processed as the original data to obtain the result data, and the result data is transmitted to other node devices; other node devices that receive the result data use the result data as one of the collected original data, and influence the result data of other node devices through the result data; The node devices in the peer-to-peer computing system perform collaborative computing as the original data is acquired and the result data is calculated; The generated results are responded to based on collaborative computing until the result data calculated by one or more node devices matches the self-processing result data.

17. The self-processing method based on data symbolization and large models according to claim 16, characterized in that: For any node device that collects various types of perception data related to the target, the current node device receives the previous result data output by the previous node device, and combines it with the current perception data collected by the current node device, distributes it through the peer computing system and transmits it to the node device that meets the computing requirements for calculation, and obtains the current result data; and transmits it to the subsequent node device in a collaborative computing manner; and so on, thereby completing the transmission of the target perception results between the node devices; The target perception result is a comprehensive state of one or more combinations of the target's behavior, attributes, state, or relationship between objects, and is represented in the result data; The current perception data represents the current behavior, attributes, state, or current state of the relationship between objects; The predecessor result data represents the predecessor behavior, attribute, state or the relationship between objects of the target predecessor. The current result data represents the current comprehensive state of the target's current behavior, attribute, state or the relationship between objects. The current comprehensive state is the combination of the current state and the previous comprehensive state.

18. The self-processing method based on data symbolization and large models according to claim 17, characterized in that: The self-processing result data generated by the large model is a processing solution for the behavior, attributes, state or relationship between objects within the monitoring range at the moment when the perception data is collected by various types of data acquisition devices; in step 2), the corresponding self-processing operation is performed based on the processing solution.

19. The self-processing method based on data symbolization and large models according to claim 13, characterized in that: When the result data calculated by one or more node devices in the peer computing system matches the request content corresponding to the self-processing result data, the result corresponding to the request content is represented in the result data output by the one or more node devices according to the preset conditions or the output of the pre-deployed program or the data processing model deployed on the node device; If it is determined that the current node device needs to respond to the request command, the current node device will send a control instruction to the execution device connected to the current node device according to the calculated result data; when the execution devices connected to all the node devices that need to respond have completed the response action, the self-processing operation is completed.

20. The self-processing method based on data symbolization and large models according to claim 1, characterized in that: The data acquisition device includes one or a combination of an image acquisition device, an electromagnetic induction device, a temperature measuring device, a vibration frequency sensing device and a laser radar. The data collected by the above devices and the three-dimensional point cloud collected by the laser radar or the point cloud generated according to the images collected by multiple image acquisition devices are jointly calculated to obtain three-dimensional points with data; the image color, contour, line, reflectivity, motion trend, electromagnetic characteristics, temperature, temperature change trend, vibration frequency, and vibration frequency change trend based on two-dimensional perception are used as additional attributes of the corresponding three-dimensional points to form a three-dimensional point cloud with attributes; combined with electromagnetic induction, temperature laws, vibration frequency change characteristics, motion correlation, and reflectivity, the correspondence between each area of ​​the three-dimensional point cloud with attributes and each part or associated parts of the consumer's 3D appearance is determined.

21. The self-processing method based on data symbolization and large models according to claim 1, characterized in that: If in step 1), new training data is obtained using perception data of different working areas, different objects, and different modalities, and the large model is trained, the large model is fine-tuned to enable the large model to adapt to the perception data of different objects and different modalities.

22. The self-processing method based on data symbolization and large models according to any one of claims 1 to 12, characterized in that: The multimodal perception data and the processing result data corresponding to the collected perception data are encoded using known encoding rules to obtain symbol encoding corresponding to the encoding rules, thereby realizing the symbolization of the multimodal perception data and the processing result data corresponding to the collected perception data; the known encoding rules include but are not limited to one or more combinations of texts, characters, super-large base sequences, known symbols of non-natural languages, and custom symbols of various natural languages.