Intelligent management method and system based on multi-modal information integrated processing

Through the intelligent management method of multimodal information integration processing, combined with video stream and RFID information, a multimodal call model is built, which solves the resource consumption and accuracy problems of forklift positioning information acquisition in singlemodal information processing, and realizes flexible debugging and Accurate access.

CN119067569APending Publication Date: 2024-12-03DONGGUAN MENGDA INDAL INVESTMENT
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411200897.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, when warehouse management is carried out based on single-modal information, the acquisition of forklift positioning information is greatly affected by the environment and consumes a lot of computing power costs, and there is a lack of intelligent management methods for multimodal information integration processing.

Method used

Using an intelligent management method based on multimodal information integration processing, multimodal data (video stream information and warehouse forklift RFID positioning information) is obtained by pre-configuring program complementary call rules, building multiple call models, generating result values ​​and configuring conditions The status value is to initiate corresponding business instructions to achieve accurate acquisition of forklift positioning information.

Benefits of technology

Save service resources, reduce development costs, and ensure the accuracy of business applications. Through multi-modal information integration processing, it realizes flexible debugging and accurate acquisition of forklift positioning information, solving the resource consumption and accuracy problems of single-modal information processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067569A_ABST
    Figure CN119067569A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses an intelligent management method and system based on multi-modal information integration processing, and the method comprises the steps: pre-configuring a program complementary calling rule based on different business scenes, and the program complementary calling rule comprises the steps: configuring a program complementary calling demand of each business scene, and obtaining a program corresponding to multi-modal data, configuring a calling model associated with each program, configuring a condition state value for a result value generated by the calling model, and configuring a calling model corresponding to each condition state value; obtaining a program complementary calling requirement of the current service scene, and executing a corresponding program complementary calling rule; executing a program complementary calling demand of the current service scene to obtain a program corresponding to the multi-modal data, executing a calling model associated with the program, obtaining a result value generated by the calling model, obtaining a condition state value corresponding to the result value, and calling the calling model corresponding to the condition state value. The problem that intelligent management cannot be performed based on multi-modal information in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an intelligent management system, method, electronic device and storage medium based on multimodal information integrated processing. Background Art

[0002] Unimodality refers to the transmission and processing of information through a specific sense or information form, such as the transmission of information through a single mode such as text, sound, and image. Multimodality refers to the transmission and processing of information through two or more senses or information forms at the same time, such as the transmission of information through the combined action of multiple modes such as text, sound, and image.

[0003] In the prior art, warehouse management is usually performed based on single-modal information, for example, video stream information is obtained through warehouse cameras, the video stream information is processed, and forklift positioning information is obtained; or,

[0004] Obtain forklift positioning information through warehouse forklift RFID positioning information;

[0005] However, obtaining forklift positioning information through warehouse forklift RFID positioning information is greatly affected by the warehouse environment, and obtaining forklift positioning information through video stream information requires a lot of computing power costs.

[0006] Therefore, there is an urgent need for an intelligent management method based on integrated processing of multimodal information. Summary of the invention

[0007] The purpose of the embodiments of the present invention is to provide an intelligent management system, method, electronic device and storage medium based on multimodal information integrated processing, so as to solve the problem that intelligent management based on multimodal information cannot be performed in the prior art.

[0008] To achieve the above objectives, an embodiment of the present invention provides an intelligent management method based on multimodal information integration processing, and the intelligent management method based on multimodal information integration processing specifically includes:

[0009] Pre-configure program complementary calling rules based on different business scenarios, wherein the program complementary calling rules include configuring the program complementary calling requirements of each business scenario to obtain the program corresponding to the multimodal data, the number of the program is one or more, configuring the calling model associated with each program, configuring the conditional state value for the result value generated by the calling model, configuring the calling model corresponding to each conditional state value, the number of the calling models is one or more;

[0010] Obtain the program complementary calling requirements of the current business scenario and execute the corresponding program complementary calling rules;

[0011] The corresponding program complementary calling rules are as follows:

[0012] Execute the program complementary call requirements of the current business scenario to obtain the program corresponding to the multimodal data, execute the call model associated with the program, obtain the result value generated by the call model, obtain the conditional status value corresponding to the result value, and call the call model corresponding to the conditional status value.

[0013] On the basis of the above technical solution, the present invention can also be improved as follows:

[0014] Furthermore, the executing corresponding program complementary calling rules also includes:

[0015] When the complementary calling requirement of the program is to obtain forklift positioning information, determining that the multimodal information is video stream information and warehouse forklift RFID positioning information;

[0016] The calling model corresponding to the warehouse forklift RFID positioning information is a positioning data acquisition model;

[0017] The calling models corresponding to the video stream information are video stream acquisition model, video stream extraction model, information comparison model and image recognition positioning analysis model. Based on the result value generated by the final calling model, corresponding business instructions are initiated, including initiating instructions to the IoT device or transmitting interface data to the system.

[0018] Furthermore, the acquiring of program complementary calling requirements and multimodal information corresponding to the program complementary calling requirements, and constructing a plurality of calling models based on the multimodal information, includes:

[0019] When the complementary calling requirement of the program is to obtain forklift positioning information, determining that the multimodal information is video stream information and warehouse forklift RFID positioning information;

[0020] The calling model corresponding to the warehouse forklift RFID positioning information is a positioning data acquisition model;

[0021] The calling models corresponding to the video stream information are a video stream acquisition model, a video stream extraction model, an information comparison model and an image recognition and positioning analysis model.

[0022] Furthermore, the pre-configuration of complementary calling rules of programs based on different business scenarios includes:

[0023] Based on business experience, conditional status values ​​are configured for the result values ​​of the calling model, or based on the historical data of the program complementary calling rules corresponding to the entire business scenario, automatic parameter adjustment configuration is performed to configure conditional status values ​​for the result values ​​of the calling model, wherein one calling model can generate one or more result values, and multiple calling models can generate one or more result values, and one or more conditional status values ​​are configured for one or more result values;

[0024] The business meaning corresponding to the result value generated by the calling model is defined through the result indicator.

[0025] Furthermore, the pre-configuration of complementary calling rules of programs based on different business scenarios also includes:

[0026] At least one call model is associated with each condition state value.

[0027] Furthermore, the pre-configuration of complementary calling rules of programs based on different business scenarios also includes:

[0028] Establishing a database, wherein the database includes conditional state values ​​and a call model corresponding to each conditional state value;

[0029] Establish the relationship between the result indicator and the condition status value and make the association;

[0030] Establish the relationship between the call model and the condition state value and associate them.

[0031] Furthermore, the obtaining of the program complementary calling requirements of the current business scenario and executing the corresponding program complementary calling rules include:

[0032] Obtaining the result value generated by the calling model, determining whether the result value has a configuration condition state value, and when the result value generated by the calling model has a configuration condition state value, generating a calling instruction to call the calling model associated with the condition state value;

[0033] When the result value generated by the calling model does not have the configuration condition status value, no calling instruction is generated and the calling model is no longer called.

[0034] Furthermore, the intelligent management method based on multimodal information integration processing also includes:

[0035] A multimodal knowledge graph is constructed based on the multimodal information, and features corresponding to the requirements are complementarily called through the multimodal knowledge graph completion program.

[0036] An intelligent management system based on multimodal information integrated processing, comprising:

[0037] A configuration module, used to pre-configure program complementary calling rules based on different business scenarios, wherein the program complementary calling rules include configuring the program complementary calling requirements of each business scenario to obtain the program corresponding to the multimodal data, the number of the program is one or more, configuring the calling model associated with each program, configuring the conditional state value for the result value generated by the calling model, and configuring the calling model corresponding to each conditional state value, the number of the calling models is one or more;

[0038] The acquisition module is used to obtain the program complementary call requirements of the current business scenario;

[0039] An execution module, used for executing corresponding program complementary calling rules;

[0040] The execution module includes a multimodal data acquisition module, a program driving module and a model calling module;

[0041] A multimodal data acquisition module is used to execute the program complementary call requirements of the current business scenario to acquire the program corresponding to the multimodal data;

[0042] The model calling module is used to execute the calling model associated with the program, obtain the result value generated by the calling model, obtain the condition state value corresponding to the result value, and call the calling model corresponding to the condition state value;

[0043] The program driving module is used to initiate corresponding business instructions based on the result value generated by the final call model, including initiating instructions to the Internet of Things device or transmitting interface data to the system.

[0044] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method are implemented when the processor executes the computer program.

[0045] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0046] The embodiments of the present invention have the following advantages:

[0047] The intelligent management method based on multimodal information integrated processing in the present invention executes corresponding program complementary calling rules based on business scenarios to save service resources. For example, multimodal data is stored in the cloud, and the edge end only calls when some data is needed; the program complementary calling requirements of each business scenario are configured to obtain the program corresponding to the multimodal data, the number of the programs is one or more, the calling model associated with each program is configured, the conditional status value is configured for the result value generated by the calling model, and the calling model corresponding to each conditional status value is configured, the number of the calling models is one or more; based on the actual situation of the scene application, the configuration can be modified at any time, and each program for obtaining multimodal data, the called model, etc. can be regarded as a unit, and then these units can be configured based on the actual scene needs, so that the multimodal structure and status value of the scene application configuration can be debugged very flexibly, the development cost can be reduced, and the accuracy of the business application can be guaranteed, which solves the problem that the prior art cannot perform intelligent management based on multimodal information. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0049] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0050] Figure 1 It is a flow chart of the intelligent management method based on multimodal information integration processing of the present invention;

[0051] Figure 2 This is a first architecture diagram of an intelligent management system based on multimodal information integration processing of the present invention;

[0052] Figure 3 This is a second architecture diagram of the intelligent management system based on multimodal information integration processing of the present invention;

[0053] Figure 4 This is a schematic diagram of the physical structure of the electronic device provided by the present invention.

[0054] The accompanying drawings are denoted as follows:

[0055] Configuration module 10, acquisition module 20, execution module 30, multimodal data acquisition module 301, model calling module 302, program driving module 303, knowledge graph construction module 40, electronic device 50, processor 501, memory 502, bus 503. DETAILED DESCRIPTION

[0056] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Figure 1 FIG. 1 is a flow chart of an embodiment of an intelligent management method based on multimodal information integration processing of the present invention, as shown in FIG. Figure 1 As shown, an intelligent management method based on multimodal information integration processing provided by an embodiment of the present invention includes the following steps:

[0058] S101, pre-configuring program complementary calling rules based on different business scenarios;

[0059] Specifically: the program complementary calling rule includes configuring the program complementary calling requirement of each business scenario to obtain the program corresponding to the multimodal data, the number of the program is one or more, configuring the calling model associated with each program, configuring the conditional state value for the result value generated by the calling model, configuring the calling model corresponding to each conditional state value, the number of the calling models is one or more;

[0060] Based on the collection of data of various modalities corresponding to IoT devices and systems accessed and applied by enterprises, develop data collection models based on different modalities and open data collection models to set the device ports that need to be collected, so as to facilitate the subsequent call of the collection model to collect the required device data, including sound data, video data, image data, etc.;

[0061] Build multiple call models based on different single modality data;

[0062] Based on business experience, conditional status values ​​are configured for the result values ​​of the calling model, or based on the historical data of the program complementary calling rules corresponding to the entire business scenario, automatic parameter adjustment configuration is performed to configure conditional status values ​​for the result values ​​of the calling model, wherein one calling model can generate one or more result values, and multiple calling models can generate one or more result values, and one or more conditional status values ​​are configured for one or more result values;

[0063] The business meaning corresponding to the result value generated by the calling model is defined through the result indicator.

[0064] At least one call model is associated with each condition state value.

[0065] Establishing a database, wherein the database includes conditional state values ​​and a call model corresponding to each conditional state value;

[0066] Establish the relationship between the result indicator and the condition status value and make associations;

[0067] Establish the relationship between the call model and the condition state value and associate them.

[0068] When the complementary calling requirement of the program is to obtain forklift positioning information, determining that the multimodal information is video stream information and warehouse forklift RFID positioning information;

[0069] The calling model corresponding to the warehouse forklift RFID positioning information is a positioning data acquisition model;

[0070] The calling models corresponding to the video stream information are a video stream acquisition model, a video stream extraction model, an information comparison model and an image recognition and positioning analysis model.

[0071] S102, obtaining the program complementary calling requirements of the current business scenario and executing the corresponding program complementary calling rules;

[0072] Specifically, executing the corresponding program complementary calling rules includes:

[0073] Execute the complementary calling requirements of the program in the current business scenario to obtain the program corresponding to the multimodal data, execute the calling model associated with the program, obtain the result value generated by the calling model, obtain the conditional state value corresponding to the result value, call the calling model corresponding to the conditional state value, and initiate the corresponding corresponding business instructions based on the result value generated by the final calling model, including initiating instructions to the IoT device or transmitting interface data to the system;

[0074] Based on the program start of the route, at least one conditional state value is defined, at least one call model is associated with each conditional state value, and at least one call model is associated with each result state value. If no call model is associated, it means that the program processing of the route is completed.

[0075] Obtaining a conditional status value for program startup, determining whether the conditional status value has an associated call model, and if so, generating a call instruction to call the call model associated with the conditional status value;

[0076] The above steps are repeated until there is no associated calling model for the conditional state value.

[0077] Obtaining a result state value generated by the calling model, determining whether the result state value has an associated calling model, and if so, generating a calling instruction to call the calling model associated with the result state value;

[0078] The above steps are repeated until there is no associated calling model for the result state value.

[0079] An embodiment:

[0080] In the warehouse intelligent management system, based on the scenario of warehouse intelligent management, a multimodal configuration call management architecture configuration is constructed. For example, based on the program startup for obtaining forklift positioning data, when the conditional status value is 1, the data acquisition model for obtaining RFID positioning data is configured, and the result status values ​​1 and 0 are defined based on the data acquisition model of RFID positioning data. For the result status value 1 (successful acquisition of RFID information), the positioning analysis model is associated to obtain the forklift positioning information; for the result status value 0 (failed to obtain RFID information), the video stream acquisition model is associated, and the result status values ​​2 and 3 are defined based on the video stream acquisition model. For the result status value 2, the video stream extraction model is associated, and the video stream extraction model is defined. The model defines result status values ​​4, 5, and 6. For result status value 4 (indicating that the image information extraction failed, is unclear, cannot be identified, etc., and the data stream needs to be collected again), the video stream collection model is associated; for result status value 5 (indicating that the image information extraction was successful, but the forklift information was not extracted), the video stream collection model is associated; for result status value 6 (indicating that the extracted image information includes the forklift), the information comparison model is associated, and the information comparison model result status values ​​7 and 8 are defined. For 7 (the forklift compared is not the forklift corresponding to the current RFID), the video stream collection model is associated; for 8 (the forklift compared is the forklift corresponding to the current RFID), the image recognition positioning analysis model is associated to obtain the forklift positioning information.

[0081] By using video stream image recognition technology to compensate for the failure to obtain forklift positioning information in time due to environmental influences in the warehouse, the forklift positioning information can be obtained by calling video stream image recognition. At the same time, the RFID positioning information of the cargo forklift is used first, which makes up for the problem that the pure use of video stream image recognition technology requires a lot of computing power costs.

[0082] The intelligent management method based on multimodal information integration processing also includes constructing a multimodal knowledge graph based on the multimodal information, and complementarily calling features corresponding to the requirements through the multimodal knowledge graph completion program.

[0083] Based on the execution of the multimodal call management architecture, the information fusion of different modes is realized, and an applicable knowledge graph is formed, which makes up for the knowledge graph data based on single-modal data. For example, based on multimodal information such as image information, audio information, and text information, a multimodal configuration call model based on multiple models is constructed, and then the information extracted based on these multimodal information is integrated to form an applicable knowledge graph. To a certain extent, it reduces resource expenditure, realizes technical supplementation, and improves data knowledge graphs.

[0084] The intelligent management method based on multimodal information integration processing includes but is not limited to the above applications. By integrating the effective information extracted by multimodal fusion technology, a complete knowledge graph is constructed and applied to different knowledge fields, technical fields and business fields.

[0085] Figure 2 FIG. 1 is a flow chart of an embodiment of an intelligent management system based on multimodal information integration processing of the present invention; Figure 2 As shown, an intelligent management system based on multimodal information integration processing provided by an embodiment of the present invention includes the following steps:

[0086] A configuration module 10 is used to pre-configure program complementary calling rules based on different business scenarios, wherein the program complementary calling rules include configuring the program complementary calling requirements of each business scenario to obtain the program corresponding to the multimodal data, the number of the program is one or more, configuring the calling model associated with each program, configuring the conditional state value for the result value generated by the calling model, and configuring the calling model corresponding to each conditional state value, the number of the calling models is one or more;

[0087] The configuration module 10 is also used for:

[0088] Based on business experience, conditional status values ​​are configured for the result values ​​of the calling model, or based on the historical data of the program complementary calling rules corresponding to the entire business scenario, automatic parameter adjustment configuration is performed to configure conditional status values ​​for the result values ​​of the calling model, wherein one calling model can generate one or more result values, and multiple calling models can generate one or more result values, and one or more conditional status values ​​are configured for one or more result values;

[0089] The business meaning corresponding to the result value generated by the calling model is defined through the result indicator.

[0090] At least one call model is associated with each condition state value.

[0091] Establishing a database, wherein the database includes conditional state values ​​and a call model corresponding to each conditional state value;

[0092] Establish the relationship between the result indicator and the condition status value and make associations;

[0093] Establish the relationship between the call model and the condition state value and associate them.

[0094] An acquisition module 20 is used to acquire the program complementary calling requirements of the current business scenario;

[0095] An execution module 30, used for executing corresponding program complementary calling rules;

[0096] The execution module 30 is also used for:

[0097] When the complementary calling requirement of the program is to obtain forklift positioning information, determining that the multimodal information is video stream information and warehouse forklift RFID positioning information;

[0098] The calling model corresponding to the warehouse forklift RFID positioning information is a positioning data acquisition model;

[0099] The calling models corresponding to the video stream information are a video stream acquisition model, a video stream extraction model, an information comparison model and an image recognition and positioning analysis model.

[0100] The execution module 30 includes a multimodal data acquisition module 301, a model calling module 302 and a program driving module 303;

[0101] The multimodal data acquisition module 301 is used to execute the program complementary call requirement of the current business scenario to acquire the program corresponding to the multimodal data;

[0102] The program driving module 303 initiates corresponding business instructions based on the result value generated by the final call model, including initiating instructions to the IoT device or transmitting interface data to the system;

[0103] The model calling module 302 is used to execute the calling model associated with the program, obtain the result value generated by the calling model, obtain the condition state value corresponding to the result value, and call the calling model corresponding to the condition state value;

[0104] Obtaining the result value generated by the calling model, determining whether the result value has a configuration condition state value, and when the result value generated by the calling model has a configuration condition state value, generating a calling instruction to call the calling model associated with the condition state value;

[0105] When the result value generated by the calling model does not have the configuration condition status value, no calling instruction is generated and the calling model is no longer called.

[0106] The knowledge graph construction module 40 is used to construct a multimodal knowledge graph based on the multimodal information, and to complementarily call the features corresponding to the requirements through the multimodal knowledge graph completion program.

[0107] Figure 4 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 4 As shown, the electronic device 50 includes: a processor 501 (processor), a memory 502 (memory) and a bus 503;

[0108] The processor 501 and the memory 502 communicate with each other via the bus 503;

[0109] The processor 501 is used to call the program instructions in the memory 502 to execute the methods provided by the above-mentioned method embodiments, for example, including: pre-configuring program complementary calling rules based on different business scenarios, wherein the program complementary calling rules include configuring the program complementary calling requirements of each business scenario to obtain the program corresponding to the multimodal data, the number of the programs is one or more, configuring the calling model associated with each program, configuring the conditional state value for the result value generated by the calling model, and configuring the calling model corresponding to each conditional state value, the number of the calling models is one or more; obtaining the program complementary calling requirements of the current business scenario, and executing the corresponding program complementary calling rules; executing the corresponding program complementary calling rules includes: executing the program complementary calling requirements of the current business scenario to obtain the program corresponding to the multimodal data, executing the calling model associated with the program, obtaining the result value generated by the calling model, obtaining the conditional state value corresponding to the result value, calling the calling model corresponding to the conditional state value, and initiating corresponding corresponding business instructions based on the result value generated by the final calling model, including initiating instructions to the IoT device or transmitting interface data to the system.

[0110] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the methods provided by the above-mentioned method embodiments, for example, including: pre-configuring program complementary calling rules based on different business scenarios, wherein the program complementary calling rules include configuring the program complementary calling requirements of each business scenario to obtain the program corresponding to the multimodal data, the number of the programs is one or more, configuring the calling model associated with each program, configuring the conditional state value for the result value generated by the calling model, and configuring the calling model corresponding to each conditional state value, the number of the calling models is one or more; obtaining the program complementary calling requirements of the current business scenario, and executing the corresponding program complementary calling rules; executing the corresponding program complementary calling rules includes: executing the program complementary calling requirements of the current business scenario to obtain the program corresponding to the multimodal data, executing the calling model associated with the program, obtaining the result value generated by the calling model, obtaining the conditional state value corresponding to the result value, calling the calling model corresponding to the conditional state value, and initiating corresponding corresponding business instructions based on the result value generated by the final calling model, including initiating instructions to the IoT device or transmitting interface data to the system.

[0111] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0114] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. An intelligent management method based on multimodal information integration processing, characterized in that: The intelligent management method based on multimodal information integration processing specifically includes: Pre-configure program complementary calling rules based on different business scenarios, wherein the program complementary calling rules include configuring the program complementary calling requirements of each business scenario to obtain the program corresponding to the multimodal data, the number of the program is one or more, configuring the calling model associated with each program, configuring the conditional state value for the result value generated by the calling model, configuring the calling model corresponding to each conditional state value, the number of the calling models is one or more; Obtain the program complementary calling requirements of the current business scenario and execute the corresponding program complementary calling rules; The corresponding program complementary calling rules are as follows: Execute the program complementary call requirements of the current business scenario to obtain the program corresponding to the multimodal data, execute the call model associated with the program, obtain the result value generated by the call model, obtain the conditional status value corresponding to the result value, call the call model corresponding to the conditional status value, and based on the result value generated by the final call model, initiate corresponding business instructions, including initiating instructions to the Internet of Things device or transmitting interface data to the system.

2. The intelligent management method based on multimodal information integration processing according to claim 1 is characterized in that: The executing corresponding program complementary calling rules also includes: When the complementary calling requirement of the program is to obtain forklift positioning information, determining that the multimodal information is video stream information and warehouse forklift RFID positioning information; The calling model corresponding to the warehouse forklift RFID positioning information is a positioning data acquisition model; The calling models corresponding to the video stream information are a video stream acquisition model, a video stream extraction model, an information comparison model and an image recognition and positioning analysis model.

3. The intelligent management method based on multimodal information integration processing according to claim 1 is characterized in that: The pre-configured complementary calling rules of programs based on different business scenarios include: Based on business experience, conditional status values ​​are configured for the result values ​​of the calling model, or based on the historical data of the program complementary calling rules corresponding to the entire business scenario, automatic parameter adjustment configuration is performed to configure conditional status values ​​for the result values ​​of the calling model, wherein one calling model can generate one or more result values, and multiple calling models can generate one or more result values, and one or more conditional status values ​​are configured for one or more result values; The business meaning corresponding to the result value generated by the calling model is defined through the result indicator.

4. The intelligent management method based on multimodal information integration processing according to claim 3 is characterized in that: The pre-configuration of complementary calling rules of programs based on different business scenarios also includes: At least one call model is associated with each condition state value.

5. The intelligent management method based on multimodal information integration processing according to claim 4 is characterized in that: The pre-configuration of complementary calling rules of programs based on different business scenarios also includes: Establishing a database, wherein the database includes conditional state values ​​and a call model corresponding to each conditional state value; Establish the relationship between the result indicator and the condition status value and make the association; Establish the relationship between the call model and the condition state value and associate them.

6. The intelligent management method based on multimodal information integration processing according to claim 5 is characterized in that: The obtaining of the program complementary calling requirements of the current business scenario and executing the corresponding program complementary calling rules include: Obtaining the result value generated by the calling model, determining whether the result value has a configuration condition state value, and when the result value generated by the calling model has a configuration condition state value, generating a calling instruction to call the calling model associated with the condition state value; When the result value generated by the calling model does not have the configuration condition status value, no calling instruction is generated and the calling model is no longer called.

7. The intelligent management method based on multimodal information integration processing according to claim 1 is characterized in that: The intelligent management method based on multimodal information integration processing also includes: A multimodal knowledge graph is constructed based on the multimodal information, and features corresponding to the requirements are complementarily called through the multimodal knowledge graph completion program.

8. An intelligent management system based on multimodal information integration processing, characterized in that: include: A configuration module, used to pre-configure program complementary calling rules based on different business scenarios, wherein the program complementary calling rules include configuring the program complementary calling requirements of each business scenario to obtain the program corresponding to the multimodal data, the number of the program is one or more, configuring the calling model associated with each program, configuring the conditional state value for the result value generated by the calling model, and configuring the calling model corresponding to each conditional state value, the number of the calling models is one or more; The acquisition module is used to obtain the program complementary call requirements of the current business scenario; An execution module, used for executing corresponding program complementary calling rules; The execution module includes a multimodal data acquisition module, a program driving module and a model calling module; The multimodal data acquisition module is used to execute the program complementary call requirements of the current business scenario to obtain the program corresponding to the multimodal data; The model calling module is used to execute the calling model associated with the program, obtain the result value generated by the calling model, obtain the condition state value corresponding to the result value, and call the calling model corresponding to the condition state value; The program driving module is used to initiate corresponding business instructions based on the result value generated by the final call model, including initiating instructions to the Internet of Things device or transmitting interface data to the system.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.