Heterogeneous storage global management scheduling method and system, electronic device and storage medium

Abstract

The application discloses a heterogeneous storage global management scheduling method and system, electronic equipment and a storage medium, and comprises the following steps: obtaining to-be-stored files; obtaining server resource information, platform capability information and storage resource information, determining target storage types and target storage positions of the to-be-stored files according to the server resource information, the platform capability information and the storage resource information; determining target storage protocols corresponding to the to-be-stored files according to the target storage types, and calling corresponding storage resource access services according to the target storage positions; and performing file uploading storage on the to-be-stored files by using the storage resource access services according to the target storage protocols. The application realizes automatic remote scheduling of storage resources under the premise of guaranteeing high availability of the storage resources, enhances disaster recovery capability of data storage, improves efficiency and reliability of heterogeneous data storage, and can be applied to the technical field of data storage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, and in particular to a heterogeneous storage global management and scheduling method and system, an electronic device and a storage medium. BACKGROUND

[0002] There are many video storage scenarios in the field of video monitoring. With the development of technology iteration and cloud technology, the storage medium for video storage has developed from the original disk array to object storage, but both the disk array and the object storage have the bottleneck of storage resource across the network.

[0003] For multiple heterogeneous storage, the previous storage management system cannot achieve centralized management and unified scheduling, and often needs to deploy different systems or even integrate different manufacturers' systems. In particular, in the storage scenario of cross-network scheduling of object storage and local storage resources of Tianyi cloud and cloud private network, the traditional storage management system cannot achieve effective compatibility, affecting the efficiency and reliability of data heterogeneous storage. In addition, the heterogeneous management of multiple types of storage resources across the network increases the uncertainty of problem troubleshooting for operation and maintenance personnel, affecting the high availability of storage resources. SUMMARY

[0004] The present application aims to at least partially solve one of the problems in the prior art.

[0005] To this end, one object of the present application is to provide a heterogeneous storage global management and scheduling method, which improves the efficiency and reliability of data heterogeneous storage and guarantees the high availability of storage resources.

[0006] Another object of the present application is to provide a heterogeneous storage global management and scheduling system.

[0007] In order to achieve the above technical purpose, the technical solution adopted by the present application comprises:

[0008] In a first aspect, the present application provides a heterogeneous storage global management and scheduling method, comprising the following steps:

[0009] Obtaining files to be stored;

[0010] Obtaining server resource information, platform capability information and storage resource information, and determining the target storage type and target storage location of each file to be stored according to the server resource information, the platform capability information and the storage resource information;

[0011] Determining the target storage protocol corresponding to each file to be stored according to the target storage type, and calling the corresponding storage resource access service according to the target storage location;

[0012] According to the target storage protocol, the file uploading storage is performed on each of the to-be-stored files by using the storage resource access service.

[0013] Further, in an embodiment of the present application, the step of obtaining the to-be-stored file specifically comprises:

[0014] According to the data stream address, the to-be-stored data stream is obtained.

[0015] The to-be-stored data stream is parsed to obtain a plurality of data frames, and the encoding format of each data frame is determined.

[0016] According to the encoding format, the data frames are classified, and the data frames of the same encoding format are encapsulated to obtain a plurality of to-be-stored files.

[0017] Further, in an embodiment of the present application, the step of obtaining the server resource information, the platform capability information and the storage resource information specifically comprises:

[0018] The server resource information is obtained through a server resource API, and the server resource information includes the server configuration and the server usage state of each server.

[0019] The platform capability information is obtained through a platform capability API, and the platform capability information includes the real-time service state, the home server, the real-time network environment and the real-time network bandwidth of each platform capability.

[0020] The storage resource information is obtained through a storage resource API, and the storage resource information includes the storage resource type, the storage space size, the current storage state, the storage stream real-time speed and the IPOS index of each storage resource.

[0021] The server resource API, the platform capability API and the storage resource API are all connected with a front-end page, and the front-end page is used to display and configure the server resource information, the platform capability information and the storage resource information.

[0022] Further, in an embodiment of the present application, the step of determining the target storage type and the target storage location of each to-be-stored file according to the server resource information, the platform capability information and the storage resource information specifically comprises:

[0023] According to the storage stream real-time speed and the IPOS index, the available storage stream speed of each storage resource is determined, and according to the storage space size and the current storage state, the available storage space of each storage resource is determined.

[0024] According to the available storage stream speed and the available storage space, a storage resource is allocated to each of the to-be-stored files, and a first storage resource corresponding to each of the to-be-stored files is determined.

[0025] According to the storage resource type corresponding to the first storage resource, the target storage type is determined.

[0026] According to the real-time service state, the real-time network environment and the real-time network bandwidth, a plurality of first platform capabilities corresponding to the first storage resource are determined.

[0027] According to the server configuration, the server usage state and the home server corresponding to the first platform capability, a plurality of first servers corresponding to the first storage resource are determined, and the target storage location is determined according to the first servers.

[0028] Further, in an embodiment of the present application, the step of determining the target storage protocol corresponding to each of the to-be-stored files according to the target storage type, and calling the corresponding storage resource access service according to the target storage location, specifically comprises:

[0029] According to the target storage type, a plurality of target storage protocols corresponding to the target storage type are selected from a preset storage protocol library.

[0030] According to the target storage location, the storage path information of the to-be-stored file is determined, and a plurality of storage resource access services corresponding to the storage path information are selected from a preset storage resource access service library.

[0031] Further, in an embodiment of the present application, the step of storing each of the to-be-stored files according to the target storage protocol and using the storage resource access service, specifically comprises:

[0032] According to the target storage protocol, a data transmission channel of the to-be-stored file is established.

[0033] According to the target storage location, a first server for storing the to-be-stored file is determined, a connection relationship between the data transmission channel and the first server is established by using the storage resource access service, and a plurality of corresponding storage partitions are configured on the first server.

[0034] The to-be-stored file is uploaded to the storage partition through the data transmission channel for file storage.

[0035] Further, in an embodiment of the present application, the heterogeneous storage global management and scheduling method further comprises the following steps:

[0036] acquire an upload delay and / or a stream file write error rate of the to-be-stored file;

[0037] when the upload delay is greater than or equal to a preset first threshold value, and / or the stream file write error rate is greater than or equal to a preset second threshold value, determine a retransmission storage type and a retransmission storage location of the to-be-stored file;

[0038] switch the to-be-stored file to a corresponding server according to the retransmission storage type and the retransmission storage location for file upload storage.

[0039] In a second aspect, an embodiment of the present application provides a heterogeneous storage global management scheduling system, comprising:

[0040] a file acquisition module configured to acquire to-be-stored files;

[0041] a storage type and storage location determination module configured to acquire server resource information, platform capability information and storage resource information, and determine target storage types and target storage locations of the to-be-stored files according to the server resource information, the platform capability information and the storage resource information;

[0042] a storage resource access service calling module configured to determine target storage protocols corresponding to the to-be-stored files according to the target storage types, and call corresponding storage resource access services according to the target storage locations;

[0043] a file storage module configured to perform file upload storage on the to-be-stored files according to the target storage protocols by using the storage resource access services.

[0044] In a third aspect, an embodiment of the present application provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the heterogeneous storage global management scheduling method in the first aspect.

[0045] In a fourth aspect, an embodiment of the present application further provides a storage medium, which is a computer readable storage medium, for computer readable storage, and the storage medium stores one or more programs, and the one or more programs are executable by one or more processors to realize the heterogeneous storage global management scheduling method in the first aspect.

[0046] The advantages and beneficial effects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood through the practice of the present application:

[0047] The embodiment of the present application acquires to-be-stored files, then acquires server resource information, platform capability information and storage resource information, determines target storage types and target storage positions of the to-be-stored files according to the server resource information, the platform capability information and the storage resource information, determines target storage protocols corresponding to the to-be-stored files according to the target storage types, and calls corresponding storage resource access services according to the target storage positions, and then can perform file uploading storage on the to-be-stored files according to the target storage protocols and by using the storage resource access services. The embodiment of the present application determines target storage types and target storage positions of to-be-stored files according to server resource information, platform capability information and storage resource information, and then determines target storage protocols and storage resource access services to perform uploading storage on the to-be-stored files, realizes calling of heterogeneous storage resources across networks and across resource types, realizes automatic remote scheduling of storage resources on the premise of guaranteeing high availability of the storage resources, enhances disaster recovery capability of data storage, and improves efficiency and reliability of heterogeneous data storage. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following introduces the drawings needed to be used in the embodiments of the present application. It should be understood that the drawings introduced below are merely for facilitating clear description of some embodiments in the technical solutions of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort on the premise of the drawings.

[0049] Figure 1 A step flow chart of the heterogeneous storage global management scheduling method provided by the embodiment of the present application;

[0050] Figure 2 A flow chart of step S101 provided by the embodiment of the present application;

[0051] Figure 3 A flow chart of step S102 provided by the embodiment of the present application;

[0052] Figure 4 Another flow chart of step S102 provided by the embodiment of the present application;

[0053] Figure 5 A flow chart of step S103 provided by the embodiment of the present application;

[0054] Figure 6 A flow chart of step S104 provided by the embodiment of the present application;

[0055] Figure 7 Another step flow chart of the heterogeneous storage global management scheduling method provided by the embodiment of the present application;

[0056] Figure 8 A schematic diagram of a storage process of a video data stream provided by an embodiment of the present application is shown in FIG. 3.

[0057] Figure 9 A schematic diagram of a structure of a heterogeneous storage global management and scheduling system provided by an embodiment of the present application is shown in FIG. 4.

[0058] Figure 10 A schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application is shown in FIG. 5. DETAILED DESCRIPTION

[0059] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which the same or similar elements or elements having the same or similar functions are denoted by the same or similar reference numerals throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and cannot be understood as limiting the present application. It should be noted that although functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the system schematic diagram or the order in the flowchart. For the step numbers in the following embodiments, they are only set for the purpose of facilitating the description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0060] In the description of the present application, the meaning of multiple is two or more, and if the first, the second is described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or implicitly indicating the sequence of indicated technical features. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0061] The heterogeneous storage global management and scheduling method provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application that implements the heterogeneous storage global management and scheduling method, but is not limited to the above forms.

[0062] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0063] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0064] like Figure 1 The diagram shown is a flowchart of one step of the heterogeneous storage global management and scheduling method provided in an embodiment of the present invention. (Refer to...) Figure 1 This invention provides a method for global management and scheduling of heterogeneous storage, specifically including the following steps:

[0065] S101. Obtain the file to be stored.

[0066] Specifically, this invention is applicable to application scenarios where multiple heterogeneous storage systems across networks and storage resource types are used in parallel and require management and flexible scheduling of multiple storage systems. For the data stream to be stored, it is parsed and encapsulated to obtain multiple files to be stored, and the storage resources required for each file to be stored and the corresponding storage location are determined, and then the heterogeneous storage of the files to be stored is realized.

[0067] like Figure 2 The diagram shown is a flowchart of step S101 provided in an embodiment of the present invention. (Refer to...) Figure 2As further optional implementation, the step of obtaining the to-be-stored file specifically comprises:

[0068] S1011, obtaining a to-be-stored data stream according to a data stream address;

[0069] S1012, performing parsing processing on the to-be-stored data stream to obtain a plurality of data frames, and determining an encoding format of each data frame;

[0070] S1013, classifying the data frames according to the encoding format, and performing encapsulation processing on the data frames of the same encoding format to obtain a plurality of to-be-stored files.

[0071] Specifically, taking video stream storage in the security field as an example, a to-be-stored video stream is obtained according to an access address of the video stream, then a plurality of video frames are obtained by performing parsing processing on the to-be-stored video stream, and an encoding format of each video frame is determined, then all the video frames are classified according to the encoding format of the video frames, and the video frames of the same encoding format are uniformly encapsulated to obtain to-be-stored files corresponding to different encoding formats, such as mp4 files, which need to be written into corresponding storage resources according to storage management requirements in the future.

[0072] In the embodiment of the application, the to-be-stored data stream obtained can be part of the content of the data stream corresponding to the data stream address, or all the content of the data stream corresponding to the data stream address. For example, a video of a preset time length can be cut from a data stream of a video format as a to-be-stored data stream, or a video in a time period defined by a start time and an end time can be cut from a data stream of a video format as a to-be-stored video stream.

[0073] S102, obtaining server resource information, platform capability information and storage resource information, and determining a target storage type and a target storage location of each to-be-stored file according to the server resource information, the platform capability information and the storage resource information.

[0074] Specifically, the embodiment of the application reasonably allocates storage resources used by each to-be-stored file according to the resource usage of each storage resource, each platform capability and each server, so as to realize balanced load of storage usage.

[0075] As shown in Figure 3 The flowchart of step S102 provided by the embodiment of the application is shown in Figure 3 As further optional implementation, the step of obtaining the server resource information, the platform capability information and the storage resource information specifically comprises:

[0076] S1021, obtaining server resource information through a server resource API, the server resource information including server configurations and server usage states of each server;

[0077] S1022, obtain platform capability information through the platform capability API, the platform capability information including real-time service states, home servers, real-time network environments and real-time network bandwidths of various platform capabilities;

[0078] S1023, obtain storage resource information through the storage resource API, the storage resource information including storage resource types, storage space sizes, current storage states, storage fluidity real-time speeds and IPOS indexes of various storage resources;

[0079] The server resource API, the platform capability API and the storage resource API are all connected with a front-end page, and the front-end page is used to display and configure the server resource information, the platform capability information and the storage resource information.

[0080] Specifically, the embodiment of the application provides a visual page of the server resource information, the platform capability information and the storage resource information through the means of connecting the front-end web page with the back-end API interface (including the server resource API, the platform capability API and the storage resource API), and provides a related configuration interface, wherein the resource information of each server includes server configuration and server use state, the related information of each platform capability includes real-time service states, home servers, real-time network environments and real-time network bandwidths, and the related information of each storage resource includes storage resource types, storage space sizes, current storage states, storage fluidity real-time speeds and IPOS indexes.

[0081] It can be understood that the storage resource types include at least one of file storage, object storage and block storage; the IOPS (Input / Output Per Second) index, i.e. the input / output quantity per second, is one of main indexes for measuring the performance of the storage resource, and can also represent the number of read / write requests that can be processed by the system per second.

[0082] As shown in FIG. 1, the embodiment of the application provides a server resource information display page 1, a platform capability information display page 2 and a storage resource information display page 3. Figure 4 As shown in FIG. 2, the embodiment of the application provides another flowchart of step S102, which is described with reference to FIG. 1. Figure 4 Further, as an optional implementation, the step of determining the target storage type and the target storage location of each to-be-stored file according to the server resource information, the platform capability information and the storage resource information specifically includes:

[0083] S1024, determining the available storage fluidity speed of each storage resource according to the storage fluidity real-time speed and the IPOS index, and determining the available storage space of each storage resource according to the storage space size and the current storage state;

[0084] S1025, performing storage resource allocation on each to-be-stored file according to the available storage fluidity speed and the available storage space, and determining the first storage resource corresponding to each to-be-stored file.

[0085] S1026. Determine the target storage type based on the storage resource type corresponding to the first storage resource;

[0086] S1027. Determine several first platform capabilities corresponding to the first storage resource based on the real-time service status, real-time network environment, and real-time network bandwidth.

[0087] S1028. Based on the server configuration, server usage status, and the server to which the first platform's capabilities belong, determine several first servers corresponding to the first storage resource, and determine the target storage location based on the first servers.

[0088] Specifically, in this embodiment of the invention, the available storage streaming speed for uploading files to be stored can be determined based on the real-time storage streaming speed and the IPOS index. The available storage space for storing files to be stored can be determined based on the storage space size and current storage status. Based on the available storage streaming speed, available storage space, file size of the files to be stored, transmission latency requirements, etc., storage resources can be rationally allocated to each file to be stored, achieving a balanced load on storage usage. After determining the first storage resource corresponding to each file to be stored, the target storage type (such as file storage, object storage, block storage, etc.) can be determined based on the storage resource type of the first storage resource. Simultaneously, several first platform capabilities required to use the first storage resource can be determined based on the real-time service status, real-time network environment, and real-time network bandwidth. Then, several first servers corresponding to the first storage resource are determined based on the home server corresponding to the first platform capability, as well as the server configuration and server usage status of each server. These first servers are the servers that provide storage resources and store the files to be stored.

[0089] S103. Determine the target storage protocol corresponding to each file to be stored according to the target storage type, and call the corresponding storage resource access service according to the target storage location.

[0090] Specifically, in order to better address the current situation of diverse and rapidly developing storage technologies, this embodiment of the invention, based on the principle of optimal resource utilization, achieves unified management and scheduling of various heterogeneous storage across networks by managing multiple storage protocols, enhances platform stability by improving the reuse of storage resources, and reduces the complexity and cost of storage management through unified and centralized management.

[0091] like Figure 5 The diagram shown is a flowchart of step S103 provided in an embodiment of the present invention. (Refer to...) Figure 5As further optional implementation, according to the target storage type, the target storage protocol corresponding to each file to be stored is determined, and the corresponding storage resource access service is called according to the target storage location, which specifically includes:

[0092] S1031, selecting a plurality of target storage protocols corresponding to the target storage type from a preset storage protocol library;

[0093] S1032, determining the storage path information of the file to be stored according to the target storage location, and selecting a plurality of storage resource access services corresponding to the storage path information from a preset storage resource access service library.

[0094] Specifically, the unified management of heterogeneous storage includes: 1) interfacing with file storage, object storage and other storage protocols, and supporting flexible switching of the corresponding storage protocol; 2) supporting file acquisition, deletion, deletion and other operations of file storage, object storage and other files.

[0095] The embodiment of the application pre-constructs a storage protocol library and a storage resource access service library, wherein the storage protocol library includes a first mapping relationship between a plurality of different storage types and corresponding target storage protocols, and based on the determined target storage type and the first mapping relationship, the target storage protocol to be used can be quickly determined; the storage resource access service library includes a second mapping relationship between a plurality of different storage paths and corresponding storage resource access services, based on the storage path information of the file to be stored determined by the target storage location, the storage resource access service to be called can be quickly determined in combination with the second mapping relationship.

[0096] S104, according to the target storage protocol, using the storage resource access service to perform file upload storage on each file to be stored.

[0097] Specifically, the embodiment of the application is provided with a capability module for storage interfacing, which interfaces with storage resources such as file storage, object storage, block storage and the like, and the capability module is composed of a storage resource access service, a storage resource basic performance acquisition service and a storage resource unified scheduling output API service, which is downward compatible with various storage resource APIs and provides a unified scheduling API in a structured manner.

[0098] As shown in Figure 6 is a flowchart of step S104 provided by the embodiment of the application, referring to Figure 6 , as further optional implementation, according to the target storage protocol, using the storage resource access service to perform file upload storage on each file to be stored, which specifically includes:

[0099] S1041, establishing a data transmission channel for the file to be stored according to the target storage protocol;

[0100] S1042, determine a first server for storing the to-be-stored file according to the target storage location, establish a connection relationship with the first server through a data transmission channel using a storage resource access service, and configure a plurality of corresponding storage partitions on the first server;

[0101] S1043, upload the to-be-stored file to the storage partition through the data transmission channel for file storage.

[0102] Specifically, a data transmission channel is established according to the determined target storage protocol and the packaging format of the to-be-stored file, and the transmission protocol, transmission speed threshold and data verification mechanism of the data transmission channel are determined; the corresponding first server is determined according to the determined target storage location, the connection relationship between the data transmission channel and the first server is established through the storage resource access service determined in the foregoing step, and a plurality of storage partitions are configured on the first server to provide storage resources (store the to-be-stored file).

[0103] As Figure 7 Another step flowchart of the heterogeneous storage global management and scheduling method provided by the embodiment of the application is shown in the figure, and Figure 7 Further, as an optional implementation, the heterogeneous storage global management and scheduling method further includes the following steps:

[0104] S105, obtain the upload delay and / or stream file write error rate of the to-be-stored file;

[0105] S106, when the upload delay is greater than or equal to a preset first threshold, and / or the stream file write error rate is greater than or equal to a preset second threshold, determine the retransmission storage type and the retransmission storage location of the to-be-stored file;

[0106] S107, switch the to-be-stored file to the corresponding server for file upload storage according to the retransmission storage type and the retransmission storage location.

[0107] Specifically, the embodiment of the application further includes resource monitoring and alarm, switching processing of heterogeneous storage, real-time statistics of file operation results through a program, alarm for abnormal operation, real-time monitoring of the rate of file writing into the storage system, system alarm for writing delay and other phenomena, alarm and automatic switching of storage for storage abnormal disconnection and hanging, and realization of statistics of the size, remaining amount and other resource information of the current use of each storage, automatic adjustment of storage availability according to the state of the object storage.

[0108] The specific collection means of the performance index usage of the system-related server in the embodiment of the application is as follows:

[0109] iftop is simulated through a network card, network card data packet statistics are captured using a pcap library, network traffic monitoring is performed, and network types are distinguished according to the basic performance of the storage resource.

[0110] The real-time usage of the CPU is obtained from the / proc / stat file in the Linux system, the usage of the local disk and the mounted distributed storage disk is obtained through the statfs system call, and the current usage of the storage resource, the stream file writing rate, the stream file writing error rate, and the uploading delay are obtained through the unified scheduling API.

[0111] The file uploading delay and the stream file writing error rate obtained through monitoring are compared with the preset threshold value respectively to determine whether the file uploading is abnormal, and when the abnormality occurs, the normal storage is reasonably called to avoid data loss caused by the uploading delay and the like; according to the real-time monitoring of the storage state, when the storage abnormality is disconnected or hung, the file operation is automatically switched to the normal storage.

[0112] In some optional embodiments, the embodiment of the present application supports the docking expansion of the new storage, optimizes the equipment at the system architecture level, and the subsequent new type storage can be quickly docked and joined in the management and control; in addition, the embodiment of the present application can realize the low-cost delivery of operation and maintenance management through the visual storage type management and control and the threshold value adjustment.

[0113] The method steps of the embodiment of the present application are described above. It can be understood that the embodiment of the present application determines the target storage type and the target storage location of the to-be-stored file according to the server resource information, the platform capability information and the storage resource information, and then determines the target storage protocol and the storage resource access service to upload and store the to-be-stored file, realizes the calling of the heterogeneous storage resource across the network and across the resource type, realizes the automatic off-site scheduling of the storage resource under the premise of guaranteeing the high availability of the storage resource, enhances the disaster recovery capability of the data storage, and improves the efficiency and reliability of the heterogeneous data storage.

[0114] The implementation scenario of the embodiment of the present application is exemplified below in combination with a specific embodiment.

[0115] Due to the particularity of the security field, the completeness and safety of the video file are required very high, and the demand for storage resources is huge. The previous system cannot be uniformly adapted and scheduled for object storage, object storage and the like, the new and old systems are difficult to be compatible, and there is a network isolation problem between the non-cloud and cloud storage resources. These problems not only cause the waste of servers, manpower and other resources in management, but also are not conducive to dynamic expansion due to poor maintainability of the system, and the single storage node brings hidden troubles in stability. Based on the embodiment of the present application, the video file can be effectively and stably managed and stored, and the heterogeneous storage scheduling of the video file in the large security background is realized.

[0116] For example, Figure 8As shown is a storage process schematic diagram of the video data stream provided by the embodiment of the present application, and the related process is as follows: the storage management service accesses the write-in service through the rtpps national standard, the write-in service returns a listening port to the storage management service; the storage management service pushes the stream to the sip system call device, the sip system returns an ssrc identifier to the storage management service; the storage management service sets the ssrc identifier; the sip system returns the national standard video ps stream to the write-in service, and the write-in service returns stream analysis success to the storage management service; the storage management service starts video stream recording, the write-in service records the video stream according to a fragmentation strategy, and writes the fragmentation information into a queue service; the storage management service reads the fragmentation information from the queue service, processes the fragmentation information into a database, and uploads the video to the storage through a video upload / delete service, and returns the upload result to the storage management service.

[0117] With the rapid development of storage technology, new storage protocols are constantly derived, and the embodiment of the present application reserves space for the docking and expansion of new storage, which can be quickly docked; with the popularity of cloud resources, more and more systems need to consider reducing resource cost and maintenance cost by uploading resources to the cloud, and it is very common for old hardware storage and new storage to have different storage architectures, and the present application can effectively solve such problems and realize effective compatibility of storage.

[0118] The previous system only supports single-architecture storage scheduling, but the same storage may have a large area of collapse caused by specific factors, and does not have the ability of quick recovery of system services, the embodiment of the present application realizes heterogeneous calling across networks and resource types based on the specific requirements of data integrity in the security field, realizes automatic remote scheduling under the premise of high availability of system services, and indirectly provides disaster recovery capability of video monitoring storage, and improves system stability.

[0119] In addition, the embodiment of the present application can uniformly manage the capacity, mounting condition and file uploading condition of the storage, timely alarm for calling exceptions of the storage to ensure that the operation and maintenance personnel respond and handle problems in time, and switch the file to the normal storage for uploading to ensure the stability of data saving.

[0120] As Figure 9 As shown is a structure schematic diagram of the heterogeneous storage global management scheduling system provided by the embodiment of the present application, referring to Figure 9 , the embodiment of the present application provides a heterogeneous storage global management scheduling system, comprising:

[0121] a file acquisition module, configured to acquire a to-be-stored file;

[0122] The storage type and storage location determination module is configured to acquire server resource information, platform capability information and storage resource information, and determine target storage types and target storage locations of the to-be-stored files according to the server resource information, the platform capability information and the storage resource information.

[0123] The storage resource access service calling module is configured to determine target storage protocols corresponding to the to-be-stored files according to the target storage types, and call corresponding storage resource access services according to the target storage locations.

[0124] The file storage module is configured to perform file uploading storage on the to-be-stored files by using the storage resource access services according to the target storage protocols.

[0125] The content in the method embodiments is applicable to the system embodiments, the system embodiments specifically implement the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0126] The embodiment of the application further provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the heterogeneous storage global management and scheduling method. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0127] As shown in Figure 10 Fig. 1 is a hardware structure schematic diagram of an electronic device provided by an embodiment of the application, and Figure 10 The embodiment of the application provides an electronic device, which comprises:

[0128] The processor 1001 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute a related program to implement the technical solutions provided by the embodiment of the application.

[0129] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 1002 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 1002 and are called and executed by the processor 1001 to implement the heterogeneous storage global management scheduling method of the embodiments of the present application;

[0130] The input / output interface 1003 is used to realize information input and output;

[0131] The communication interface 1004 is used to realize the communication interaction between the device and other devices, which can realize communication through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0132] The bus 1005 transmits information between various components (such as the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004) of the device;

[0133] Among them, the processor 1001, the memory 1002, the input / output interface 1003 and the communication interface 1004 realize the communication connection between each other in the device through the bus 1005.

[0134] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium, used for computer readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned heterogeneous storage global management scheduling method.

[0135] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0136] The embodiment of the present application further discloses a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computer device to perform the method shown in the embodiment of the present application. Figure 1 The method shown in the embodiment of the present application.

[0137] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially concurrently or the blocks mentioned above can be executed in reverse order at times. Furthermore, the embodiments presented and described in the flowcharts of the present application are provided by way of example only. The disclosed methods are not limited to the operations and logical flows presented in this specification. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of a larger operation are executed independently.

[0138] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the functions and / or features described above can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules, unless otherwise specified. It is also to be understood that detailed discussion of the actual implementation of each module is unnecessary to an understanding of the present application. Rather, the actual implementation is within the routine skill of engineers familiar with the attributes, functions and internal relationships of the various functional modules disclosed in the devices herein. Accordingly, the present application is not limited to purely hardware implementations, but also encompasses software implementations, including program instructions stored on computer-readable media. Moreover, the disclosed specific concepts are merely illustrative and are not intended to limit the scope of the present application, which is defined by the full scope of the appended claims and equivalents thereof.

[0139] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or parts of the technical solutions that essentially contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described above in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0140] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a list of executable instructions for implementing logic functions, which can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instructions execution systems, apparatus or devices. For the purpose of this specification, the "computer-readable medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0141] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the above programs are printed, because the programs can be obtained electronically, for example, by optical scanning of the paper or other medium, followed by editing, interpreting or otherwise processing, if necessary, in other suitable ways, to be electronically obtained and then stored in the computer memory.

[0142] It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or their equivalents, can be employed for implementation: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0143] In the above description of the present specification, the description of the terms "one embodiment / one example", "another embodiment / another example", or "certain embodiments / certain examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0144] Although the embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made thereto without departing from the principles and spirit of the present application, the scope of which is defined by the claims and their equivalents.

[0145] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the above-described embodiments, and those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present application, and these equivalent modifications or substitutions are included in the scope defined by the claims of the present application.

Claims

1. A heterogeneous storage global management scheduling method, characterized in that, The method comprises the following steps: obtaining a file to be stored; obtaining server resource information, platform capability information and storage resource information, and determining a target storage type and a target storage location of each of the files to be stored according to the server resource information, the platform capability information and the storage resource information; determining a target storage protocol corresponding to each of the files to be stored according to the target storage type, and calling a corresponding storage resource access service according to the target storage location; performing file upload storage on each of the files to be stored according to the target storage protocol and using the storage resource access service; the step of obtaining server resource information, platform capability information and storage resource information specifically comprises: obtaining the server resource information through a server resource API, wherein the server resource information comprises server configurations and server usage states of each server; obtaining the platform capability information through a platform capability API, wherein the platform capability information comprises real-time service states, home servers, real-time network environments and real-time network bandwidths of each platform capability; obtaining the storage resource information through a storage resource API, wherein the storage resource information comprises storage resource types, storage space sizes, current storage states, storage fluidization real-time speeds and IPOS indicators of each storage resource; wherein the server resource API, the platform capability API and the storage resource API are connected with a front-end page, and the front-end page is used to display and configure the server resource information, the platform capability information and the storage resource information; the step of determining a target storage type and a target storage location of each of the files to be stored according to the server resource information, the platform capability information and the storage resource information specifically comprises: determining available storage fluidization speeds of each of the storage resources according to the storage fluidization real-time speeds and the IPOS indicators, and determining available storage spaces of each of the storage resources according to the storage space sizes and the current storage states; allocating storage resources to each of the files to be stored according to the available storage fluidization speeds and the available storage spaces, and determining first storage resources corresponding to each of the files to be stored; determining the target storage type according to a storage resource type corresponding to the first storage resource; determining a plurality of first platform capabilities corresponding to the first storage resource according to real-time service states, real-time network environments and real-time network bandwidths; determining a plurality of first servers corresponding to the first storage resource according to server configurations, server usage states and home servers corresponding to the first platform capabilities, and determining the target storage location according to the first servers.

2. The heterogeneous storage global management scheduling method according to claim 1, wherein, the step of obtaining a file to be stored specifically comprises: obtaining a data stream to be stored according to a data stream address; performing analysis processing on the data stream to be stored to obtain a plurality of data frames, and determining encoding formats of each of the data frames; Classify the data frames according to the encoding format, and perform encapsulation processing on the data frames of the same encoding format to obtain a plurality of the to-be-stored files.

3. The heterogeneous storage global management scheduling method of claim 1, wherein, The step of determining the target storage protocol corresponding to each of the to-be-stored files according to the target storage type, and calling the corresponding storage resource access service according to the target storage location, specifically comprises: Selecting the corresponding target storage protocol from a preset storage protocol library according to the target storage type; According to the target storage location, the storage path information of the to-be-stored file is determined, and the corresponding storage resource access service is selected from a preset storage resource access service library according to the storage path information.

4. The heterogeneous storage global management scheduling method of claim 1, wherein, The step of performing file upload storage on each of the to-be-stored files by using the storage resource access service according to the target storage protocol, specifically comprises: According to the target storage protocol, a data transmission channel of the to-be-stored file is established; According to the target storage location, a first server for storing the to-be-stored file is determined, a connection relationship between the data transmission channel and the first server is established by using the storage resource access service, and a plurality of corresponding storage partitions are configured on the first server; The to-be-stored file is uploaded to the storage partition through the data transmission channel for file storage.

5. The heterogeneous storage global management scheduling method according to any one of claims 1 to 4, characterized in that, The heterogeneous storage global management and scheduling method further comprises the following steps: Obtain the upload delay and / or stream file write error rate of the to-be-stored file; When the upload delay is greater than or equal to a preset first threshold, and / or the stream file write error rate is greater than or equal to a preset second threshold, the retransmission storage type and the retransmission storage location of the to-be-stored file are determined; According to the retransmission storage type and the retransmission storage location, the to-be-stored file is switched to the corresponding server for file upload storage.

6. A heterogeneous storage global management scheduling system, characterized in that, Comprise: File acquisition module, for obtaining to-be-stored file; Storage type and storage location determination module, for obtaining server resource information, platform capability information and storage resource information, and determining the target storage type and the target storage location of each of the to-be-stored files according to the server resource information, the platform capability information and the storage resource information; Storage resource access service calling module, for determining the target storage protocol corresponding to each of the to-be-stored files according to the target storage type, and calling the corresponding storage resource access service according to the target storage location; File storage module, for performing file upload storage on each of the to-be-stored files by using the storage resource access service according to the target storage protocol; The server resource information, platform capability information and storage resource information are obtained, which specifically comprises: The server resource information is obtained through a server resource API, and the server resource information includes the server configuration and server usage state of each server; The platform capability information is obtained through a platform capability API, and the platform capability information includes the real-time service state, home server, real-time network environment and real-time network bandwidth of each platform capability; The storage resource information is obtained through a storage resource API, and the storage resource information includes storage resource types, storage space sizes, current storage states, storage fluidity real-time speeds, and IPOS indexes of respective storage resources; The server resource API, the platform capability API, and the storage resource API are connected with a front-end page, and the front-end page is used to display and configure the server resource information, the platform capability information, and the storage resource information; The target storage type and the target storage location of each of the to-be-stored files are determined according to the server resource information, the platform capability information, and the storage resource information, and the determination specifically includes: The available storage fluidity speeds of the respective storage resources are determined according to the storage fluidity real-time speeds and the IPOS indexes, and the available storage spaces of the respective storage resources are determined according to the storage space sizes and the current storage states; The respective to-be-stored files are allocated to storage resources according to the available storage fluidity speeds and the available storage spaces, to determine first storage resources corresponding to the respective to-be-stored files; The target storage type is determined according to the storage resource type corresponding to the first storage resource; A plurality of first platform capabilities corresponding to the first storage resource are determined according to the real-time service state, the real-time network environment, and the real-time network bandwidth; A plurality of first servers corresponding to the first storage resource are determined according to the first platform capabilities, the server configuration, the server usage state, and the home server, and the target storage location is determined according to the first servers.

7. An electronic device, comprising: The electronic device includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the steps of the heterogeneous storage global management scheduling method according to any one of claims 1 to 5.

8. A storage medium, the storage medium being a computer readable storage medium for computer readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the steps of the heterogeneous storage global management scheduling method according to any one of claims 1 to 5.

Images